Improve documentation

DEVELOPMENT.md

@@ -1,119 +1,190 @@
-# Development
+# Development Guide
+
+This guide will help you set up the development environment, deploy the backend, and configure necessary integrations and credentials.
+
+<!-- TOC -->
+* [Development Guide](#development-guide)
+  * [Prerequisites](#prerequisites)
+  * [Setup](#setup)
+    * [1. Install Firebase CLI](#1-install-firebase-cli)
+    * [2. Install Project Dependencies](#2-install-project-dependencies)
+    * [3. Run the Project Locally](#3-run-the-project-locally)
+  * [Deployment](#deployment)
+    * [1. Login to Firebase](#1-login-to-firebase)
+    * [2. Deploy Project to Firebase](#2-deploy-project-to-firebase)
+  * [Optional Local Setup for Firebase Admin SDK](#optional-local-setup-for-firebase-admin-sdk)
+    * [Setting Up Credentials](#setting-up-credentials)
+      * [Linux / MacOS](#linux--macos)
+      * [Windows (PowerShell)](#windows-powershell)
+  * [Integrations & Environment Variables](#integrations--environment-variables)
+    * [Discord Integration](#discord-integration)
+    * [GitHub Integration](#github-integration)
+    * [Internal Token (for Self-Authentication)](#internal-token-for-self-authentication)
+  * [Updating Configuration Variables in Firebase](#updating-configuration-variables-in-firebase)
+  * [Local Development Commands](#local-development-commands)
+  * [Troubleshooting & Tips](#troubleshooting--tips)
+<!-- TOC -->
+
+## Prerequisites
+
+Ensure you have the following installed:
+
+- **Node.js** (Recommended: Version 20 or later)
+- **npm** (Node package manager)
 
 ## Setup
 
-Install firebase globally
+### 1. Install Firebase CLI
+
+To interact with Firebase, install the Firebase CLI globally:
 
 ```bash
-npm i -g firebase-tools
+npm install -g firebase-tools
 ```
 
-Install dependencies
+### 2. Install Project Dependencies
+
+Navigate to the project directory and install all required dependencies:
 
 ```bash
 npm install
 ```
 
-Run everything locally
+### 3. Run the Project Locally
+
+You can serve the Firebase functions, Firestore, and any other emulated services locally:
+
+```bash
+firebase emulators:start
+```
+
+Alternatively, to only run specific services, use the `--only` flag:
 
 ```bash
-firebase serve
+firebase emulators:start --only functions,firestore
 ```
 
+This command starts up all emulators based on your configuration in `firebase.json`.
+
+---
+
 ## Deployment
 
-___Note:__ for this you will need access to the project._
+_Note: Deployment requires access to the Firebase project. You can also deploy to a different project by changing the project ID in the `.firebaserc` file._
+
+### 1. Login to Firebase
 
-Login to your account
+Ensure you're authenticated with the correct Firebase account:
 
 ```bash
 firebase login
 ```
 
-Deploy everything
+### 2. Deploy Project to Firebase
+
+Deploy all services (functions, Firestore rules, hosting, etc.) to the configured Firebase project:
 
 ```bash
 firebase deploy
 ```
 
-## Additional local setup (optional)
+To deploy specific services only, use the `--only` flag:
+
+```bash
+firebase deploy --only functions
+```
+
+---
+
+## Optional Local Setup for Firebase Admin SDK
 
-#### Credentials
+### Setting Up Credentials
 
-To be able to use functions that use the Firebase AdminSDK you need to set credentials.
+To use Firebase Admin SDK locally (e.g., for accessing Firestore securely), you need to provide service account credentials:
 
-1. Download the service account file from firebase
-2. Set the path to that file to an environment variable
+1. Download the service account JSON file from your Firebase project settings.
+2. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the path of this file.
 
-__Linux / MacOS__
+#### Linux / MacOS
 
 ```bash
-export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"
+export GOOGLE_APPLICATION_CREDENTIALS="$HOME/Downloads/service-account-file.json"
 ```
 
-__Windows (PowerShell)__
+#### Windows (PowerShell)
 
 ```ps
 $env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"
 ```
 
-_(for more information, please see the [docs](https://firebase.google.com/docs/admin/setup))_
+_For more information, see the [Firebase Admin SDK setup documentation](https://firebase.google.com/docs/admin/setup)._
 
-#### Integrations
+---
 
-To test specific functionality like the integrations you will also have to set the following environment variables
+## Integrations & Environment Variables
 
-__Discord__
+To test integrations like Discord and GitHub, set the following environment variables:
+
+### Discord Integration
 
 ```bash
-export discord.token="my_discord_token"
+export DISCORD_TOKEN="your_discord_token"
 ```
 
-__Github__
+### GitHub Integration
 
 ```bash
-export github.client-secret="my_github_app_client_secret"`
-export github.private-key="my_github_app_private_key"
+export GITHUB_CLIENT_SECRET="your_github_app_client_secret"
+export GITHUB_PRIVATE_KEY="your_github_app_private_key"
 ```
 
-__Internal__
+### Internal Token (for Self-Authentication)
 
-_Internal token is used for self-authentication and for communication with th
-[docker repo](https://github.com/Unity-CI/docker)._
+The internal token is used for secure communication, such as with the [Docker repo](https://github.com/Unity-CI/docker).
 
-```
-export internal.token="my_internal_token"
+```bash
+export INTERNAL_TOKEN="your_internal_token"
 ```
 
-_(value can be any single-line string, as long as it's the same in the docker repo)_
+The token can be any string, as long as it matches the token configured in the related Docker repo.
 
-## Local Commands
+---
 
-In order to run firebase locally simply use
+## Updating Configuration Variables in Firebase
 
-```
-firebase serve
+If you need to set or update configuration variables (e.g., when migrating environments or rotating security tokens), use the following commands:
+
+```bash
+firebase functions:config:set discord.token="your_discord_token"
+firebase functions:config:set github.client-secret="your_github_app_client_secret"
+firebase functions:config:set github.private-key="your_github_app_private_key"
+firebase functions:config:set internal.token="your_internal_token"
 ```
 
-To only run one component, like `hosting`, `functions` or `firestore` you may use the `--only` flag.
+---
 
-```
-firebase serve --only functions
-```
+## Local Development Commands
 
-If everything works, finally deploy the changes
+- **Start Firebase Emulators:** To run all services locally:
+  ```bash
+  firebase emulators:start
+  ```
 
-```
-firebase deploy
-```
+- **Start Specific Emulators Only:** To run specific components (e.g., `functions` only):
+  ```bash
+  firebase emulators:start --only functions
+  ```
 
-## Updating env/config variables 
+- **Deploy Changes:** After making changes, deploy to Firebase:
+  ```bash
+  firebase deploy
+  ```
 
-_Typically this is only needed when migrating to new firebase project or environment, or when security token needs to rotate._
+---
 
-```
-firebase functions:config:set discord.token="my_discord_token"
-firebase functions:config:set github.client-secret="my_github_app_client_secret"
-firebase functions:config:set discord.private-key="my_github_app_private_key"
-firebase functions:config:set internal.token="my_internal_token"
-```
+## Troubleshooting & Tips
+
+- **Ensure Environment Variables are Set:** Make sure all necessary environment variables are set correctly before running the emulators or deploying.
+- **Testing Integrations Locally:** When testing functionality that requires external services (e.g., Discord, GitHub), ensure your local credentials are set up as described above.
+
+Happy coding!

README.md

@@ -1,47 +1,78 @@
 # GameCI Versioning Backend
 
-## Unity version ingest
+The GameCI Versioning Backend automates the tracking, scheduling, and reporting of Unity version builds and Docker image workflows. It integrates with GitHub and Discord to streamline CI/CD processes and notifications.
 
-TODO - Describe how it works
+## Overview
 
-## game-ci/docker version ingest
+This backend handles the ingestion of both Unity versions and GameCI Docker versions. It uses Firebase for scheduling workflows, managing job queues, and storing data.
 
-TODO - Describe how it works
+---
 
-## Scheduler
+## Unity Version Ingest
 
-Each CiJob starts its own workflow.
+[See `scrapeVersions.ts`](functions/src/logic/ingestUnityVersions/scrapeVersions.ts)
 
-Each Workflow generates multiple CiBuilds: one per baseOs-targetPlatform combination.
-The CiJob workflow will report back a CiBuild for each combination.
+The backend regularly scrapes [Unity's version archive page](https://unity.com/releases/editor/archive) to detect new releases or updates to Unity editor versions. When new versions are found:
+1. They are added to the Firestore database.
+2. Notifications are sent to Discord channels to keep maintainers informed.
+3. New CI build jobs are scheduled as necessary.
 
-For example:
+---
 
-```text
-...
-  ubuntu-<version>-linuxIl2cpp
-  ubuntu-<version>-webgl
-  windows-<version>-webgl
-...
-```
+## GameCI/Docker Version Ingest
+
+This backend monitors and ingests new versions for the GameCI Docker images. These versions are tracked in conjunction with Unity versions, ensuring Docker images are built and maintained for each compatible Unity version.
+
+---
 
-An endpoint from this backend will listen to the reports and update the database.
-Each CiBuild starts with the status "started" after it is being reported.
+## Scheduler
 
-When the last CiBuild is set to "published", the CiJob for that version is also set to "completed".
-Completed CiJobs are reported to Discord.
+The scheduler coordinates the creation and tracking of CI jobs and builds:
+- **CiJob Workflow**: Each `CiJob` corresponds to a workflow for building Docker images for Unity versions. This workflow schedules multiple `CiBuilds`.
+- **CiBuild Creation**: A `CiJob` generates multiple `CiBuilds`, with each build representing a unique combination of `baseOs` and `targetPlatform`.
+  - Examples:
+    ```
+    ubuntu-<version>-linuxIl2cpp
+    ubuntu-<version>-webgl
+    windows-<version>-webgl
+    ```
+- **Status Tracking**: Once a `CiBuild` is created, it starts with the status `started`. The backend listens for updates on these builds:
+  - When a build completes successfully, its status is updated to `published`.
+  - Once all `CiBuilds` for a `CiJob` are published, the `CiJob` is marked as `completed`.
+- **Notifications**: The status of `CiJobs` and `CiBuilds` is reported to Discord for visibility and tracking.
+
+---
 
 ## Ingeminator
 
-TODO - Describe how it works
+_The "Ingeminator" is responsible for rescheduling builds that fail or require retries._
+
+The backend includes logic to detect and retry failed builds. This ensures that all Unity and Docker versions are correctly built, and no job is left incomplete. Details about how the "Ingeminator" works are to be documented.
+
+---
 
 ## Database Backup
 
-The firestore database can be backed up with the following command:
-`yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
+To back up the Firestore database, use the following command:
+
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS="./path/to/serviceAccountKey.json"
+yarn run backfire export ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
+```
+
+To restore a backup:
+
+```bash
+export GOOGLE_APPLICATION_CREDENTIALS="./path/to/serviceAccountKey.json"
+yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile $GOOGLE_APPLICATION_CREDENTIALS
+```
+
+_It's recommended to empty the database before restoring to avoid data conflicts, or use flags such as `overwrite` and `merge` to control restoration rules._
+
+More information about the `backfire` tool can be found in the [official documentation](https://github.com/benyap/firestore-backfire).
+
+---
 
-Similarly, it can be used to restore a backup with:
-`yarn run backfire import ./export/versioningBackendBackup --project unity-ci-versions --keyFile <PATH_TO_GOOGLE_CLOUD_SERVICE_ACCOUNT_KEYFILE.json>`
+## Development
 
-You likely would want to empty the database before restoring but you can also use flags like overwrite, merge, etc to control the restoration
-rules.
+For instructions on setting up the development environment, running the project locally, and deploying changes, see [DEVELOPMENT.md](./DEVELOPMENT.md).