Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V2 Migration #47

Merged
merged 28 commits into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
679f933
Fix issues discovered during deployment
AndrewKahr Apr 22, 2023
3c56b9d
Fix compile issues
AndrewKahr Apr 22, 2023
4915b72
Fix predeploy
AndrewKahr Apr 22, 2023
d12b2ac
Add test function deployment
AndrewKahr Apr 22, 2023
67e3480
Fix linting
AndrewKahr Apr 22, 2023
db20947
Fix linting
AndrewKahr Apr 22, 2023
e6a260e
Pin versions, add module to package type
AndrewKahr Apr 22, 2023
843bd8a
Fix linting
AndrewKahr Apr 22, 2023
3cfea6d
Revert to commonjs
AndrewKahr Apr 22, 2023
8c1ce71
Revert module resolution
AndrewKahr Apr 22, 2023
3f1171c
Fix missing token field
AndrewKahr Apr 22, 2023
97c06d3
Remove token
AndrewKahr Apr 22, 2023
f18e100
Fix lint
AndrewKahr Apr 22, 2023
cd6a9c9
Revert node fetch version
AndrewKahr Apr 22, 2023
0a6821a
Cleanup test function
AndrewKahr Apr 22, 2023
7aba8d0
Use force delete
AndrewKahr Apr 22, 2023
6d93f00
Finish migration to v2 functions. Converts to new secrets management.…
AndrewKahr Jun 4, 2024
59f9247
Fix formatting
AndrewKahr Jun 4, 2024
1e9e46a
Remove require
AndrewKahr Jun 4, 2024
abdc8e9
Read file contents
AndrewKahr Jun 4, 2024
cedc50a
Downgrade node-fetch
AndrewKahr Jun 4, 2024
318af17
Downgrade octokit
AndrewKahr Jun 4, 2024
2280e5e
Verbose secret paths
AndrewKahr Jun 4, 2024
3478c03
Improve service account credential read
AndrewKahr Jun 4, 2024
f605d87
Force service account credential
AndrewKahr Jun 4, 2024
5d4219e
Revert secret names
AndrewKahr Jun 4, 2024
42782bd
Add fallback path to service account credentials
AndrewKahr Jun 4, 2024
a43b241
Fixes from code review. Docs on database backup/restore
AndrewKahr Jun 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 51 additions & 20 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -1,61 +1,92 @@
name: 🚀

on:
pull_request: { }
push: { branches: [ main ] }
pull_request: {}
push: { branches: [main] }

jobs:
test:
name: 🧪 Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 16
node-version: 20
cache: 'yarn'
- name: install dependencies
run: yarn && yarn --cwd ./functions
- name: run linter
run: yarn lint && yarn --cwd ./functions lint
- name: run tests
run: yarn test && yarn --cwd ./functions test
# - name: Upload test results
# uses: actions/upload-artifact@v1
# with:
# name: Test results
# path: "**/artifacs"
# - name: Upload test results
# uses: actions/upload-artifact@v1
# with:
# name: Test results
# path: "**/artifacs"

build:
name: 🛠 Build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 16
node-version: 20
cache: 'yarn'
- name: install dependencies
run: yarn && yarn --cwd ./functions
- name: build
run: yarn --cwd ./functions build

testDeploy:
name: Test Deploy
needs: [test, build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: 'yarn'
- name: install dependencies
run: yarn && yarn --cwd ./functions
- name: Deploy test to Firebase
uses: w9jds/firebase-action@v13.10.2
with:
args: deploy --only functions:testFunction
env:
GCP_SA_KEY: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_UNITY_CI_VERSIONS }}'
- name: Cleanup Firebase Test
uses: w9jds/firebase-action@v13.10.2
with:
args: functions:delete testFunction --force
env:
GCP_SA_KEY: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_UNITY_CI_VERSIONS }}'
Comment on lines +43 to +66
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would want to keep it as this is a great way to ensure the project will deploy properly for the actual functions since it requires roughly the same permissions as the main functions and has been setup to require all the secrets that the other functions might use as a deploy test


deploy:
name: ✨ Deploy
needs: [ test, build ]
needs: [test, build, testDeploy]
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 16
yarn: 'yarn'
node-version: 20
cache: 'yarn'
- name: install dependencies
run: yarn && yarn --cwd ./functions
- name: Deploy to Firebase
uses: w9jds/firebase-action@v11.24.1
uses: w9jds/firebase-action@v13.10.2
with:
args: deploy
env:
FIREBASE_TOKEN: ${{ secrets.FIREBASE_TOKEN }}
GCP_SA_KEY: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_UNITY_CI_VERSIONS }}'
- name: Cleanup Firebase Test
uses: w9jds/firebase-action@v13.10.2
with:
args: functions:delete testFunction --force
env:
GCP_SA_KEY: '${{ secrets.FIREBASE_SERVICE_ACCOUNT_UNITY_CI_VERSIONS }}'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to use --except flag and deploy in a single transaction, than to later delete a partial deployment.

3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -68,4 +68,5 @@ node_modules/
# IDE
.idea
.vs
.vscode
.vscode
functions/.yarn/
6 changes: 3 additions & 3 deletions firebase.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
},
"functions": {
"predeploy": [
"npm --prefix \"$RESOURCE_DIR\" run lint",
"npm --prefix \"$RESOURCE_DIR\" run test",
"npm --prefix \"$RESOURCE_DIR\" run build"
"yarn --cwd \"$RESOURCE_DIR\" lint",
"yarn --cwd \"$RESOURCE_DIR\" test",
"yarn --cwd \"$RESOURCE_DIR\" build"
]
},
"hosting": {
Expand Down
1 change: 1 addition & 0 deletions functions/.yarnrc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
nodeLinker: node-modules
51 changes: 28 additions & 23 deletions functions/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,35 +12,40 @@
"test": "echo \"No tests yet, feel free to add\" && exit 0"
},
"engines": {
"node": "16"
"node": "20"
},
"main": "lib/index.js",
"dependencies": {
"@octokit/auth-app": "^4.0.9",
"@octokit/rest": "^19.0.7",
"@octokit/auth-app": "^6.1.1",
"@octokit/rest": "^20.1.1",
"eris": "^0.17.2",
"firebase-admin": "^11.7.0",
"firebase-functions": "^4.3.0",
"firebase-admin": "^12.1.1",
"firebase-functions": "^5.0.1",
"httpie": "^1.1.2",
"jsdom": "^21.1.1",
"lodash": "^4.17.20",
"node-fetch": "^3.3.1",
"semver": "^7.3.2"
"jsdom": "^24.1.0",
"lodash": "^4.17.21",
"node-fetch": "^2.7.0",
"semver": "^7.6.2"
},
"devDependencies": {
"@types/jsdom": "^21.1.1",
"@types/lodash": "^4.14.162",
"@types/node": "^18.15.11",
"@types/node-fetch": "^2.5.7",
"@types/semver": "^7.3.4",
"@types/ws": "^8.5.4",
"@typescript-eslint/eslint-plugin": "^5.59.0",
"@typescript-eslint/parser": "^5.59.0",
"eslint": "^8.38.0",
"eslint-plugin-import": "^2.22.0",
"firebase-functions-test": "^3.1.0",
"jest": "^29.5.0",
"typescript": "^5.0.4"
"@octokit/types": "^13.5.0",
"@types/jsdom": "^21.1.7",
"@types/lodash": "^4.17.4",
"@types/node": "^20.14.0",
"@types/node-fetch": "^2.6.11",
"@types/semver": "^7.5.8",
"@types/ws": "^8.5.10",
"@typescript-eslint/eslint-plugin": "^7.12.0",
"@typescript-eslint/parser": "^7.12.0",
"eslint": "^8.57.0",
"eslint-plugin-import": "^2.29.1",
"firebase-functions-test": "^3.3.0",
"jest": "^29.7.0",
"typescript": "^5.4.5"
},
"private": true
"private": true,
"volta": {
"node": "20.14.0",
"yarn": "1.22.22"
}
}
1 change: 1 addition & 0 deletions functions/src/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ export { reportNewBuild } from './reportNewBuild';
export { reportPublication } from './reportPublication';
export { unityVersions } from './unityVersions';
export { retryBuild } from './retryBuild';
export { testFunction } from './testFunction';
5 changes: 2 additions & 3 deletions functions/src/api/queueStatus.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
import { functions } from '../service/firebase';
import { Request } from 'firebase-functions/v2/https';
import { onRequest, Request } from 'firebase-functions/v2/https';
import { Response } from 'express-serve-static-core';
import { CiJobs } from '../model/ciJobs';
import { CiBuilds } from '../model/ciBuilds';

export const queueStatus = functions.https.onRequest(async (req: Request, res: Response) => {
export const queueStatus = onRequest(async (req: Request, res: Response) => {
const jobs = await CiJobs.getAll();
const builds = await CiBuilds.getAll();

Expand Down
17 changes: 0 additions & 17 deletions functions/src/api/repoVerions.ts

This file was deleted.

15 changes: 15 additions & 0 deletions functions/src/api/repoVersions.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import { onRequest, Request } from 'firebase-functions/v2/https';
import { logger } from 'firebase-functions/v2';
import { Response } from 'express-serve-static-core';
import { RepoVersionInfo } from '../model/repoVersionInfo';

export const repoVersions = onRequest(async (request: Request, response: Response) => {
try {
const versions = await RepoVersionInfo.getAllIds();

response.send(versions);
} catch (err) {
logger.error(err);
response.send('Oops.');
}
});
67 changes: 40 additions & 27 deletions functions/src/api/reportBuildFailure.ts
Original file line number Diff line number Diff line change
@@ -1,43 +1,56 @@
import { firebase, functions } from '../service/firebase';
import { Request } from 'firebase-functions/v2/https';
import { onRequest, Request } from 'firebase-functions/v2/https';
import { logger } from 'firebase-functions/v2';
import { Response } from 'express-serve-static-core';
import { Token } from '../config/token';
import { BuildFailure, CiBuilds } from '../model/ciBuilds';
import { CiJobs } from '../model/ciJobs';
import { Discord } from '../service/discord';
import { defineSecret } from 'firebase-functions/params';

export const reportBuildFailure = functions.https.onRequest(async (req: Request, res: Response) => {
try {
if (!Token.isValid(req.header('authorization'))) {
firebase.logger.warn('unauthorised request', req.headers);
res.status(403).send('Unauthorized');
return;
}
const discordToken = defineSecret('DISCORD_TOKEN');
const internalToken = defineSecret('INTERNAL_TOKEN');

export const reportBuildFailure = onRequest(
{ secrets: [discordToken, internalToken] },
async (req: Request, res: Response) => {
const discordClient = new Discord();
await discordClient.init(discordToken.value());

try {
if (!Token.isValid(req.header('authorization'), internalToken.value())) {
logger.warn('unauthorised request', req.headers);
res.status(403).send('Unauthorized');
return;
}

const { body } = req;
firebase.logger.debug('Build failure report incoming.', body);
const { body } = req;
logger.debug('Build failure report incoming.', body);

const { jobId, buildId, reason } = body;
const failure: BuildFailure = { reason };
const { jobId, buildId, reason } = body;
const failure: BuildFailure = { reason };

await CiJobs.markFailureForJob(jobId);
await CiBuilds.markBuildAsFailed(buildId, failure);
await CiJobs.markFailureForJob(jobId);
await CiBuilds.markBuildAsFailed(buildId, failure);

firebase.logger.info('Build failure reported.', body);
res.status(200).send('OK');
} catch (err: any) {
const message = `
logger.info('Build failure reported.', body);
res.status(200).send('OK');
} catch (err: any) {
const message = `
Something went wrong while reporting a build failure
${err.message}
`;
firebase.logger.error(message, err);
await Discord.sendAlert(message);
logger.error(message, err);

await discordClient.sendAlert(message);

if (req.body?.jobId?.toString().startsWith('dryRun')) {
await CiBuilds.removeDryRunBuild(req.body.buildId);
await CiJobs.removeDryRunJob(req.body.jobId);
}

if (req.body?.jobId?.toString().startsWith('dryRun')) {
await CiBuilds.removeDryRunBuild(req.body.buildId);
await CiJobs.removeDryRunJob(req.body.jobId);
res.status(500).send('Something went wrong');
}

res.status(500).send('Something went wrong');
}
});
await discordClient.disconnect();
},
);
Loading