Skip to content

Commit

Permalink
Cron jobs setup for offers and fx rates updaters
Browse files Browse the repository at this point in the history
  • Loading branch information
lwitkowski committed Jul 30, 2024
1 parent 35a623b commit 2802410
Showing 7 changed files with 71 additions and 31 deletions.
16 changes: 15 additions & 1 deletion .github/workflows/cd-backend.yaml
Original file line number Diff line number Diff line change
@@ -84,4 +84,18 @@ jobs:
containerAppEnvironment: cae-aerooffers-prod
containerAppName: ca-aerooffers-api
imageToDeploy: ${{ env.API_DOCKER_IMAGE }}
disableTelemetry: true
disableTelemetry: true

- name: 'Deploy Job - update fx rates'
run: |
az containerapp job update \
-g rg-aerooffers \
--name aerooffers-update-fx-job \
--image ${{ env.API_DOCKER_IMAGE }}
- name: 'Deploy Job - update offers'
run: |
az containerapp job update \
-g rg-aerooffers \
--name aerooffers-update-offers-job \
--image ${{ env.API_DOCKER_IMAGE }}
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -10,24 +10,24 @@ This project aims at reviving www.aero-offers.com - invaluable source of price t
### Project structure (building blocks / deployment units)
- `frontend` - vue.js application deployed as dockerized static web app served by nginx
- `backend/api` - python flask app with few REST endpoints, reversed proxied by nginx serving frontend (not exposed directly to the internet).
- `backend/jobs` - python scripts triggered by scheduled job (e.g once a day)
- `backend/jobs` - python scripts triggered by scheduled job (e.g once a day). Please mind those jobs may be much much more resource heavy than API, and should not be triggered from within `api` container which is optimised to handle REST api traffic.
- `job_fetch_offers` - scans few portals (e.g. soaring.de) and stores new offers in the database (not yet classified)
- `job_reclassify_offers` - assigns manufacturer and model to new (not yet classified) offers stored in the database
- `job_update_exchange_rates` - updates currency exchange rates based ok ECP api
- `db` - PostgreSQL 16 database with DDL scripts managed by Flyway
- `db` - PostgreSQL 16 database with DDL scripts managed by Flyway. Currently running inside cheapest possible Azure VM.

### Prod environment
Currently the project is being onboarded to Azure Cloud (still WIP).
Currently, the project is being onboarded to Azure Cloud (still WIP).

### TODO
- [x] deploy working ui, api and db to Azure
- [x] fix segelflug spider/crawler
- [ ] managed db with persistent storage (it's running in ephemeral container atm)
- [ ] fix other spiders/crawlers
- [ ] use Azure secrets for db credentials
- [ ] setup cron triggers for crawlers, reclassifier and FX rates updater (Azure Functions?)
- [x] good enough DB setup (cheap VM)
- [x] use Azure secrets for db credentials
- [x] setup cron triggers for crawlers, reclassifier and FX rates updater
- [ ] infra as code (biceps or terraform)
- [ ] document infra and env topology
- [ ] fix other spiders/crawlers
- [ ] human readable domain (aero-offers.com?)
- [ ] fix aircraft type dropdown
- [ ] fix & polish CSS in UI
@@ -44,7 +44,7 @@ Currently the project is being onboarded to Azure Cloud (still WIP).

Start Postgres in docker (available for debugging via `localhost:25432`):
```bash
docker-compose up ca-aerooffers-db flyway
docker-compose up postgres flyway
```

Unzip database backup (optional) and load into DB
57 changes: 40 additions & 17 deletions azure/setup_azure_infra.sh
Original file line number Diff line number Diff line change
@@ -2,6 +2,12 @@ RESOURCE_GROUP=rg-aerooffers
LOCATION="Switzerland North"
ENV_NAME=cae-aerooffers-prod
ACR=lwitkowski.azurecr.io
DOCKER_IMAGE_TAG=e314458
DB_HOST=172.161.107.160
DB_PORT=5432
DB_NAME="aerooffers"
DB_USER="aerooffers"
DB_PASS="TIpV7iMz QxrY0lee Ylw6VXxV"

az group create \
--name $RESOURCE_GROUP \
@@ -22,26 +28,13 @@ az identity create \

docker login $ACR

az containerapp create \
--name ca-aerooffers-db \
--resource-group $RESOURCE_GROUP \
--environment $ENV_NAME \
--registry-server $ACR \
--image $ACR/aerooffers-db:1 \
--env-vars POSTGRES_DB="aircraft_offers" POSTGRES_USER="aircraft_offers" POSTGRES_PASSWORD="aircraft_offers" \
--target-port 5432 \
--ingress internal \
--transport tcp \
--cpu 0.25 --memory 0.5Gi \
--min-replicas 1 --max-replicas 1

az containerapp create \
--name ca-aerooffers-api \
--resource-group $RESOURCE_GROUP \
--environment $ENV_NAME \
--registry-server $ACR \
--image $ACR/aerooffers-api:a7e18a5 \
--env-vars DB_HOST="ca-aerooffers-db" DB_PORT="5432" DB_NAME="aircraft_offers" DB_USER="aircraft_offers" DB_PW="aircraft_offers" \
--image $ACR/aerooffers-api:$DOCKER_IMAGE_TAG \
--env-vars DB_HOST="???" DB_PORT="5432" DB_NAME="???" DB_USER="???" DB_PW="???" \
--target-port 80 \
--ingress internal \
--transport tcp \
@@ -53,9 +46,39 @@ az containerapp create \
--resource-group $RESOURCE_GROUP \
--environment $ENV_NAME \
--registry-server $ACR \
--image $ACR/aerooffers-ui:a7e18a5 \
--image $ACR/aerooffers-ui:$DOCKER_IMAGE_TAG \
--target-port 80 \
--cpu 0.25 --memory 0.5Gi \
--min-replicas 1 --max-replicas 1 \
--ingress external \
--query properties.configuration.ingress.fqdn
--query properties.configuration.ingress.fqdn


# create scheduled jobs
az containerapp job create \
--name aerooffers-update-fx-job \
--resource-group $RESOURCE_GROUP \
--environment $ENV_NAME \
--trigger-type "Schedule" \
--cron-expression "15 8,16 * * *" \
--replica-timeout 1800 \
--registry-server $ACR \
--image $ACR/aerooffers-api:$DOCKER_IMAGE_TAG \
--secrets "db-user=$DB_USER" "db-password=$DB_PASS" \
--env-vars "DB_HOST=$DB_HOST" "DB_PORT=$DB_PORT" "DB_NAME=$DB_NAME" "DB_USER=secretref:db-user" "DB_PW=secretref:db-password" \
--command "run_update_fx_rates.sh" \
--cpu "0.25" --memory "0.5Gi"

az containerapp job create \
--name aerooffers-update-offers-job \
--resource-group $RESOURCE_GROUP \
--environment $ENV_NAME \
--trigger-type "Schedule" \
--cron-expression "17 3 * * *" \
--replica-timeout 1800 \
--registry-server $ACR \
--image $ACR/aerooffers-api:$DOCKER_IMAGE_TAG \
--secrets "db-user=$DB_USER" "db-password=$DB_PASS" \
--env-vars "DB_HOST=$DB_HOST" "DB_PORT=$DB_PORT" "DB_NAME=$DB_NAME" "DB_USER=secretref:db-user" "DB_PW=secretref:db-password" \
--command "run_update_offers.sh" \
--cpu "1" --memory "2Gi"
File renamed without changes.
1 change: 1 addition & 0 deletions backend/run_update_fx_rates.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
python3 job_update_fx_rates.py
2 changes: 2 additions & 0 deletions backend/run_update_offers.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
./run_spiders.sh
./run_reclassifier.sh
10 changes: 5 additions & 5 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -11,16 +11,16 @@ services:
ports:
- '8081:80' # exposed for debugging and local frontend development without python
environment:
- DB_HOST=ca-aerooffers-db
- DB_HOST=postgres
- DB_PORT=5432
- DB_NAME=aircraft_offers
- DB_USER=aircraft_offers
- DB_PW=aircraft_offers
depends_on:
- ca-aerooffers-db
- postgres
- flyway

ca-aerooffers-db:
postgres:
image: postgres:16.3-alpine
environment:
- POSTGRES_DB=aircraft_offers
@@ -31,8 +31,8 @@ services:

flyway:
image: flyway/flyway:10-alpine
command: -url=jdbc:postgresql://ca-aerooffers-db:5432/aircraft_offers -user=aircraft_offers -password=aircraft_offers migrate
command: -url=jdbc:postgresql://postgres:5432/aircraft_offers -user=aircraft_offers -password=aircraft_offers migrate
volumes:
- ./db/migrations:/flyway/sql
depends_on:
- ca-aerooffers-db
- postgres

0 comments on commit 2802410

Please sign in to comment.