Aero-offers

This project aims at reviving aero-offers.com - invaluable source of price trends for gliders and other aircrafts, originally developed and maintained by @rthaenert

Development

Project structure (building blocks / deployment units)

Currently, the project is running on Azure Cloud as a set of Azure Container Apps and Jobs.

• ui - vue.js application, bundled as Docker image based on nginx. Deployed as Azure Container App (running 24/7)
• backend/api - python flask app with few REST endpoints, reversed proxied by nginx serving frontend (not exposed directly to the internet). Deployed as Azure Container App (running 24/7)
• backend/jobs - python scripts triggered periodically. Much more resource-heavy than API, bundled in the same Docker image as api, deployed as Azure Container Job (with overridden command)
  • run_update_offers - scans few portals (e.g. soaring.de) and stores new offers in the database (not yet classified), then assigns manufacturer and model to new (not yet classified) offers stored in the database
  • run_update_fx_rates - updates currency exchange rates from ECP REST api
• db - PostgreSQL 16 database with DDL scripts managed by Flyway. Currently running inside cheapest possible Azure VM.

Deployment

Trunk Based Development and Continuous Deployment is utilized here - all changes pushed/merged to main are automatically deployed to production env.

TODO

• Scraper: update db offer if price or location (or any other parameter) has changed
• Dev: end 2 end tests of crawlers
• UI: consent banner for GA
• Infra: db daily backups
• Infra: infra as code (biceps or terraform)
• document infra and env topology
• Backend: fix and enable other spiders/crawlers
• UI: aero-offers.com
• Improve aircraft types structure and introduce 2 levels: glider (e.g Discus 2c 18m) and model (Discus 2cFES 18m) as prices between models sometimes differ significantly
• UI: fix & polish CSS in UI
• utilize https://github.com/weglide/GliderList as source of truth for glider types/models
• UI: admin panel for manual (re) classification (or community-based)
• crawler for Facebook Marketplace - do they have nice api?
• crawler for https://www.aircraft24.de
• crawler for http://www.airplanemart.com
• crawler for http://www.aeronave.de/1-luftfahrzeuge/listings.html
• crawler for https://plane-sale.com

Running locally without Python nor NodeJS

docker compose up --build - starts postgres, python backend and UI apps (http://localhost:8080/)

Prerequisites for local development with hot reloads

• python 3.12+, pip3, flask
• docker (compose)
• npm

Start Postgres in docker (available for debugging via localhost:25432):

docker-compose up postgres flyway

Unzip database backup (optional) and load into DB

cd db && unzip -qq prod_dump_2024_06_31.sql.zip

Install python packages

cd backend
pip3 install -r requirements.txt -r tests/requirements.txt 

Start backend api (python app):

cd backend
./start_api.sh

Start UI (vue app):

cd ui
npm run dev

UI has own, detailed README.md file.

Run crawlers/spiders & reclassifier:

cd backend
./run_spiders.sh && ./run_classifier.sh