Skip to content

Latest commit

 

History

History
149 lines (101 loc) · 6.2 KB

DEVELOPMENT.md

File metadata and controls

149 lines (101 loc) · 6.2 KB

aws-tileserver - Developer documentation

This page gives a detailled overview of all techiques used for this project.

AWS-Architecture

Overall architecture

architecture

Creating the postgis-client Docker-image

  1. User pushes new commit to Github
  2. Github creates webhook for AWS CodeBuild
  3. New Docker-Image is created and pushed to ECR

Serving static Content

  1. Client sends http-request to URL assigned to CloudFront Distribution.
  2. CloudFront reads/caches static content from public S3 Bucket.
  3. Http-response is sent to Client.

Serving vectortiles

  1. Client sends http-request to URL assigned to CloudFront Distribution.
  2. CloudFront reads/caches vectortiles from S3 Bucket.
  3. Temporary-redirect (307) to tileserver if vectortile is not available on S3.
  4. Tileserver generates vectortile with data from RDS-postgres-instance, sends it to client and also stores it on S3.

Lambda Implementation Details

Database

The database-instance MUST support at least postgis 2.4.0. Otherwise vectortiles can't be created.

Vector Tiles

aws-tileserver supports configurable REST-endpoints for vector tiles according to Vector Tile Specification 2.1. Each endpoint provides access to a vectortile with configurable layers.

SQL-Query

Each layer is resolved to the following query:

(SELECT ST_AsMVT(q, '${layer.name}', ${layerExtend}, 'geom') as data FROM
    (SELECT ${prefix}ST_AsMvtGeom(
        ${geom},
        ${bbox},
        ${layerExtend},
        ${buffer},
        ${clip_geom}
        ) AS geom${keys}
    FROM ${layer.table} WHERE (${geom} && ${bbox})${where}${postfix}) as q)

All resulting layers are merged into one SQL query:

SELECT ( [${layer1} [|| ${layer2} [|| ...]]] ) as data

Performance, Benchmarks & Timing

stack

Setup

  1. 990 (+5 for warm-up) HTTP/2-Requests (IPv4) were made to https://tileserver.cyclemap.link/local/14/8691/5677.mvt
  2. Everything is deployed to eu-central-1
  3. Client timing was collected with curl (see tools/benchmark.sh)
  4. Lambda durations were collected from CloudWatch
  5. The raw results can be found in docs/benchmark.ods

Update 2023

While upgrading to Nodejs 18, I repeated the benchmarks on 2023-09-06 07:00 UTC. Interestingly, the timing has changed in general:

Node Version Lambda Timing
Nodejs 12 2019 303 ms
Nodejs 12 2023 867 ms
Nodejs 18 2023 670 ms

Further investigation is needed to determine the root cause. Maybe the database is now bigger (=slower).

Next Steps

  • Move database to Serverless Aurora PostgreSQL to reduce monthly costs. Won't do. Resume after pause is 30s+ and keeping 2 ACUs hot at all times is too expensive.
  • Evaluate Data API for Aurora Serverless. Won't do. See above. According to this review, performance also seems bad compared to API-calls.
  • move terraform-state to s3-bucket. Done!
  • Security-Review for Lambda-Code (e.g. SQL-Injection, ...)
  • Change all scripts to use Postgres environment variables (PGUSER, ...) Only relevant for database processing. Is out of scope here.
  • Omit Postgres credentials altogether and use IAM-role instead
  • move lambda-function out of VPC to reduce cold-start-time Not needed anymore. See https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/
  • Add raster endpoint with node-mapbox-gl-native to serve pre-rendered raster-images.
  • Check how blue-green deployments could be realized with API Gateway and Lambda.

References

AWS

Typescript, JavaScript

Terraform, Infrastructure-as-Code

Postgres

Benchmark, Optimization

UI, Design

Vectortiles