README.md

infra

This folder contains all the infrastructure needed for setting up the data platform.

localstack-services

I am using the following localstack services:

s3

I am using S3 to store data.

secretsmanager

I am using Secrets Manager to store secrets.

airbyte

I am using Airbyte to extract and load data to the postgres database.

airflow

I am using Airflow to orchestrate ETL jobs.

dbt

I am using dbt to transform data within the postgres database.

To retrieve the password for the postgresql user postgres, run the following:

kubectl get secret --namespace dbt postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode ; echo

grafana

I am using Grafana to monitor all services.

To retrieve the password for the grafana user admin, run the following:

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

localstack

I am using localstack to create a local stack of AWS resoures.

prometheus

I am using Prometheus to scrape metrics of all services.

superset

I am using Superset to visualise data.