Skip to content

Latest commit

 

History

History
36 lines (27 loc) · 752 Bytes

data_engineer.md

File metadata and controls

36 lines (27 loc) · 752 Bytes

Roadmap

data-engineer-roadmap

Skills

Just some of the skills Data Engineering professionals use. One may not need to know all of them, but one technology from each area would make you well-rounded.

Software Engineering / Application Development

  • Python
  • Java
  • Scala

Databases / Data Stores

  • SQL / Relational
  • NoSQL
  • Graph (not critically important)

Analysis / Query

  • SQL

Parallel Processing / Distributed Computing

  • Spark
  • Pandas / Dask

Cloud

  • AWS: S3, Lambda, DynamoDB, Kinesis, Batch ...
  • Serverless frameworks, eg. AWS CDK, serverless, chalice

Workflow Management / Job Orchestration

  • Airflow
  • Luigi

MLOps

  • Docker
  • Kubernetes