CareerOS
Sign inStart for free
Learning Paths

Data Engineering

Move data reliably from where it is made to where it is used: SQL and modelling, pipelines and orchestration, warehouses, batch and streaming.

Finish this and you can build a tested, scheduled pipeline from raw files to a modelled warehouse that an analyst trusts.

Starts at
Beginner
Topics
4
Projects
4
Of material
79 h
  • SQL
  • Data modelling
  • Python
  • Airflow
  • dbt
  • Spark
Level
Format
  1. 01

    SQL and relational databases

    Beginner

    Queries, joins, window functions, indexes and transactions: the language every data job is built on.

    Build this

    Answer ten business questions in SQL

    Load a real public dataset into Postgres and answer questions a manager would ask.

    1. Load and constrain the data with keys and types, and note what does not fit.
    2. Write ten questions first, then the queries, using at least three window functions.
    3. Make two slow queries fast with indexes and show the plans before and after.

    Done when: A repository with the schema, the ten queries and their answers, and the two query plans.

    NYC TLC trip records
  2. 02

    Data modelling for analytics

    Intermediate

    Facts, dimensions and slowly changing data: models that make the right question easy to ask.

    Build this

    A tested star schema in dbt

    Model the dataset from the SQL project for analysts.

    1. Decide the grain of each fact table and write it down.
    2. Build staging, dimension and fact models in dbt.
    3. Add tests for keys, nulls and accepted values, and fix what they find.
    4. Publish the docs site with the lineage graph.

    Done when: A dbt project with passing tests and a published documentation site.

  3. 03

    Pipelines and orchestration

    Intermediate

    Ingest, transform and load on a schedule, with retries, backfills and alerts when something breaks.

    Build this

    A pipeline that survives a bad day

    Schedule the whole flow from raw files to your dbt models.

    1. Ingest monthly files idempotently, so a rerun never duplicates rows.
    2. Orchestrate ingest, dbt run and dbt test as one DAG with retries.
    3. Backfill a year and time it.
    4. Break it on purpose with a malformed file and show the alert and recovery.

    Done when: The DAG, the backfill timing, and a write-up of the failure drill.

  4. 04

    Batch at scale, and streaming

    Advanced

    When data no longer fits on one machine, and when it never stops arriving.

    Build this

    Live aggregates from a stream

    Compute a rolling metric from an event stream.

    1. Produce events by replaying the trip data into Kafka at speed.
    2. Consume and aggregate in five-minute windows.
    3. Handle late events and explain your choice of watermark.
    4. Compare with the batch answer and explain any difference.

    Done when: The streaming job, a chart of the live metric, and the batch comparison.

Not sure this is your direction?

A three-minute counselling sample is free.

Try counselling
Data Engineering · Learning Paths · CareerOS