All concepts

Orchestration & DAGs

Declare which task depends on which and let the scheduler decide what can run now, what must wait, and what to retry.

Pipelines & Orchestration · Beginner · ~6 min

In plain English

A kitchen pass: the sauce can't go out before it's made, two cooks work in parallel on independent dishes, and if one burns you redo that dish, not the service.

Why it's worth your time

A pipeline that can only run 'now' can never be repaired — and every pipeline eventually needs repairing.

If you remember three things

  • One run per logical interval is what makes a day re-runnable
  • The task takes the interval as a parameter; never call now()
  • A task should be the smallest thing you'd want to retry alone

Overview

An orchestrator turns a pile of scripts into a dependency graph. Tasks are nodes, dependencies are edges, and because the graph is acyclic there is always a valid order. The scheduler then handles everything cron can't: it runs independent branches in parallel, holds a task until its inputs exist, retries transient failures with backoff, and — crucially — keeps one run per logical interval, so a failed day can be re-run on its own without disturbing the others. That last property is what makes the DAG worth the operational weight: a pipeline that can only run 'now' can never be repaired.

In an interview

An orchestrator like Airflow, Dagster or Prefect models the pipeline as a directed acyclic graph of tasks. It runs independent branches in parallel, blocks a task until its upstream succeeds, retries with backoff, and keeps one isolated run per scheduled interval so any single day can be re-run. Cron gives you none of that — it gives you a time and a hope.

Production defaults

Concurrency
set pools and max-active-runs before you need a backfill
Alerting
alert on SLA misses, not only on failures — a hung task fires nothing
Data passing
through object storage, never through orchestrator metadata

What breaks

  • A re-run produces different output — Something read the wall clock. Parameterise every task by the interval.
  • Backfill takes down the source database — No concurrency cap. Set pools first, then backfill newest-first.

Watch it explained

What is Apache Airflow? — codebasics, 8:41

Related