Pipeline Failure & On-Call

The pipeline will fail. What decides whether that's an inconvenience or an incident is whether the failure stops before it reaches a published table.

Data Platform in Production · Intermediate · ~5 min

In plain English

A kitchen that would rather serve a late dish than a wrong one. The customer waits; nobody gets food poisoning.

Why it's worth your time

Data failures are silent and cumulative — the pipeline keeps serving, and what it serves is wrong.

If you remember three things

Fail closed: gate publication, so stale-and-correct beats fresh-and-wrong
Alert on dataset SLAs, not on task status
Recovery is a parameterised re-run — then cascade it downstream

Overview

Data on-call differs from service on-call in one important way: the damage is usually silent and cumulative rather than immediate and obvious. A web service that falls over stops serving; a pipeline that fails badly keeps serving — wrong numbers, to people who act on them. So the design goal is not merely 'recover quickly', it is 'fail closed': quality gates that block publication, so the consumer sees stale-but-correct data and an SLA breach rather than fresh-and-wrong data and no signal at all.

In an interview

Design pipelines to fail closed: quality gates between producing and publishing, so a bad run leaves yesterday's correct data in place and raises an SLA alert. Retry transient failures automatically with backoff, page only on breached SLAs for owned datasets, and make every task idempotent so recovery is a re-run rather than a repair.

Production defaults

Pattern: write → audit → publish, always in that order
Paging: SLA breaches on owned datasets; task noise goes to a ticket
Restatement: publish a note whenever published numbers change

What breaks

Fixed the source, dashboards still wrong — The downstream cascade wasn't triggered. Automate it as part of recovery.
Same pipeline fails weekly — That's a contract problem, not bad luck. Handle the upstream defensively or agree a contract.

Watch it explained

This Airflow Pipeline Was Broken. Here’s How an AI Agent Helped Me Fix It. — Altimate: AI Teammates for Data Engineering, 6:37