Idempotency & Backfills

Running the same job twice must leave the world exactly as it was after running it once — otherwise every retry corrupts your data.

Pipelines & Orchestration · Intermediate · ~6 min

In plain English

Rewriting a whiteboard column from scratch each time, instead of adding to what's already there. Do it twice and the board looks identical.

Why it's worth your time

Retries are automatic and invisible. Without idempotency every one of them silently doubles a day of data.

If you remember three things

Own the partition and replace it, or MERGE on a deterministic key
The key must come from immutable source fields only
Making your task idempotent doesn't fix the marts built on top of it

Overview

A pipeline task will run more than once. The network drops, the worker is preempted, someone re-runs a day to fix a bug, and the orchestrator retries on its own. Idempotency is the property that makes all of that safe: the same input window produces the same output, no matter how many times it executes. The mechanism is almost always the same — the task owns a partition and replaces it wholesale, rather than appending into a shared table. Get this right and a backfill is boring. Get it wrong and every retry silently doubles a day's revenue, which you will discover a month later when someone notices the numbers.

In an interview

Idempotency means re-running a task leaves the same result as running it once. In practice: partition the output by the run's interval and overwrite that partition atomically, or write with a MERGE keyed on a deterministic business key. Append-only writes are the classic non-idempotent pattern — one retry and the day is counted twice.

Production defaults

Write pattern: compute → temp location → atomic partition swap
Backfill order: newest first, with capped concurrency
Restatement marker: emit one when a partition is rewritten so downstream knows

What breaks

A day's revenue is exactly 2× or 3× — INSERT on retry. Switch to delete-and-replace or MERGE, then re-run the affected partitions.
Backfilled the source, numbers still wrong — Downstream marts weren't refreshed. Cascade the re-run.

Watch it explained

What is Data Pipeline? | Why Is It So Popular? — ByteByteGo, 5:25