Batch vs Streaming

Wait and process a pile every hour, or process each record as it lands — the difference is what you're willing to pay for freshness.

DE Foundations · Beginner · ~5 min

In plain English

Emptying the postbox once an hour, or standing at the slot catching each letter. The second is faster and much harder to do without dropping something.

Why it's worth your time

Most teams pay streaming's operational cost for freshness no decision actually requires.

If you remember three things

Batch is a function of a window — that's what makes it replayable
Streaming buys seconds and costs late data, durable state, and no easy re-run
Choose by the latency the decision needs, written as a number

Overview

Batch collects records into a window — an hour, a day — and processes them together. It is simple to reason about, trivially replayable, and cheap, because you amortise startup and read whole files at once. Streaming processes each record as it arrives, keeping running state, so results are seconds old instead of hours. The cost is that everything gets harder: you handle out-of-order and late data, you keep state that must survive restarts, you can't just re-run yesterday to fix a bug, and you carry an always-on cluster. The right question in an interview and in design review is never 'batch or streaming', it is 'what decision does this data drive, and how stale can it be before that decision changes'.

In an interview

Batch groups records and processes them on a schedule: simple, cheap, easy to replay, minutes-to-hours stale. Streaming processes records as they arrive with running state: seconds fresh, but you inherit late and out-of-order events, durable state, checkpointing, and no simple re-run. Choose by the latency the decision actually needs — most dashboards don't need seconds, and fraud checks can't wait for hours.

Production defaults

Start batch: move only the tables that fail the stated SLA
One definition: if you run both paths, derive them from one implementation of the logic
Source of truth: the daily batch recomputation; the stream is the fast approximation

What breaks

Streaming and batch totals disagree — Late events. Reconcile the stream against the batch nightly and treat batch as canonical.
A streaming bug fix is a week of work — That's the real cost of the choice. Check whether hourly batch would have met the SLA.

Watch it explained

What is Stream Processing? | Batch vs Stream Processing | Data Pipelines | Real-Time Data Processing — BI Insights Inc, 6:19