Delivery Semantics

At-most-once loses records, at-least-once duplicates them, and exactly-once is at-least-once plus somewhere to deduplicate.

Streaming & CDC · Advanced · ~5 min

In plain English

Post a letter and it may go missing; post it twice and it may arrive twice. You can't stop the duplicate — you can make the recipient ignore the second copy.

Why it's worth your time

'Exactly-once' is the most misunderstood guarantee in streaming, and using it correctly means knowing it's about the sink, not the wire.

If you remember three things

At-most-once loses, at-least-once duplicates, exactly-once hides duplicates
Idempotent writes on a deterministic key cover most real cases
Transactional sinks cost checkpoint-interval latency

Overview

A message crosses a network that can drop anything, to a consumer that can crash at any moment. If you acknowledge before processing, a crash loses the record — at-most-once. If you acknowledge after, a crash between processing and acknowledging replays it — at-least-once. There is no third network. So 'exactly-once' never means the message is delivered once; it means duplicates are made invisible, either by an idempotent write keyed on something stable, or by a transaction that commits the output and the offset together so they can't disagree. Knowing that reframing is the difference between using the guarantee correctly and trusting a checkbox.

In an interview

At-most-once acknowledges before processing and loses data on a crash. At-least-once acknowledges after and replays on a crash, so duplicates are guaranteed. Exactly-once isn't a delivery property — it's end-to-end effect, achieved by committing the output and the consumer offset in one transaction, or by making the write idempotent on a deterministic key. Pick at-least-once plus idempotent writes unless you can prove you need more.

Production defaults

Default choice: at-least-once plus an idempotent MERGE
Dedup key: derived from immutable source fields only; test that reprocessing reproduces it
Monitoring: measure the duplicate rate; an unmeasured guarantee isn't one

What breaks

Duplicates despite exactly-once enabled — The sink isn't transactional, or the key includes a timestamp. Check what the key is actually built from.
End-to-end latency stuck at 60s — Transactional sink making output visible only at checkpoints. Lower the interval or drop to idempotent writes.

Watch it explained

013 Delivery Semantics At least once at most once exactly once3m 58s — Prateek Ashtikar, 3:58