All concepts

Windowing & Streaming Aggregation

An unbounded stream has no end, so you can't sum it — you cut it into windows, and how you cut it decides what the number means.

Streaming & CDC · Advanced · ~5 min

In plain English

You can't total an infinite queue, so you count in five-minute slices — or in visits, which start and end when the person does.

Why it's worth your time

The window shape decides both what the number means and whether the job stays inside its memory budget.

If you remember three things

  • Tumbling for published period totals; sliding for alerting; session for visits
  • State ≈ active keys × open windows × accumulator size
  • Aggregate incrementally; never buffer raw events per window

Overview

You cannot aggregate an infinite stream; you can only aggregate finite slices of it. Tumbling windows are fixed, non-overlapping buckets — hourly revenue, one row per hour, every event counted once. Sliding windows overlap — a five-minute count emitted every minute — so each event lands in several windows and the series is smooth enough to alert on. Session windows have no fixed size at all: they group events separated by less than a gap timeout, which is how you measure a user's visit rather than an arbitrary clock interval. Each shape holds state per key per open window, and that state is the thing that decides whether the job stays up.

In an interview

Tumbling windows are fixed and non-overlapping — each event counted once, ideal for reporting periods. Sliding windows overlap, so each event lands in several and the metric moves smoothly, at the cost of more state. Session windows close after a gap of inactivity, which is how you measure a visit. State grows with keys times open windows, so the window shape is a memory decision as much as a semantic one.

Production defaults

Distinct counts
HyperLogLog rather than a set of ids
Session cap
always set a maximum duration so an always-on key can't grow forever
Checkpoints
alarm when duration exceeds half the interval

What breaks

  • Job OOMs after a few hours — Buffered events or unbounded sessions. Switch to incremental accumulators and cap session length.
  • Totals double when summed — Sliding windows overlap. Use tumbling for anything published as a period total.

Watch it explained

(17) Time Windows in Fabric Streaming | Tumbling, Hopping, Sliding, Session & Snapshot | DP-700 Exam — Raghu Veer Tech, 7:20

Related