All concepts

Partitioning & Clustering

Put the data in folders named after the thing you filter on, so most queries never open most folders.

DE Foundations · Intermediate · ~5 min

In plain English

Filing invoices in one folder per month. Asked for August, you open one folder. Inside it, keeping them in customer order means you can find one without reading all of them.

Why it's worth your time

On a pay-per-byte warehouse, one partition filter is routinely a 99% cost reduction — and one careless function call removes it.

If you remember three things

  • Partition on the low-cardinality column you almost always filter on
  • Cluster on the high-cardinality ones you filter on sometimes
  • A function around the partition column usually disables pruning

Overview

Partitioning splits a table into physically separate chunks by the value of a column — almost always a date. A query filtered to one week then opens seven directories instead of five years of them, and the bytes it never reads are bytes you never pay for. Clustering (or Z-ordering, or sorting) works inside those chunks: it orders rows so that the min/max statistics of each file are narrow, letting the engine skip files a partition filter alone couldn't. The two are complementary and both fail the same way — pick a column with too many distinct values and you get a million tiny partitions, which is slower than no partitioning at all.

In an interview

Partitioning physically separates rows by a column value, usually a date, so a filtered query prunes whole directories before reading anything. Clustering sorts rows within each partition so per-file min/max statistics are tight and the engine can skip individual files too. Partition on a low-cardinality column you almost always filter on; cluster on the high-cardinality ones you filter on sometimes.

Production defaults

Granularity
daily on event date; monthly if a day is under ~100 MB
Partition size
aim for ≥ a few hundred MB each
Guardrail
require a partition filter on your biggest tables

What breaks

  • Query scans everything despite a date filter — The predicate wraps the column in a function, or filters event time while you partitioned on ingestion time.
  • Planning takes longer than scanning — Too many partitions. Coarsen the granularity — you likely partitioned on something high-cardinality.

Watch it explained

How does BigQuery store data? — Google Cloud Tech, 8:20

Related