All concepts

Warehouse Cost & Performance

The bill is bytes scanned times how often you scan them — and almost every surprise on it is a dashboard refreshing every five minutes over five years of data.

Data Platform in Production · Intermediate · ~5 min

In plain English

A taxi meter that runs on distance. Most of the bill turns out to be a car circling the block every five minutes with nobody in it.

Why it's worth your time

The biggest line item is usually a dashboard on auto-refresh that nobody has opened since March.

If you remember three things

  • Cost = bytes scanned × frequency; attack both
  • Attribute spend per query, user and dashboard before optimising anything
  • The top ten queries are usually most of the bill

Overview

Modern warehouses charge for compute in one of two shapes: per byte scanned, or per second of a running cluster. Either way the lever is the same — read less, less often. Partition pruning and column projection cut bytes per query. Incremental models cut how much a rebuild touches. Materialising a repeatedly-computed aggregate cuts the repetition. And the single largest recurring cost in most platforms isn't the pipeline at all, it is BI: a dashboard on a five-minute auto-refresh that nobody has opened since March, scanning the full history every time.

In an interview

Cost is bytes scanned times frequency. Cut bytes with partition pruning, column projection and clustering; cut frequency with incremental models, materialised aggregates and result caching. Then attribute spend per query, per user and per dashboard — the top ten queries are usually most of the bill, and auto-refreshing dashboards are usually in that ten.

Production defaults

Guardrails
required partition filter on huge tables plus a per-query byte cap
Refresh
move nightly full rebuilds to incremental with an explicit lookback
Compute
auto-suspend idle warehouses; size per workload, not per platform

What breaks

  • Bill doubled with no new pipelines — A BI service account. Check query history grouped by user before anything else.
  • Optimised the pipeline, bill unchanged — The pipeline wasn't the cost. Attribute first, tune second.

Watch it explained

BigQuery Cost Optimization: Select Queries — Google Cloud Tech, 4:49

Related