All concepts

dbt & the Transformation Layer

Every table is a SELECT statement in version control, and the tool works out the order to run them in.

Pipelines & Orchestration · Intermediate · ~6 min

In plain English

Every table is a recipe card that names the other cards it needs. Lay them out and the order to cook in is obvious without anyone writing it down.

Why it's worth your time

It's what turned warehouse SQL from a pile of scheduled scripts into something with version control, tests, CI and lineage.

If you remember three things

  • ref() infers the DAG, so lineage can't drift from the code
  • Materialisation is the main performance lever: view, table, or incremental
  • staging → intermediate → marts keeps business logic in one layer

Overview

dbt made transformation look like software. Each model is a file containing one SELECT; referencing another model with ref() both inserts the right table name and declares a dependency, so the DAG is inferred from the SQL rather than maintained by hand. From that one idea everything else follows: the tool can build models in dependency order, materialise each as a view, a table or an incremental table, run tests as queries that must return zero rows, and generate lineage documentation that is correct because it is derived. The layered convention — staging that renames and casts, intermediate that joins, marts that the business queries — is what keeps a thousand models navigable.

In an interview

dbt turns each warehouse table into a version-controlled SELECT. ref() between models infers the dependency graph, so dbt knows the build order and the lineage without anyone drawing it. It adds materialisation strategies, tests that must return zero rows, and generated docs. The convention is staging → intermediate → marts, so business logic lives in one layer instead of thirty dashboards.

Production defaults

CI
dbt build on every pull request against a scratch schema
Incremental
always with a unique_key and a lookback window for late data
Staging
one-to-one with sources; rename and cast only, no joins

What breaks

  • Incremental model has duplicates — No unique_key, so re-processed rows appended. Add one and full-refresh once.
  • Lineage graph is missing edges — A hard-coded table name instead of ref(). It also breaks the dev/prod swap.

Watch it explained

Introduction to DBT (Data Build Tool) | ETL Vs ELT — SleekData, 4:29

Related