All concepts

Lineage & Impact Analysis

Two questions, one graph: where did this number come from, and what breaks if I change this column?

Data Platform in Production · Intermediate · ~5 min

In plain English

A family tree for numbers. Point at any figure and trace it back to where it came from, or forward to everyone who'd notice if you changed it.

Why it's worth your time

It converts 'we changed a column and something broke somewhere' into a list of owners to notify before you ship.

If you remember three things

  • Derive it from SQL and orchestration metadata; never draw it by hand
  • Column-level is what makes impact analysis precise enough to act on
  • It's an index to query from CI and alerting, not a diagram to admire

Overview

Lineage is the dependency graph of your data — which tables feed which, and ideally which columns feed which. It answers two questions that come up constantly and are miserable without it. Upstream: an executive asks where a figure comes from, and you can trace it back through the marts to the raw source rather than reading five hundred lines of SQL. Downstream: you want to change a column, and you can list every model, dashboard and ML feature that would break. The only lineage worth having is derived — parsed from the SQL and the orchestrator — because hand-maintained diagrams are wrong within a month.

In an interview

Lineage is the derived graph of table and column dependencies. Upstream it answers 'where did this number come from'; downstream it answers 'what breaks if I change this'. Parse it from SQL and orchestration metadata rather than drawing it, because manual diagrams rot. Column-level lineage is what makes impact analysis precise enough to act on.

Production defaults

Scope
extend it into the BI tool — that's where the real consumers are
CI use
a PR that changes a column prints who it affects
Cleanup
lineage + query logs finds the assets nobody has read in a year

What breaks

  • Impact analysis flags 40 dashboards for a one-column change — Table-level lineage. Column-level narrows it to the few that read that column.
  • Downstream owners still hear from stakeholders first — Lineage exists but isn't wired into alerting. Notify consumers automatically on upstream failure.

Watch it explained

Trace Your Data's Journey From Source to Dashboard with Atlan's End-to-End Column Level Data Lineage — Atlan, 2:11

Related