All concepts

Schema Evolution & Data Contracts

Upstream renamed a column on a Tuesday and nobody told you — a contract is how that becomes their build failure instead of your incident.

Pipelines & Orchestration · Intermediate · ~5 min

In plain English

A supplier changing box sizes without telling you. A contract is the agreement that they check with you first — enforced at their end, not yours.

Why it's worth your time

Most pipeline breakage isn't a bug in your code; it's an unannounced change upstream, and only the producer can prevent it.

If you remember three things

  • Adding an optional field with a default is safe; renaming is delete plus add
  • Enforce in the producer's CI or it's a wiki page
  • SELECT * in a model is a standing invitation for silent breakage

Overview

Most pipeline breakage isn't a bug in the pipeline; it's an unannounced change upstream. A field is renamed, a type widens, an enum gains a value, an optional field starts arriving null. Schema evolution is the set of rules for which changes are safe: adding an optional field is backward compatible, removing or renaming one is not. A data contract makes those rules enforceable — the producing team declares the schema and its guarantees, CI checks proposed changes against it, and a breaking change fails the producer's build rather than silently poisoning the consumer's tables the next morning.

In an interview

Schema evolution rules say which changes consumers can absorb: adding an optional field is backward compatible; renaming, removing, or narrowing a type is not. A data contract encodes the schema plus its guarantees — ownership, freshness, semantics — and enforces it in the producer's CI, so breaking changes fail their build instead of your pipeline.

Production defaults

Registry
full compatibility enforced, checked in the producer's pipeline
Renames
add, dual-write, migrate consumers, then drop after a deprecation window
Consumers
explicit column lists, and alert when a source gains or loses a column

What breaks

  • Downstream types changed overnight — SELECT * absorbed an upstream column addition. Pin explicit columns.
  • New enum value became 'other' — A default mapping turned a visible break into wrong data. Fail loudly on unknown values.

Watch it explained

Schema Evolution with Zero Down Time | Designing Event-Driven Microservices — Confluent, 7:28

Related