All concepts

Guardrail Metrics

Every metric can be moved by making the product worse — a guardrail is the number that catches you doing it.

Metrics & KPIs · Intermediate · ~4 min

In plain English

A speedometer next to the stopwatch. You're trying to get there faster, but not by driving through a school zone.

Why it's worth your time

Anything you optimise hard enough will be optimised by making the product worse, unless something is watching.

If you remember three things

  • Chosen by asking how you'd cheat the primary metric
  • Tested for non-inferiority, not for significance
  • Tolerance agreed before the results exist

Overview

Optimising a single metric reliably finds the cheapest way to move it, which is often not the way you intended. More aggressive notifications raise engagement and raise uninstalls. Auto-renewing subscriptions raise retention and raise refund requests. A guardrail metric is a second measurement, chosen specifically because it would degrade if you moved the primary metric the wrong way, and monitored with a different question: not 'did it improve' but 'did it get meaningfully worse'.

In an interview

A guardrail is a metric you don't expect to improve but refuse to damage — latency, unsubscribes, refund rate, support contacts, crash rate. It is checked for non-inferiority rather than significance: the question is whether the confidence interval rules out a harmful drop, not whether it moved. Pair one with every headline metric and every experiment.

Production defaults

Universal set
latency, error rate, crashes, total revenue — on every experiment
Blocking
a tripped guardrail blocks launch by default; overriding is a recorded decision
Power
state whether the test could have detected a meaningful guardrail change

What breaks

  • Guardrail flat, so it's safe — Probably underpowered for it. Say what change you could have detected.
  • Tolerance debated after results — Pre-register it. Post-hoc tolerance is infinitely flexible.

Watch it explained

How Online evaluation metrics and guardrail metrics actually Works in 10 Minutes — How Tech Actually Works, 8:52

Related