All concepts

Sample Ratio Mismatch

You asked for a 50/50 split and got 50.4/49.6. On two million users that is not rounding — it means something is filtering your users, and the result is void.

Experimentation · Advanced · ~4 min

In plain English

You dealt two equal piles and one has 200 more cards. You don't argue about the cards — you find out what's eating them.

Why it's worth your time

It's a direct signal that randomisation broke, which invalidates the causal claim the whole test rests on.

If you remember three things

  • Chi-square on observed vs intended split, threshold p < 0.001
  • Always a mechanical cause: redirect drop-off, SDK failure, filtering, a dropping join
  • Never reweight to fix it — the missing users are missing non-randomly

Overview

Sample ratio mismatch is a statistically significant deviation between the observed arm split and the intended one. It matters far beyond the split itself: randomisation is the assumption every causal claim rests on, so if the split is broken, something is systematically removing or misassigning users, and whatever that something is has almost certainly also biased the outcome. SRM is therefore the first check to run and a hard gate — a test with SRM should not be interpreted at all, no matter how good the headline number looks.

In an interview

SRM is a significant difference between the actual and intended arm split, tested with a chi-square goodness-of-fit test. It signals broken randomisation — a redirect that drops users, an SDK that fails on one variant, bot filtering that hits arms unequally, or a logging gap. Any test showing SRM is invalid and must be debugged, not interpreted.

Production defaults

Automation
run the check on every experiment and gate the results page on it
Debugging
check per segment and per day; the segment names the bug
Assignment
server-side beats redirect-based, which is the commonest cause

What breaks

  • SRM only on Android — The treatment SDK is failing on those devices, so those users never log. Fix and restart.
  • SRM appears only in trigger-based analysis — The treatment changes who reaches the trigger. Analyse from assignment, not from trigger.

Watch it explained

Understanding SRM (Sample Ratio Mismatch) in A/B Testing [CRO / Experimentation] — Lucia from The Initial, 6:45

Related