Simpson's Paradox

Version B wins on mobile, wins on desktop, and loses overall. Both facts are true.

Analytics Foundations · Intermediate · ~5 min

In plain English

A player with a better shooting percentage than a rival in both halves can still finish the game with a worse overall percentage — if they took most of their shots in the harder half.

Why it's worth your time

It's not a curiosity; it's the routine result of comparing groups whose composition differs, which is most non-randomised comparisons.

If you remember three things

Needs two things: different base rates per segment, and different segment mixes per arm
The aggregate is measuring composition, not performance
Standardise to a common mix to compare like with like

Overview

Simpson's paradox is what happens when a comparison of two aggregates is dominated by the mix of subgroups rather than by performance within them. If B was shown mostly to mobile traffic, and mobile converts worse than desktop for reasons that have nothing to do with B, then B's overall number is dragged down by where its users came from — even though it beat A in every single segment. It is not a curiosity; it is a routine failure in any analysis where the groups being compared have different compositions, which includes almost every non-randomised comparison anyone runs.

In an interview

Simpson's paradox is a reversal: a treatment wins in every subgroup yet loses in the total, because the subgroups have different base rates and the treatment's traffic mix is weighted toward the worse-performing ones. The aggregate is measuring composition, not performance. Segment first, and if the segment mixes differ, the aggregate comparison is not valid.

Production defaults

Habit: break every headline comparison down by platform, market and user type
Reporting: if mixes differ, publish standardised rates and name the weighting
Experiments: check the arm split within each major segment, not just overall

What breaks

Variant wins every segment and loses overall — Unequal traffic mix. Report the segment-level result and standardise for a single headline figure.
Segmenting until the answer changes — That's p-hacking with dimensions. Pre-declare the segments you'll cut by.

Watch it explained

Simpsons Paradox [Explained] AP Statistics — Michael Porinchak - AP Statistics & AP Precalculus, 6:34