Confidence Intervals

"+2.1%" is a guess. "+2.1%, somewhere between +0.4% and +3.8%" is a result you can make a decision with.

Experimentation · Intermediate · ~5 min

In plain English

A weather forecast of '18°C' versus '15 to 21°C'. The second tells you whether to pack a jacket.

Why it's worth your time

It shows direction, magnitude and precision at once — everything the p-value throws away.

If you remember three things

Read it against your break-even threshold, not against zero
Width shrinks with √n: halving it costs 4× the users
CUPED narrows intervals substantially at zero extra traffic

Overview

A point estimate hides its own uncertainty, and the uncertainty is usually what determines the decision. A 95% confidence interval gives the range of effects consistent with the data: if the whole interval sits above your break-even threshold, ship; if it spans zero, you cannot distinguish the effect from nothing; if it is enormous, the test was too small to conclude anything either way. Reporting intervals instead of point estimates changes conversations, because it makes 'we don't know yet' a visible, respectable answer rather than something that has to be argued for.

In an interview

A 95% confidence interval is the range of effect sizes consistent with the observed data; across many repeated experiments, 95% of such intervals contain the true effect. Its width is driven by sample size and variance. Read it against your decision threshold rather than against zero — an interval entirely above break-even is a ship decision, and a wide interval means the test was underpowered.

Production defaults

Always: interval on the DIFFERENCE between arms, not one per arm
Variance reduction: CUPED with a pre-period covariate before asking for more traffic
Ratios & percentiles: bootstrap; the normal approximation doesn't hold

What breaks

'No effect' with an interval of [−5%, +8%] — That's no power, not no effect. Say what you could have detected.
Intervals overlap so 'no difference' — Overlapping per-arm intervals don't imply that. Build the interval on the difference.

Watch it explained

Confidence Interval In Statistics | Confidence Interval Explained With Example | Simplilearn — Simplilearn, 5:39