All concepts

Correlation vs Causation

Users who enable notifications retain twice as well — so should you force notifications on, or are you just describing people who already liked the product?

Analytics Foundations · Beginner · ~5 min

In plain English

People carrying umbrellas and wet pavements go together. Confiscating the umbrellas will not dry the pavement.

Why it's worth your time

Only a causal claim justifies an action, and feature-adoption metrics are confounded almost by construction.

If you remember three things

  • Four explanations: A→B, B→A, a common cause, or chance
  • Controls only adjust for confounders you measured
  • Say 'associated with' unless you randomised

Overview

Two variables moving together admits four explanations: A causes B, B causes A, something else causes both, or it is chance. Analytics work overwhelmingly produces the third — a confounder — because the people who adopt a feature differ from the people who don't in every way that also predicts retention. The reason this matters commercially is that only a causal claim justifies an action. If notifications cause retention, turning them on helps; if engaged users simply enable notifications, forcing them on annoys everyone and moves nothing.

In an interview

Correlation has four possible causes: A→B, B→A, a confounder driving both, or coincidence. Feature-adoption metrics are almost always confounded, because adopters self-select. Only a randomised experiment cleanly separates them; when you can't run one, quasi-experimental designs — difference-in-differences, regression discontinuity, instrumental variables — get you closer, with assumptions you must state.

Production defaults

Language
'associated with' by default; 'causes' only after an experiment
Before acting
cost the smallest experiment that would test it — usually a small holdout
Quasi-experimental
if using difference-in-differences, plot and show the pre-period trends

What breaks

  • Forced the feature on everyone and the metric didn't move — The correlation was selection. Adopters were already engaged; the feature wasn't doing the work.
  • 'We controlled for engagement' — That covers one confounder. List the ones you couldn't measure — that list is the limit of the claim.

Watch it explained

Correlation vs causation explained by Dr Nic with examples — Dr Nic's Maths and Stats, 4:28

Related