Sampling & Selection Bias

Your survey says 92% of users love the redesign. It was shown in-app, to people still using the app.

Analytics Foundations · Beginner · ~5 min

In plain English

Asking people in the queue whether the queue is too long. Everyone who gave up and left isn't there to answer.

Why it's worth your time

More data doesn't fix it — a million biased responses give you a very precise estimate of the wrong thing.

If you remember three things

Survivorship, non-response, self-selection, truncation — the four recurring shapes
The response rate is often the strongest finding in the survey
Incomplete cohorts must be blank, never zero

Overview

Every number you compute describes the rows you have, and the question is always whether those rows represent the population you mean. Selection bias is the systematic gap between the two, and it does not go away with more data — a million responses from a biased sample give you a precise estimate of the wrong thing. The classic shapes recur endlessly: survivorship (you only measure who stayed), non-response (unhappy users don't answer), self-selection (opt-in beta users are enthusiasts), and truncation (the analysis window quietly excludes the slow cases).

In an interview

Selection bias is a systematic difference between who you measured and who you meant. More data doesn't fix it — it just makes the wrong estimate tighter. Watch for survivorship (only measuring who stayed), non-response (who answered differs from who didn't), self-selection (opt-in samples are enthusiasts), and truncation (the window excludes slow outcomes).

Production defaults

Habit: name the target population, then list every reason a row could be missing
Surveys: always publish the response rate beside the result
When unfixable: report a best-case/worst-case bound instead of a point estimate

What breaks

Satisfaction is high and churn is rising — You surveyed inside the product, so only survivors answered. Sample all accounts, including cancelled ones.
Newest cohort's conversion looks catastrophic — Its window hasn't elapsed. Blank the cell rather than counting non-conversions.

Watch it explained

What is Selection Bias | Explained in 2 min — Productivity Guy, 2:18