All concepts
K-Nearest Neighbors
Predict by asking the closest labeled examples to vote.
Classical ML · Beginner · ~8 min
In plain English
To guess whether a new restaurant is good, look at the five most similar restaurants you already know and go with the majority.
Why it's worth your time
It's the clearest illustration of what 'similar' means numerically — and the mental model behind every vector search system you'll build later.
If you remember three things
- No training at all; all the work happens at query time
- Feature scaling is mandatory — distance mixes all columns
- Small k is noisy, large k blurs real boundaries
Overview
A lazy, non-parametric method that classifies or regresses a new point from its k nearest labeled examples — majority vote for classification, average for regression. There's no training beyond storing the data; all work happens at query time.
How it works
- 40 stored examples KNN has no training step — it just stores every labeled point. Two classes sit in a scaled feature plane.
- A query arrives A new, unlabeled point appears. We predict its class from whoever it lands nearest to.
- Euclidean distance Measure the straight-line distance from the query to every stored point.
- The k nearest Keep only the k closest points. Everything else is ignored — the decision is purely local.
- Vote with k The k neighbors vote; the majority class wins (regression would average their targets instead).
- The prediction The query is labeled by that majority vote. Ties break by distance or a smaller k.
- k is the smoothing knob Small k = jagged, noise-sensitive boundary; large k = smoother but blurs real structure. Turn the k dial to feel it.
In an interview
K-nearest neighbors predicts from the k closest training points under some distance metric: majority label for classification, mean target for regression. It's non-parametric and stores all data, so it captures local structure well but query cost grows with dataset size, and it requires feature scaling for distances to be meaningful.
Production defaults
- k
- odd, around √n as a starting point, tuned on validation
- Distance
- Euclidean on scaled features; cosine when magnitude is meaningless (text embeddings)
- At scale
- past ~100k points use an ANN index (HNSW/IVF) — exact KNN is O(n) per query
What breaks
- Accuracy collapses on wide data — Curse of dimensionality — distances concentrate. Reduce dimensions first, or use a model that selects features.
- Predictions are slow in production — You're doing a linear scan. This is exactly the problem vector databases exist to solve.