All concepts

Ensemble Methods

Combine many models so their errors cancel or specialize.

Classical ML · Intermediate · ~8 min

In plain English

A panel of judges beats any single judge, but only if they disagree for different reasons. Identical judges are just one judge in an expensive suit.

Why it's worth your time

Nearly every winning production model is an ensemble of some kind — knowing which type fixes which problem is the actual skill.

If you remember three things

  • Bagging cuts variance; boosting cuts bias; stacking learns the blend
  • Diversity among members is what makes it work
  • You pay in compute, memory and interpretability

Overview

Combines multiple models so their individual errors cancel or specialize, beating any single member. Bagging averages independent models to cut variance, boosting trains models sequentially to cut bias, and stacking learns a meta-model to blend predictions.

How it works

  1. Start: Base Models Train multiple weak or diverse learners: trees, linear models, or specialized predictors.
  2. Base Models -> Bagging Average models trained on resampled data to reduce variance, as in random forests.
  3. Bagging -> Boosting Train models sequentially so each new learner corrects the previous errors.
  4. Boosting -> Stacking A meta-model learns how to combine base model predictions.
  5. Stacking -> Robust Output Good ensembles often outperform any single member while smoothing failure modes.

In an interview

Ensembles aggregate many models for a more robust prediction. Bagging (e.g. random forests) averages models on bootstrap samples to reduce variance; boosting (e.g. gradient boosting) trains learners sequentially so each corrects the last, reducing bias; stacking trains a meta-model on base predictions. Diversity among members is what makes the combination work.

Production defaults

High variance problem
bagging / random forest
High bias problem
boosting (XGBoost, LightGBM)
Stacking
meta-learner on OUT-OF-FOLD predictions only — in-fold predictions leak and the blend memorizes

What breaks

  • Ensemble no better than its best member — Members are too similar. Vary model family, features, or seeds — correlated errors don't cancel.
  • Stacked model great in CV, bad live — Almost always in-fold leakage in the meta-features. Regenerate them out-of-fold.

Watch it explained

Tutorial 42 - Ensemble: What is Bagging (Bootstrap Aggregation)? — Krish Naik, 6:27

Related