How does ensemble learning combine predictive models?
Ensemble learning combines multiple predictive models to produce one aggregate prediction. It works best when members are individually reasonable but make different errors; combining uncorrelated errors reduces overall error. The main trade-off is higher training cost, memory use, and lower interpretability for stronger predictive performance.
Read full answerHide full answer
Applies tomachine learning concepts; Python 3.12 with scikit-learn 1.5+, XGBoost 2.0+, LightGBM 4.x, and CatBoost 1.2.x
How It Works
An ensemble combines member outputs by averaging, voting, or fitting a meta-model. For equally weighted models whose errors have variance σ² and common pairwise correlation ρ:
ensemble_variance = ρ × σ² / M
Here, M is the number of models, σ² is each model’s error variance, and ρ measures how strongly their errors move together. When errors are uncorrelated, averaging reduces variance; highly correlated errors make additional models less useful. Bagging mainly lowers variance, whereas boosting mainly lowers bias.
Main Methods
| Method | How it works | Strengths and weaknesses | Typical scale |
|---|---|---|---|
| Bagging | BaggingClassifier/BaggingRegressor trains one learner per bootstrap sample. Regression outputs are averaged; classification uses hard voting by default. |
Parallel and variance-reducing; correlated members add little | About M learner fits and M predictions |
| Random forests | RandomForestClassifier/RandomForestRegressor adds random feature subsets at each split. |
Strong general-purpose baseline; larger and less interpretable than one tree | Cost grows with trees, depth, and features considered per split |
| Boosting | AdaBoostClassifier/AdaBoostRegressor reweights errors; gradient boosting fits residuals or negative gradients, as in XGBoost, LightGBM, CatBoost, and HistGradientBoostingClassifier. |
Reduces bias and captures complex patterns; can overfit noisy labels | Roughly M sequential learner fits |
| Voting | VotingClassifier combines hard labels or averaged probabilities. |
Simple baseline; cannot learn how much to trust each member | About M member predictions |
| Stacking | StackingClassifier trains a meta-model on out-of-fold base predictions. |
Flexible nonlinear combination; leaks if predictions are not out-of-fold | Base fits for each fold plus final fits |
Hyperparameters and Practical Guidance
-
Number of estimators: More estimators usually stabilize bagging and boosting but increase training and inference cost.
-
Learning rate: A smaller boosting learning rate makes each tree contribute less, often requiring more estimators and early stopping.
-
Maximum depth: Shallower trees reduce overfitting and computation; deeper trees capture interactions but can memorize noise.
-
Subsampling: Row or feature sampling creates diversity and can improve generalization.
-
Early stopping:
HistGradientBoostingClassifiercan stop automatically when validation performance stops improving. -
Practical choices: Trees generally need no scaling, but missing-value support varies by implementation. For imbalanced classes, use precision, recall, F1, ROC-AUC, or PR-AUC as appropriate because accuracy can mislead. SHAP can explain complex ensembles without making them fully transparent. CPU training is normally adequate for moderate ensembles; GPUs mainly help large boosting workloads.
Code Example
This compares one tree, a random forest, histogram gradient boosting, and stacking. The test set remains separate, while stacking generates out-of-fold predictions internally.
from sklearn.datasets import make_classification
from sklearn.ensemble import (
HistGradientBoostingClassifier,
RandomForestClassifier,
StackingClassifier,
)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import (
StratifiedKFold,
cross_val_score,
train_test_split,
)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(
n_samples=1_000,
n_features=20,
n_informative=8,
n_redundant=4,
random_state=42,
)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.20, stratify=y, random_state=42
)
models = {
"Decision tree": DecisionTreeClassifier(
max_depth=5, random_state=42
),
"Random forest": RandomForestClassifier(
n_estimators=200, random_state=42, n_jobs=1
),
"Histogram gradient boosting": HistGradientBoostingClassifier(
learning_rate=0.1,
max_iter=200,
early_stopping=True,
random_state=42,
),
"Stacking": StackingClassifier(
estimators=[
("tree", DecisionTreeClassifier(
max_depth=5, random_state=42
)),
("logistic", make_pipeline(
StandardScaler(),
LogisticRegression(max_iter=1_000, random_state=42),
)),
],
final_estimator=LogisticRegression(
max_iter=1_000, random_state=42
),
cv=5,
passthrough=False,
n_jobs=1,
),
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for name, model in models.items():
scores = cross_val_score(
model, X_train, y_train, cv=cv, scoring="accuracy", n_jobs=-1
)
model.fit(X_train, y_train)
test_accuracy = accuracy_score(y_test, model.predict(X_test))
print(
f"{name}: CV accuracy = {scores.mean():.3f} "
f"+/- {scores.std():.3f}; "
f"test accuracy = {test_accuracy:.3f}"
)
The scores depend on the dataset and library version, so the code prints the measured results rather than claiming fixed values.
Common Pitfalls
-
Correlated base models: Several similar learners may make nearly identical mistakes.
-
Leaky stacking: Training the meta-model on predictions from models fitted on the same examples produces overly optimistic results; use out-of-fold predictions.
-
Boosting on noisy labels: Too many rounds can fit random noise; tune the learning rate, depth, estimator count, and early stopping.
-
Invalid validation: Repeatedly tuning against the test set invalidates its final-evaluation role.
-
Interview follow-ups: Be ready to explain variance versus bias, out-of-fold stacking, early stopping, and why correlated models add little.