What is a Random Forest and how does it work?
A Random Forest is an ensemble of many decision trees, each trained on a bootstrap sample of the data and allowed to consider only a random subset of features at each split, whose predictions are combined by majority vote (classification) or averaging (regression). The two sources of randomness make the trees different from one another, so their errors partly cancel: the forest keeps the low bias of deep trees while greatly reducing their variance. It is accurate with little tuning, handles non-linear relationships and interactions, and needs no feature scaling.
Read full answerHide full answer
Applies toPython 3.12, scikit-learn 1.5+
How It Works
-
Bootstrap: for each of
n_estimatorstrees, draw n rows with replacement (about 63.2% unique rows per tree). -
Grow a tree: at every node, pick a random subset of
max_featuresfeatures and choose the best split among them only (Gini or entropy for classification, squared error for regression). -
No pruning by default: trees grow deep (
max_depth=None) so each has low bias. -
Aggregate: average the trees’ class probabilities (scikit-learn soft voting) or numeric predictions.
-
Out-of-bag (OOB) evaluation: each row is predicted only by the trees that didn’t see it, giving a free validation estimate.
Why It Works
-
Averaging B trees with variance σ² and pairwise correlation ρ gives variance ρσ² + (1 − ρ)σ²/B.
-
More trees shrink the second term; random feature selection lowers ρ, shrinking the first.
Code Example
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
rf = RandomForestClassifier(n_estimators=300, oob_score=True, n_jobs=-1, random_state=42)
rf.fit(X_train, y_train)
print("OOB score:", round(rf.oob_score_, 3))
print("test accuracy:", round(rf.score(X_test, y_test), 3))
top = np.argsort(rf.feature_importances_)[::-1][:5]
print("top features:", list(X.columns[top]))
Strengths and Weaknesses
| Strengths | Weaknesses |
|---|---|
| Accurate with default settings | Large models, slower prediction than one tree |
| Handles non-linearity and interactions | Less interpretable than a single tree |
| No scaling needed; robust to outliers | Can’t extrapolate beyond training target range (regression) |
| OOB estimate and feature importances | Impurity importances are biased |
| Parallel training | Tuned gradient boosting is often more accurate |
Common Pitfalls
-
Trusting impurity-based
feature_importances_for high-cardinality features; checkpermutation_importance. -
Ignoring class imbalance; use
class_weight="balanced"or resampling. -
Very deep trees on large data use a lot of memory; limit
max_depthormin_samples_leaf.
Interview Tips
-
Name both sources of randomness (rows and features) and explain decorrelation.
-
Mention OOB evaluation and that adding trees doesn’t overfit.