What are bias and variance in machine learning?
Bias is systematic prediction error caused by overly restrictive assumptions, while variance is prediction instability caused by sensitivity to the particular training sample. High bias commonly corresponds to underfitting, whereas high variance corresponds to overfitting. The main trade-off is choosing enough model capacity to learn the underlying pattern without learning sample-specific noise.
Read full answerHide full answer
Applies tomachine learning concepts (all versions)
Bias and Variance
-
Bias: At a fixed input
x, the difference between the model’s average prediction over many possible training samples and the true conditional mean of the target. A high-bias model makes systematic errors because its assumptions or capacity are too restrictive. -
Variance: How much a model’s prediction at a fixed input changes when it is trained on different samples. A high-variance model is unstable and may fit training-specific noise.
-
Underfitting: Both training and validation errors remain high because the model cannot represent the relevant pattern.
-
Overfitting: Training error is much lower than validation error because the model has learned patterns that do not generalize.
Bias and variance are distinct from irreducible noise: reducing either source of estimation error does not remove randomness in the target itself.
Squared-Error Decomposition
For squared prediction error at a fixed input x, the expected test error can be decomposed as:
Expected test MSE at x = Bias_x² + Variance_x + sigma²
-
Expected test MSE at x: Average squared difference between the true target and a prediction from a model trained on a randomly selected training dataset, averaged over possible datasets and targets.
-
Bias_x²: Squared difference between the model’s expected prediction at
xand the true conditional meanE[Y | X = x]. -
Variance_x: Variation in the model’s predictions at
xacross different possible training datasets. -
sigma²: Irreducible conditional variance of
YgivenX = x; even a perfect model cannot predict this random component.
More flexible models usually have lower bias but higher variance, while more constrained models generally show the opposite pattern.
Diagnosing Bias and Variance
Learning curves vary the training-set size, while validation curves vary model complexity or regularization. Use cross-validation so that preprocessing and model fitting occur only on each fold’s training portion.
| Training error | Validation error | Likely diagnosis |
|---|---|---|
| Low | Low and close to training error | Good generalization |
| Low | Substantially higher | High variance |
| High | High and close to training error | High bias |
| High | Lower than training error | Unusual; check leakage, sampling differences, or evaluation bugs |
A high-variance model often benefits from additional training data, whereas a high-bias model may plateau at poor performance as data increases. A validation curve that initially improves and then deteriorates indicates that increasing complexity has moved the model from underfitting toward overfitting.
The classic bias-variance trade-off is often illustrated with a U-shaped test-error curve, but double descent is an important exception: sufficiently large neural networks can enter a second high-performance regime as model size increases. This behavior is setup-dependent rather than universal.
Remedies and Ensembles
| Problem | Possible remedies | Trade-off |
|---|---|---|
| High bias | Add informative features, increase model capacity, reduce excessive regularization, or train iterative models longer | Greater capacity can increase variance |
| High variance | Gather more data, use L1/L2 regularization, simplify the model, select features, apply dropout where appropriate, use early stopping, or use bagging | Greater simplicity or regularization can increase bias |
| Ensemble | How it works | Main strength | Main weakness | Typical use | Approximate training scale |
|---|---|---|---|---|---|
| Bagging and random forests | Train diverse models on resampled data and average their predictions; random forests also partition features at splits | Primarily reduces variance and improves stability | Less interpretable and potentially expensive for large forests | Unstable, high-variance models | O(B × C) for B base models with per-model cost C |
| Boosting | Train models sequentially so later models emphasize earlier errors | Often reduces bias and can provide strong predictive performance | Sequential training and possible overfitting on noisy data | Models that underfit the signal | Roughly O(B × C), with less parallelism than bagging |
Common Pitfalls
-
Fit scalers, imputers, encoders, and feature selectors only on training folds, normally through a
PipelineorColumnTransformer; fitting them on all data causes leakage. -
Choose metrics that match the task: MAE, RMSE, and R² for regression; precision, recall, F1, ROC-AUC, and PR-AUC for classification when relevant. Accuracy can mislead with imbalanced classes because a large majority class can dominate the score even when minority-class performance is poor.
-
Scaling is important for k-NN, SVM, PCA, and many gradient-based methods, but is generally unnecessary for tree-based models.
-
Handle missing values explicitly, keep the final test set unused until model selection is complete, and fix random seeds when reproducibility matters.
-
Do not repeatedly tune against the same validation set because repeated experimentation effectively overfits that set.