Skip to content
Machine Learning & Data Science

Model Evaluation interview questions and answers

55 questions, 30 with code. About 5 h 56 min to read them all.

By TestLab · Updated

Plan for an interview
55 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is model evaluation in machine learning?

Model evaluation is the process of measuring how well a trained model predicts unseen data and using those measurements to compare models, tune choices, and estimate production performance. It determines whether a model has learned a generalizable pattern rather than memorized its training data. Its main trade-off is that realistic evaluation requires representative held-out data and additional computation, whereas simpler evaluations may be fast but misleadingly optimistic.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Evaluation Setup

A reliable evaluation separates data according to when information becomes available:

  • Training set: Fits model parameters.

  • Validation set: Supports model selection and hyperparameter tuning.

  • Test set: Remains untouched until all decisions are finalized, then provides a final performance estimate.

For limited data, cross-validation repeatedly trains on different folds and evaluates on the held-out fold:

  • KFold: Suitable for ordinary independent observations.

  • StratifiedKFold: Preserves approximate class proportions across folds and is commonly used for classification.

  • GroupKFold: Keeps observations from the same group in one fold, preventing group-level leakage.

  • TimeSeriesSplit: Uses earlier observations for training and later ones for validation, preserving temporal order.

Preprocessing that learns from data—such as scaling, imputation, encoding, or feature selection—must use only the training portion of each split. When a scikit-learn Pipeline is passed to cross-validation, each transformation is fitted only on that fold’s training data before being applied to its validation fold.

Classification Metrics

Let TP be true positives, TN true negatives, FP false positives, FN false negatives, and n the number of observations.

Metric Formula Meaning
Accuracy (TP + TN) / n Fraction classified correctly
Precision TP / (TP + FP) Fraction of predicted positives that are correct
Recall TP / (TP + FN) Fraction of actual positives detected
Specificity TN / (TN + FP) Fraction of actual negatives correctly rejected
F1 2 * (Precision * Recall) / (Precision + Recall) Harmonic mean of precision and recall
  • ROC curve: Plots TPR = TP / (TP + FN) against FPR = FP / (FP + TN) across decision thresholds.

  • ROC-AUC: The area under the ROC curve; for distinct scores, it equals the probability that a random positive receives a higher score than a random negative. A value of 0.5 corresponds to random ranking.

  • Precision-recall curve: Plots precision and recall across decision thresholds.

  • PR-AUC: The area under the precision-recall curve; it is often more informative when positives are rare.

  • Log loss: For binary labels y_i in {0, 1} and predicted probabilities p_i, -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where log is the natural logarithm. It penalizes confident incorrect predictions heavily.

  • Brier score: (1/n) * sum((p_i - y_i)^2), where lower values indicate better probability predictions.

  • Reliability curve: Compares the mean predicted probability in each probability bin with the observed positive frequency, calculated as positives_in_bin / observations_in_bin.

Accuracy can mislead on imbalanced data. If only 1% of cases are positive, always predicting the negative class achieves 99% accuracy while detecting no positives.

Regression Metrics

For n observations with actual values y_i and predictions y_hat_i:

Metric Formula Interpretation
MAE (1/n) * sum(abs(y_i - y_hat_i)) Average absolute prediction error
MSE (1/n) * sum((y_i - y_hat_i)^2) Penalizes larger errors more strongly
RMSE sqrt(MSE) Error in the target’s original units
R² 1 - sum((y_i - y_hat_i)^2) / sum((y_i - mean(y))^2) Performance relative to predicting the evaluation-set mean
MAPE (100/n) * sum(abs((y_i - y_hat_i) / y_i)) Percentage error; unstable near zero
Quantile loss (1/n) * sum(q * max(y_i - y_hat_i, 0) + (1 - q) * max(y_hat_i - y_i, 0)) Pinball loss at quantile q, where q is between 0 and 1

MAE is less sensitive to outliers than RMSE. R² may be negative when a model performs worse than predicting the evaluation set’s mean. MAPE is undefined when an actual value is zero and unstable when values are close to zero.

Model Selection and Baselines

  • Start with a baseline, such as DummyClassifier or DummyRegressor, to verify that the model adds meaningful predictive value.

  • Use GridSearchCV for small parameter grids and RandomizedSearchCV for large search spaces.

  • Select metrics that reflect the cost of errors rather than relying only on accuracy or R².

  • Use nested cross-validation when tuning and reporting performance: inner folds select hyperparameters, while outer folds estimate the entire selection procedure.

  • Compare models with confidence intervals or paired statistical tests rather than treating small score differences as conclusive.

  • Keep the final test set untouched until model and threshold selection are complete.

Common Pitfalls

  • Data leakage: Validation or test information enters preprocessing or feature construction.

  • Test-set tuning: Repeatedly evaluating candidates on the test set inflates the final score.

  • Invalid splitting: Random splits break temporal or group relationships.

  • Wrong metric: The metric may not reflect the business objective or class imbalance.

  • Overfitting: A model may fit training data well but generalize poorly.

  • Distribution shift: Offline metrics may not reflect live traffic; production impact should be monitored or validated through A/B testing.

  • Ignored preprocessing requirements: Scaling matters for k-NN, SVM, PCA, and distance-based methods, but generally not for tree-based models.

Interview Tips

Be ready to explain how you selected the split, prevented leakage, chose metrics, established a baseline, and estimated uncertainty. Common follow-ups concern ROC-AUC versus PR-AUC, grouped or temporal validation, probability calibration, and why nested cross-validation gives a more honest estimate.

2

How do training, validation, and test datasets differ?

The training dataset fits model parameters, the validation dataset selects settings such as hyperparameters and thresholds, and the test dataset provides a final estimate on unseen examples. This separation reduces data leakage and selection bias. The main trade-off is allocating enough data for reliable learning while preserving an untouched test set for an honest evaluation.

Read full answer

Applies tomachine learning concepts and scikit-learn 1.5+ (Python 3.12)

Evaluation Setup

A robust evaluation separates the available data into three logical sets drawn from the same target population:

Dataset Purpose Typical actions Main risk
Training Fit learned parameters Train weights or trees and fit encoders, imputers, and scalers Insufficient or unrepresentative data
Validation Make modeling decisions Tune hyperparameters, compare models, select features or thresholds, and stop training Repeated tuning overfits the validation set
Test Estimate final generalization Evaluate the frozen procedure after all decisions are complete Using test results to make further decisions

Ratios such as 70/15/15 are only starting points; dataset size, class balance, collection cost, and production representativeness matter more. Fit every data-dependent preprocessing step only on the training portion. Calculating scaling statistics or imputing values before splitting leaks information from validation and test examples.

After selecting the approach, a final model may be retrained on training and validation data using the selected settings, while the test set remains untouched. Compare complex classifiers with a DummyClassifier baseline to verify that their added complexity produces meaningful improvement.

Cross-Validation and Model Selection

A fixed validation split can be sensitive to how the data were divided. Cross-validation creates several training-validation folds:

  • KFold: General data without important grouping or ordering.

  • StratifiedKFold: Classification folds that preserve class proportions.

  • GroupKFold: Grouped observations whose group membership must remain in one fold.

  • TimeSeriesSplit: Time-ordered data, with earlier observations training and later observations validating.

cross_validate can evaluate a Pipeline with several scorers. For tuning, GridSearchCV searches an exhaustive parameter grid, while RandomizedSearchCV samples larger search spaces; preprocessing should be inside the Pipeline so it is fitted only on each training fold.

Nested cross-validation uses inner folds for selection and outer folds to evaluate the entire selection procedure, producing a more honest estimate. When comparing final candidates, report uncertainty with a confidence interval or use a paired test, such as a paired bootstrap over out-of-fold predictions, rather than relying only on one winning score.

Metrics That Match the Problem

For binary classification, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN): the fraction of correct predictions. It can be misleading for imbalanced classes.

  • Precision = TP / (TP + FP): the fraction of predicted positives that are correct.

  • Recall = TP / (TP + FN): the fraction of actual positives detected.

  • F1 = 2 * precision * recall / (precision + recall): their harmonic mean.

  • Specificity = TN / (TN + FP): the fraction of actual negatives correctly rejected.

  • ROC curve: plots FPR = FP / (FP + TN) against TPR = TP / (TP + FN) across thresholds; ROC-AUC is the area under that curve.

  • PR curve: plots recall = TP / (TP + FN) against precision = TP / (TP + FP); PR-AUC is the area under that curve and is generally more informative for rare positives.

  • Log loss = -1/n * sum(y * log(p) + (1 - y) * log(1 - p)): here, n is the number of examples, y is the true binary label, p is the predicted positive probability, and log is the natural logarithm.

  • Brier score = 1/n * sum((p - y)^2): the mean squared difference between predicted probabilities and outcomes. Reliability curves compare predicted and observed probabilities to reveal miscalibration.

For regression, let n be the number of examples, y_i the true target, and y_hat_i the prediction:

  • MAE = 1/n * sum(|y_i - y_hat_i|): errors in the target’s original units.

  • MSE = 1/n * sum((y_i - y_hat_i)^2): penalizes larger errors more heavily.

  • RMSE = sqrt(MSE): retains the target units while emphasizing large errors.

  • R² = 1 - sum((y_i - y_hat_i)^2) / sum((y_i - y_bar)^2), where y_bar is the target mean: performance relative to predicting the mean.

  • MAPE = 100/n * sum(|y_i - y_hat_i| / |y_i|): percentage error, but unstable when y_i is zero or close to zero.

  • Quantile loss = |y - y_hat| * (tau if y >= y_hat, otherwise 1 - tau), where tau is the target quantile: appropriate for predicting conditional percentiles.

Common Pitfalls

  • Data leakage: Fitting preprocessing on all data contaminates evaluation.

  • Tuning on the test set: Repeated decisions after viewing test scores turn it into another validation set.

  • Overfitting the validation set: Too many configurations can make its score overly optimistic.

  • Wrong split strategy: Random splits can break temporal or group relationships.

  • Wrong metric: Accuracy can hide poor minority-class performance.

  • Offline–online mismatch: Validate important product decisions with an A/B test when feasible.

Interviewers commonly ask why the test set must remain untouched, how thresholds should be tuned, when cross-validation is preferable, and how leakage from preprocessing, groups, or time can be prevented.

3

What is cross-validation and why use it in machine learning?

Cross-validation is a resampling technique that evaluates a model across multiple train-validation splits to estimate how well it generalizes to unseen data. It reduces dependence on one fortunate or unfortunate split and supports model and hyperparameter selection, at the cost of additional training time and some bias from reusing the dataset.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Training data: Used to fit model parameters.

  • Validation data: Used to select hyperparameters, features, and models. Cross-validation creates several rotating validation folds rather than relying on one fixed split.

  • Test data: Kept untouched until the evaluation design and selected model are finalized. It provides a final performance estimate that was not influenced by tuning.

  • Preprocessing: Each transformation must be fitted only on the current training fold. A scikit-learn Pipeline ensures this behavior and prevents leakage.

In each fold, the model is trained on k - 1 folds and evaluated on the held-out fold. Scores are then aggregated across folds, commonly using the mean and standard deviation. After selection, the model can be refitted on all development data before the untouched test set is used once.

Resampling Strategies

Variant When to Use Pros Cons
KFold I.i.d. data General-purpose validation May produce imbalanced folds; use shuffle=True when randomized folds are appropriate
StratifiedKFold Classification Preserves class proportions as closely as integer counts allow Requires enough minority-class samples for the requested folds
RepeatedKFold More stable estimates Repeats randomized folds to quantify variability Requires multiple model fits
LeaveOneOut Small datasets Almost all observations train each model Each model trains on n - 1 observations and validates on one, so the estimate can have high variance and needs n fits
GroupKFold Repeated observations from groups, such as patients Keeps each group entirely in one fold Requires reliable group identifiers
TimeSeriesSplit Time-ordered data Trains on earlier observations and tests on later ones Not appropriate when temporal order has no meaning

Random splits can leak information when observations share groups, patients, households, or temporal dependencies. Cross-validation estimates uncertainty under the selected splitting strategy; it cannot repair a fundamentally inappropriate split.

Metrics and Formulas

For binary classification, let TP mean true positives, TN true negatives, FP false positives, and FN false negatives:

  • accuracy = (TP + TN) / (TP + TN + FP + FN)

  • precision = TP / (TP + FP): the fraction of predicted positives that are correct.

  • recall = TP / (TP + FN): the fraction of actual positives that are detected.

  • F1 = 2 * precision * recall / (precision + recall): the harmonic mean of precision and recall.

  • specificity = TN / (TN + FP): the fraction of actual negatives correctly identified.

A ROC curve plots true-positive rate, which equals recall, against false-positive rate, which equals 1 - specificity; ROC-AUC summarizes ranking performance across thresholds. A precision-recall curve plots precision against recall, and PR-AUC is especially informative for rare positive classes because it reflects performance under the observed positive prevalence. Average precision is related to the PR curve but is not identical to its trapezoidal area.

For probabilistic predictions, log loss = -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where n is the number of examples, y_i is the true binary label, and p_i is the predicted probability of the positive class. The Brier score, (1/n) * sum((p_i - y_i)^2), measures probabilistic accuracy, while a reliability curve compares predicted probabilities with observed positive frequencies.

Accuracy can mislead on imbalanced data, so precision, recall, F1, PR-AUC, log loss, or calibration metrics may be more appropriate. Regression evaluations commonly use MAE, MSE, RMSE, R², MAPE, and quantile loss; MAPE is unstable when the target values are near zero.

Model Selection

  • Compare candidates on the same folds and report the score distribution, mean, and variability; paired fold differences are more informative than comparing only independent averages.

  • Include a simple baseline, such as DummyClassifier, to verify that the model provides meaningful improvement.

  • GridSearchCV and RandomizedSearchCV select hyperparameters through cross-validation. In nested cross-validation, inner folds perform tuning while outer folds provide an honest estimate of the entire selection procedure.

  • A separate test set remains useful when available because cross-validation repeatedly uses the development data and can produce optimistic results if too many candidates are evaluated.

Common Pitfalls

  • Leakage: Fit scaling, imputation, encoding, feature selection, and target-based transformations inside each training fold, preferably with Pipeline or ColumnTransformer.

  • Wrong split: Use stratified folds for ordinary classification, grouped folds for related observations, and time-based splits for temporal data.

  • Wrong metric: Do not rely on accuracy for severe class imbalance; select metrics tied to the real error costs.

  • Tuning contamination: Do not repeatedly inspect ordinary validation folds and then report those same results as an unbiased final estimate; use nested cross-validation or an untouched test set.

  • Reproducibility: Fix random seeds when randomized folds or estimators are used.

  • Business mismatch: Offline model metrics may not represent production impact, so validate important product decisions with online experiments such as A/B tests.

4

What are precision, recall, and F1-score in classification?

Precision, recall, and F1-score evaluate how correctly a classification model identifies a particular class. Precision measures result reliability, recall measures how much of the class the model finds, and F1-score balances them; the main trade-off is that raising the decision threshold often improves precision while reducing recall.

Read full answer

Applies tomachine learning classification concepts

Formulas

For the positive class:

  • True positives (TP): Positive cases correctly predicted as positive.

  • False positives (FP): Negative cases incorrectly predicted as positive.

  • False negatives (FN): Positive cases incorrectly predicted as negative.

Precision answers: “Of the cases predicted positive, how many were actually positive?”

Precision = TP / (TP + FP)

Recall answers: “Of the cases that were actually positive, how many did the model find?”

Recall = TP / (TP + FN)

F1-score is the harmonic mean of precision and recall:

F1 = 2 * (Precision * Recall) / (Precision + Recall)

The harmonic mean penalizes situations where one metric is very low. A model cannot achieve a high F1-score by having only strong precision or only strong recall.

Worked Example

Suppose a model evaluates 100 cases, of which 20 are actually positive:

  • It correctly identifies 15 positives: TP = 15.

  • It incorrectly labels 10 negatives as positive: FP = 10.

  • It misses 5 actual positives: FN = 5.

Therefore:

  • Precision = 15 / (15 + 10) = 0.60

  • Recall = 15 / (15 + 5) = 0.75

  • F1 = 2 * (0.60 * 0.75) / (0.60 + 0.75) ≈ 0.67

The model predicts 25 positive cases, but only 15 are correct, so its precision is 60%. It finds 15 of the 20 actual positive cases, so its recall is 75%.

Choosing the Right Metric

Metric Main question Best suited for
Precision Are predicted positives trustworthy? False positives are expensive
Recall Are most actual positives found? Missing a positive is expensive
F1-score Is there a useful balance? Both false positives and false negatives matter
Accuracy Are all predictions correct? Classes are reasonably balanced
Specificity Are negatives correctly rejected? False positives are the primary concern
PR-AUC Is ranking useful across thresholds? Rare positive classes or imbalance

For example:

  • Spam detection: High precision is often preferred because false positives can hide legitimate messages.

  • Disease screening: High recall may be prioritized because missed cases can be serious.

  • Fraud detection: F1 or PR-AUC may be useful, but the final business cost of false positives and false negatives should guide the threshold.

Accuracy can be misleading when the positive class is rare. A model predicting every case as negative might achieve 99% accuracy in a dataset containing only 1% positives while having zero recall.

Multi-Class Classification

Precision, recall, and F1-score can be calculated independently for each class by treating that class as positive and all other classes as negative. They may then be summarized using:

  • Macro F1: Average each class’s F1 equally, highlighting poor performance on rare classes.

  • Weighted F1: Average using class frequencies, reflecting overall performance while favoring frequent classes.

  • Micro F1: Aggregate TP, FP, and FN across classes. In single-label multiclass classification, micro F1 equals accuracy.

Common Pitfalls

  • Treating F1 as always sufficient: Choose metrics based on the real cost of different errors.

  • Ignoring class imbalance: Prefer precision, recall, F1, PR-AUC, and confusion matrices when classes are imbalanced.

  • Reporting only one threshold: These metrics depend on the model’s decision threshold; evaluate performance across thresholds when appropriate.

  • Averaging without context: State whether scores are class-specific, macro-averaged, weighted, or micro-averaged.

  • Changing the positive class: “Positive” is relative to the evaluation setup, so state which class each metric describes.

5

What is a confusion matrix in model evaluation?

A confusion matrix summarizes a classification model’s predictions by comparing predicted classes with actual classes. It identifies the kinds of errors a model makes, but it does not provide a single performance score, and those errors must be interpreted in light of class imbalance and the chosen decision threshold.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Before evaluating a classifier, reserve a test set that remains untouched until final evaluation. Use the remaining data for training, validation, and cross-validation:

  • Training set: fits model parameters.

  • Validation set: selects hyperparameters and decision thresholds.

  • Test set: measures final, unbiased generalization performance.

  • Cross-validation: estimates performance across multiple folds, reducing dependence on one split.

Use StratifiedKFold for ordinary classification, GroupKFold when observations belong to dependent groups, and TimeSeriesSplit for time-ordered data. Compare models with GridSearchCV or RandomizedSearchCV, preferably inside a Pipeline. Use nested cross-validation when tuning and performance estimation must both be unbiased. Compare against a DummyClassifier baseline and report uncertainty using fold-level confidence intervals or paired statistical comparisons.

Matrix Structure

For binary classification, a common convention is:

Actual Predicted Negative Positive
Negative TN FP
Positive FN TP
  • TP: actual positives correctly predicted.

  • TN: actual negatives correctly predicted.

  • FP: actual negatives incorrectly predicted as positive.

  • FN: actual positives incorrectly predicted as negative.

For multiclass classification, entry (i, j) contains the number of examples whose true class was i and whose predicted class was j. Diagonal entries are correct predictions; off-diagonal entries reveal which classes the model confuses.

Classification Metrics

The confusion matrix supports several metrics:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN): fraction of all predictions that are correct. It can be misleading when classes are imbalanced.

  • Precision = TP / (TP + FP): fraction of predicted positives that are truly positive.

  • Recall = TP / (TP + FN): fraction of actual positives that the model detects.

  • F1 = 2 * Precision * Recall / (Precision + Recall): harmonic mean of precision and recall.

  • Specificity = TN / (TN + FP): fraction of actual negatives correctly identified.

  • Log loss = -1/n * sum(y * log(p) + (1 - y) * log(1 - p)): penalizes incorrect or overconfident probabilities. Here, n is the number of examples, y is the binary label, and p is the predicted probability of class 1.

  • Brier score = 1/n * sum((p - y)^2): mean squared difference between predicted probabilities and binary outcomes. Lower is better.

Always select metrics based on the business cost of false positives and false negatives rather than reporting accuracy alone.

Curves and Calibration

  • ROC curve: plots true-positive rate, TP / (TP + FN), against false-positive rate, FP / (FP + TN), across thresholds.

  • ROC-AUC: summarizes ranking quality across those thresholds; it may look optimistic for severely imbalanced data.

  • Precision-recall curve: plots precision against recall across thresholds.

  • PR-AUC: summarizes the precision-recall curve and is generally more informative when the positive class is rare.

  • Calibration: checks whether predicted probabilities match observed frequencies. Reliability curves compare predicted probability bins with actual positive rates, while the Brier score summarizes probabilistic accuracy.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# 1. Create fixed development and untouched test splits.
data = load_breast_cancer()
X, y = data.data, data.target

X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25, random_state=42, stratify=y_dev
)

# 2. Keep preprocessing inside the Pipeline to prevent leakage.
model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scorers = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "pr_auc": "average_precision",
    "log_loss": "neg_log_loss",
}

# 3. Estimate performance using only training folds.
scores = cross_validate(
    model, X_train, y_train, scoring=scorers, cv=cv, n_jobs=-1
)
for name, values in scores.items():
    print(f"{name}: {values.mean():.3f} +/- {values.std():.3f}")

# 4. Select a threshold using validation data only.
model.fit(X_train, y_train)
positive_index = np.flatnonzero(model.classes_ == 1)[0]
val_probability = model.predict_proba(X_val)[:, positive_index]

precision, recall, thresholds = precision_recall_curve(y_val, val_probability)
f1 = 2 * precision[:-1] * recall[:-1] / (
    precision[:-1] + recall[:-1]
)
best_threshold = thresholds[np.argmax(f1)]

# 5. Evaluate once on the untouched test set.
test_probability = model.predict_proba(X_test)[:, positive_index]
test_prediction = (test_probability >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_prediction, labels=[0, 1]))
print(classification_report(y_test, test_prediction, labels=[0, 1]))
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probability):.3f}")
print(f"Test PR-AUC (average precision): {average_precision_score(y_test, test_probability):.3f}")

Common Pitfalls

  • Data leakage: fitting a scaler, encoder, or imputer on all data exposes test information. Put preprocessing inside a Pipeline.

  • Wrong metric: accuracy can hide poor minority-class recall; report precision, recall, F1, PR-AUC, and the confusion matrix for imbalanced tasks.

  • Threshold leakage: do not choose a threshold using test results.

  • Invalid splitting: random folds are unsuitable for temporal or grouped observations.

  • Overfitting the validation set: repeated manual tuning eventually turns validation data into training data; use cross-validation or a separate test set.

  • Offline–online mismatch: an offline metric may not reflect business impact. Validate important changes through an A/B test when possible.

6

What are the ROC curve and AUC in classification?

The ROC curve shows how a classifier’s true-positive rate changes with its false-positive rate across decision thresholds. ROC-AUC summarizes that curve with one number, making it useful for comparing models’ ranking performance on the same evaluation data, but it does not select an operating threshold or guarantee calibrated probabilities. For heavily imbalanced datasets, precision-recall analysis is often more informative about minority-class performance.

Read full answer

Applies tomachine-learning concepts

Evaluation Setup

  • Training set: Used to fit model parameters and preprocessing.

  • Validation set: Used to select hyperparameters, thresholds, or calibration methods.

  • Test set: Kept untouched until the final evaluation.

  • Cross-validation: Reuses the development data to estimate performance more reliably. Use StratifiedKFold for ordinary classification, GroupKFold when observations belong to related groups, and TimeSeriesSplit for time-ordered observations.

  • Fit preprocessing such as scaling, imputation, and encoding only on the training portion of each split. A scikit-learn Pipeline helps enforce this inside cross-validation and prevents preprocessing leakage.

ROC Curve

For each candidate decision threshold, classify a sample as positive when its predicted score or probability is at least that threshold. The ROC curve plots:

  • True-positive rate (TPR): TPR = TP / (TP + FN)

  • False-positive rate (FPR): FPR = FP / (TN + FP)

Here, TP is the number of true positives, FN the number of false negatives, FP the number of false positives, and TN the number of true negatives. FPR appears on the horizontal axis, while TPR appears on the vertical axis.

The diagonal from (0, 0) to (1, 1) represents random ranking. A curve above the diagonal indicates that positive samples generally receive higher scores than negative samples. ROC analysis evaluates ranking across thresholds rather than performance at one selected threshold.

ROC-AUC and Precision-Recall Analysis

ROC-AUC, or area under the ROC curve, summarizes how well the model separates the classes across all thresholds. It can be interpreted as the probability that a randomly selected positive receives a higher score than a randomly selected negative, counting a tie as half a success. A value of 0.5 corresponds to random ranking, while 1.0 represents perfect separation. ROC-AUC should generally be compared only between models evaluated on the same population and labels.

The precision-recall curve plots precision against recall as the threshold changes:

  • Precision: TP / (TP + FP)

  • Recall: TP / (TP + FN)

Precision-recall analysis is often preferable when positive cases are rare because it directly exposes the precision cost of detecting minority-class positives. PR-AUC means the area under this curve, but its exact calculation depends on the integration convention. In scikit-learn, average_precision_score computes average precision, a commonly used PR summary that is not identical to the trapezoidal PR-AUC. Average precision has a random-ranking baseline equal to the positive-class prevalence.

Related Metrics

Metric Formula Purpose
Accuracy (TP + TN) / (TP + TN + FP + FN) Overall correctness; potentially misleading with imbalanced classes
Precision TP / (TP + FP) Fraction of predicted positives that are correct
Recall TP / (TP + FN) Fraction of actual positives detected
F1 2 × Precision × Recall / (Precision + Recall) Harmonic mean of precision and recall
Specificity TN / (TN + FP) Fraction of actual negatives correctly identified
Log loss -(1/n) × sum(y log(p) + (1-y) log(1-p)) Evaluates both discrimination and probability quality
Brier score (1/n) × sum((p-y)^2) Measures probability calibration

For log loss and Brier score, y is the true binary label, p is the predicted probability, and n is the number of observations. A reliability curve compares predicted probability bins with the observed positive frequency; unlike ROC-AUC, these probability metrics assess whether confidence matches outcomes.

Common Pitfalls

  • Tuning on the test set: Select hyperparameters and thresholds using validation or cross-validation, then evaluate once on the untouched test set.

  • Ignoring imbalance: Do not rely on accuracy alone; inspect precision, recall, F1, the confusion matrix, ROC-AUC, and precision-recall results.

  • Assuming AUC selects a threshold: Choose an operating point based on the relative costs of false positives and false negatives.

  • Confusing discrimination with calibration: A high ROC-AUC does not imply that predicted probabilities are accurate or well calibrated.

  • Using inappropriate validation splits: Use grouped or time-based splits when observations are related or ordered.

  • Ignoring deployment objectives: Offline metrics may not reflect the business objective, so final online impact should be validated through controlled experiments such as A/B tests.

7

When is accuracy a misleading metric for classification models?

Accuracy is the proportion of predictions a classification model gets correct. It is simple and useful when classes are balanced and false positives and false negatives have similar costs, but it can hide serious errors on rare classes. Its main trade-off is interpretability versus sensitivity to class imbalance and unequal error costs.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Evaluation Setup

A reliable evaluation process separates data before training:

  • Training set: Used to fit the model and learn parameters.

  • Validation set: Used for model selection, hyperparameter tuning, and threshold selection.

  • Test set: Kept untouched until final evaluation so the reported result reflects performance on unseen data.

  • Cross-validation: Rotates training and validation folds to obtain a more stable estimate:

  • KFold for general data.

  • StratifiedKFold for classification with imbalanced classes.

  • GroupKFold when observations from the same person, customer, or device must not appear in different folds.

  • TimeSeriesSplit for time-ordered data, where future observations must not predict earlier ones.

Any preprocessing that learns from data—such as scaling, imputation, or feature selection—must be fitted only on each training fold, commonly through a scikit-learn Pipeline.

Why Accuracy Can Mislead

Let TP mean true positives, TN true negatives, FP false positives, and FN false negatives. Let n be the total number of observations.

Accuracy = (TP + TN) / n

This formula counts every correct prediction equally. Consider a dataset with 99 negative cases and one positive case. A model that predicts every case as negative achieves 99% accuracy while failing to identify the only positive case. This is especially misleading when positive cases represent fraud, disease, or system failure.

Accuracy is less informative when:

  • Classes are highly imbalanced.

  • False positives and false negatives have different consequences.

  • Different subgroups have different error rates.

  • The decision threshold matters more than the model’s overall ranking ability.

  • Probabilistic predictions need to be well calibrated.

Metrics That Reveal More

  • Precision = TP / (TP + FP): Of the cases predicted positive, the proportion that are actually positive. It answers, “When the model raises an alert, how often is it correct?”

  • Recall = TP / (TP + FN): Of the actual positive cases, the proportion detected. It answers, “How many positive cases does the model find?”

  • Specificity = TN / (TN + FP): Of the actual negative cases, the proportion correctly rejected.

  • F1 = 2 × (Precision × Recall) / (Precision + Recall): The harmonic mean of precision and recall. It is useful when both false positives and false negatives matter, although it still hides the two errors if they differ greatly.

For ranked predictions:

  • ROC curve: Plots the true-positive rate against the false-positive rate across decision thresholds. ROC-AUC summarizes how well the model ranks a randomly selected positive above a randomly selected negative.

  • Precision-recall curve: Shows precision and recall across thresholds. PR-AUC, commonly calculated with average_precision_score, is generally more informative than ROC-AUC when the positive class is rare.

  • Log loss: Averages -(y × log(p) + (1 - y) × log(1 - p)), where y is the true binary label and p is the predicted probability of the positive class. It strongly penalizes confident but incorrect probabilities.

  • Brier score: Calculates the mean squared difference between predicted probabilities and binary outcomes. Lower values indicate better probability calibration.

  • Reliability curve: Compares predicted probabilities with observed positive rates across probability ranges, revealing whether predictions are overconfident or underconfident.

Choosing Metrics

Use a metric aligned with the actual decision:

Objective Useful metrics
Balanced classes and similar error costs Accuracy
Minimize unnecessary false alarms Precision
Detect as many positive cases as possible Recall
Balance precision and recall F1 or PR-AUC
Compare classification thresholds ROC curve and ROC-AUC
Evaluate probability quality Log loss, Brier score, reliability curve

A DummyClassifier provides a useful baseline. If a sophisticated model does not outperform it on an appropriate metric, the added complexity may not be justified.

Common Pitfalls

  • Selecting accuracy for a heavily imbalanced dataset without also reporting precision, recall, F1, and PR-AUC.

  • Tuning thresholds or hyperparameters on the test set instead of using validation or cross-validation.

  • Reporting only an aggregate score when subgroup performance differs substantially.

  • Interpreting ROC-AUC as though it directly represents operational performance at the chosen threshold.

  • Ignoring probability calibration when predictions are used as confidence scores.

  • Using random splits for time-series or grouped data, which can cause leakage.

  • Assuming offline metrics reflect business impact without validating the deployed system through an A/B test.

8

What are overfitting and underfitting in machine learning?

Overfitting occurs when a model learns noise and sample-specific patterns in the training data, so its performance drops on unseen data. Underfitting occurs when a model is too constrained to represent important patterns, so it performs poorly on both training and unseen data. The main trade-off is between model complexity and generalization.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Evaluation Setup

Establish the evaluation setup before comparing models:

  • Training set: Used to fit model parameters.

  • Validation set: Used for hyperparameter tuning, model selection, and possibly early stopping.

  • Test set: Kept untouched until final evaluation so it provides an unbiased performance estimate.

For limited data, use cross-validation: KFold generally, StratifiedKFold for classification, GroupKFold when related observations must remain together, and TimeSeriesSplit for time-ordered data. Fit every scaler, encoder, imputer, or feature selector only on each training fold; a scikit-learn Pipeline helps prevent leakage.

When tuning and evaluating a model, use nested cross-validation: inner folds select the model or hyperparameters, while outer folds estimate generalization. Compare candidates using paired fold scores with confidence intervals or suitable paired tests, and include a DummyClassifier baseline.

Overfitting and Underfitting

  • Overfitting: Training performance is high, but validation or test performance is substantially worse.

  • Causes: excessive capacity, insufficient data, overly complex features, or repeated tuning against the validation set.

  • Remedies: regularization, more data, data augmentation where appropriate, early stopping, feature selection, or a simpler model.

  • Underfitting: Training and validation performance are both poor.

  • Causes: insufficient capacity, overly strong regularization, inadequate features, unrealistic assumptions, or poor optimization.

  • Remedies: increase capacity, reduce regularization, add informative features, or improve data quality.

Learning curves comparing training and validation performance help diagnose these conditions. Moderate scores alone do not establish either condition; also consider uncertainty, data quality, and metric suitability.

Bias–Variance Trade-off

A simplified decomposition of expected prediction error is:

Expected prediction error = Bias² + Variance + Irreducible noise

  • Bias: Systematic error from incorrect or overly simple assumptions; high bias commonly corresponds to underfitting.

  • Variance: Sensitivity of predictions to changes in the training sample; high variance commonly corresponds to overfitting.

  • Irreducible noise: Random variation that cannot be explained using the available information.

Increasing complexity often reduces bias but increases variance. Regularization or more data can reduce variance, while additional capacity or useful features can reduce bias.

Choosing Evaluation Metrics

For binary classification:

Metric Formula Meaning
Accuracy (TP + TN) / N Fraction correct; misleading with imbalanced classes.
Precision TP / (TP + FP) Fraction of predicted positives that are truly positive.
Recall TP / (TP + FN) Fraction of actual positives detected.
Specificity TN / (TN + FP) Fraction of actual negatives correctly rejected.
F1 2 × Precision × Recall / (Precision + Recall) Harmonic mean of precision and recall.
Log loss -1/N × Σ[y log(p) + (1-y) log(1-p)] Penalizes confident incorrect probabilities.
Brier score 1/N × Σ(p-y)² Measures probability calibration.

Here, TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives; N is the number of examples; y is the true binary label; and p is the predicted positive probability.

The ROC curve plots TPR = TP / (TP + FN) against FPR = FP / (FP + TN) as the threshold changes. ROC-AUC is its area and can be expressed as P(score_positive > score_negative), with ties counting one half. The precision-recall curve plots precision against recall over thresholds and is generally more informative when the positive class is rare. A common PR summary, average precision, is AP = Σ_k (R_k - R_{k-1}) × P_k, where R_k is recall and P_k is precision at threshold k. Reliability curves compare predicted probabilities with observed positive frequencies.

For regression:

Metric Formula
MAE 1/N × Σ abs(e_i)
MSE 1/N × Σ(e_i²)
RMSE sqrt(MSE)
R² 1 - SSE / SST
MAPE 100/N × Σ(abs(e_i) / abs(y_i))
Quantile loss max(alpha × e_i, (alpha - 1) × e_i)

Here, e_i = y_i - yhat_i is the prediction error, y_i is the actual target, yhat_i is the prediction, SSE is the residual sum of squares, SST = Σ(y_i - y_bar)², and alpha is the target quantile. MAE is directly interpretable in target units, RMSE penalizes large errors more strongly, and R² measures variance explained. MAPE is undefined when a target is zero and unstable near zero.

In scikit-learn, cross_validate can evaluate several scorers inside a Pipeline; classification_report and confusion_matrix summarize results, roc_auc_score computes ROC-AUC, and average_precision_score computes average precision. Use precision_recall_curve on validation predictions to select a threshold.

Common Pitfalls

  • Interpreting low training error alone as evidence of a good model.

  • Repeatedly tuning against the test set or overfitting the validation set.

  • Applying preprocessing before splitting the data or leaking information between folds.

  • Using accuracy for severely imbalanced classification.

  • Using random splits for time-series or grouped observations.

  • Assuming offline metrics reflect online impact; important products should validate business outcomes through controlled experiments such as A/B tests.

Interview Tips

Be ready to explain learning curves, the generalization gap, regularization, data leakage, appropriate metrics for imbalanced classes, and when additional data is preferable to a simpler model.

9

How do learning curves help evaluate model performance?

Learning curves show how a model’s training and validation performance changes as the training-set size grows. They reveal whether performance is limited mainly by bias, variance, or insufficient data, helping guide data collection, regularization, and model changes. The main trade-off is the computational cost of repeatedly fitting models across sample sizes and folds in exchange for clearer guidance than a single validation score provides.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

A learning curve should be produced entirely within the development data so that the final test set remains untouched until model selection is complete.

  • Training data: Used to fit each model.

  • Validation folds: Used to estimate performance on unseen observations.

  • Test data: Evaluated only once after the modeling and tuning decisions are final.

Scikit-learn’s learning_curve uses cross-validation within the supplied training data. At each requested sample size, it fits models on subsets of each fold’s training partition and records training and held-out validation scores. Use StratifiedKFold for classification, especially with imbalanced classes, GroupKFold when samples from the same group must remain together, and TimeSeriesSplit when temporal order must be preserved.

How to Read Learning Curves

The horizontal axis is the number of training examples, and the vertical axis is the evaluation score. Assuming higher scores are better:

  • High bias: Training and validation scores are both low and relatively close. More data alone may provide little benefit; additional features or greater model capacity may be needed.

  • High variance: Training performance is high while validation performance is substantially lower. More data, stronger regularization, or a less complex model may help.

  • Good fit: Training and validation performance are close, and validation performance has improved or stabilized.

  • Noisy small samples: Early estimates may fluctuate because they use few observations, so variation across folds matters.

For k folds, the mean score at sample size n can be written as mean_score(n) = (1 / k) * sum(fold_score_i). Here, n is the number of training examples, k is the number of folds, and fold_score_i is the score from fitting on n examples and evaluating against fold i.

Metrics to Plot

The metric determines what the curve diagnoses:

Metric Formula or meaning
Accuracy (TP + TN) / (TP + TN + FP + FN)
Precision TP / (TP + FP)
Recall TP / (TP + FN)
Specificity TN / (FP + TN)
F1 2 * precision * recall / (precision + recall)
ROC curve Connects (FPR, TPR) pairs across thresholds, where TPR = TP / (TP + FN) and FPR = FP / (FP + TN)
ROC-AUC Area under the ROC curve; a threshold-independent ranking measure
PR curve Connects precision and recall pairs across thresholds
PR-AUC Area under the PR curve or average precision; generally more informative for rare positives
Log loss -1/n * sum(y*log(p) + (1-y)*log(1-p))
Brier score 1/n * sum((p-y)^2)
MAE 1/n * sum(abs(y - y_hat))
MSE 1/n * sum((y - y_hat)^2)
RMSE sqrt(MSE)
R² 1 - sum((y - y_hat)^2) / sum((y - y_mean)^2)
MAPE 100/n * sum(abs(y - y_hat) / abs(y))

TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives. For log loss and Brier score, y is the true binary label, p is the predicted positive probability, and n is the observation count. For regression, y is the true value, y_hat is the prediction, and y_mean is the target mean. MAPE is unstable or undefined when the true value is near zero. Quantile loss, max(q*(y-y_hat), (q-1)*(y-y_hat)), evaluates errors asymmetrically for quantile q; values above 0.5 penalize underprediction more heavily. Reliability curves assess calibration by comparing predicted probabilities with observed positive rates.

Model Selection and Practical Guidance

  • Use classification_report and confusion_matrix for fixed-threshold summaries, and roc_auc_score and average_precision_score for threshold-independent evaluation. cross_validate can evaluate several scorers under one consistent split design.

  • Use precision_recall_curve on validation predictions to examine precision-recall trade-offs and select a threshold; do not choose that threshold using test results.

  • Compare with a DummyClassifier baseline. Use GridSearchCV or RandomizedSearchCV for hyperparameter search, and nested cross-validation when the model family, preprocessing, and hyperparameters are all selected.

  • Use fold-level confidence intervals or paired comparisons when judging whether model differences are meaningful rather than relying on one mean score.

  • Put scaling, imputation, encoding, and feature selection inside a Pipeline so they are fitted only on each fold’s training partition. Scaling is particularly important for distance-based, kernel, PCA, and gradient-descent models; trees generally do not require it.

  • For time series or grouped data, preserve the required split structure. Learning-curve generation is expensive because it repeats fitting across sample sizes and folds, but n_jobs can parallelize independent fits.

Common Pitfalls

  • Data leakage: Fitting preprocessing before splitting makes validation results overly optimistic.

  • Validation overfitting: Repeatedly selecting models or thresholds from the same validation folds can overfit those folds.

  • Wrong metric: Accuracy can conceal poor minority-class performance; use precision, recall, F1, PR-AUC, and calibration when relevant.

  • Wrong split strategy: Random splits are inappropriate for temporal or dependent grouped observations.

  • Misreading variability: A large training-validation gap is not automatically overfitting; inspect fold variation and absolute performance.

  • Business mismatch: An offline metric may not represent the online objective. Validate important product changes with A/B tests and monitor business outcomes such as conversion rate or incremental value.

10

How do explained variance and R-squared differ in regression?

Explained variance measures how much variability remains in a model’s prediction errors relative to the variability of the target. R-squared measures how much a model improves on a baseline that always predicts the target mean. The key trade-off is diagnostic focus: explained variance is useful for examining error variability, while R-squared measures goodness of fit relative to the mean baseline and penalizes constant prediction bias.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Evaluate regression models on data that was not used to fit them:

  • Training set: used to fit model parameters.

  • Validation set: used to select hyperparameters, features, or thresholds.

  • Test set: used for the final evaluation and kept untouched until model selection is complete.

  • Cross-validation: uses KFold for ordinary regression, GroupKFold for dependent groups, and TimeSeriesSplit when time order matters. StratifiedKFold is primarily used for classification.

  • Use GridSearchCV or RandomizedSearchCV on the training data for model selection. Nested cross-validation is preferable when an unbiased estimate of the entire tuning procedure is required.

A baseline such as scikit-learn’s DummyRegressor is important: a complex model is not useful unless it consistently outperforms that baseline.

Metric Definitions

For observations with true values y_i and predictions y_hat_i:

Explained variance

EV = 1 - Var(y_i - y_hat_i) / Var(y_i)

Here:

  • Var(y_i - y_hat_i) is the variance of the residuals, or prediction errors.

  • Var(y_i) is the variance of the true target around its mean.

  • A value of 1 means the residuals have zero variance.

  • A negative value means the residual variance exceeds the target variance.

R-squared, also called the coefficient of determination, is:

R² = 1 - sum((y_i - y_hat_i)^2) / sum((y_i - y_mean)^2)

Here:

  • y_mean is the mean of the true target.

  • The numerator is the model’s squared error.

  • The denominator is the squared error from always predicting y_mean.

  • R² = 0 means no improvement over that baseline.

  • Negative R² means the model performs worse than the baseline.

The difference can be expressed precisely. If residual bias is b = mean(y_i - y_hat_i), then:

R² = EV - b² / Var(y)

Therefore, explained variance ignores constant bias, while R-squared penalizes it. These ratio formulas assume that the target has nonzero variance.

Practical Comparison

Property Explained variance R-squared
Main question How variable are the remaining errors? How much better than predicting the mean?
Accounts for constant bias No Yes
Effect of a constant offset Unchanged Decreases for a nonzero offset
Can be negative Yes Yes
Maximum value 1 1
Common use Residual-variance diagnostics Model comparison and goodness of fit

For example, if the target is [0, 2, 4, 6] and predictions are [1, 3, 5, 7], every residual equals -1. Explained variance is 1 because residual variance is zero, but R-squared is 0.8 because the consistent offset is penalized.

Neither metric should automatically be interpreted as a percentage, especially when the score is negative. They are also dataset-dependent, so comparing them across unrelated targets requires additional context.

Common Pitfalls

  • Using the test set for tuning: repeatedly examining test scores turns it into a validation set and produces optimistic results.

  • Ignoring systematic bias: high explained variance can conceal a consistent prediction offset.

  • Choosing the wrong cross-validation strategy: random folds can leak information in time-series or grouped data.

  • Reporting only one metric: inspect MAE, RMSE, R-squared, residual plots, and prediction errors; R-squared alone does not reveal error magnitude or pattern.

  • Data leakage: fit preprocessing only on training folds, preferably with a Pipeline.

  • Ignoring uncertainty: compare models using repeated cross-validation, confidence intervals, or paired statistical tests rather than one score.

  • Assuming offline improvement guarantees business value: validate important outcomes through an online A/B test when appropriate.

Interviewers commonly ask why R-squared can be negative, how explained variance responds to a constant shift, which metric is more sensitive to outliers, and how the evaluation setup prevents leakage.

11

How do you evaluate regression model performance?

Regression model evaluation measures how accurately a model predicts continuous targets and whether its errors are acceptable for the application. MAE is easy to interpret and less influenced by large errors, while MSE and RMSE penalize large misses more heavily. Reliable evaluation uses leakage-free validation, an untouched test set, meaningful baselines, and metrics aligned with the real-world cost of errors.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Use three distinct data roles:

  • Training set: Fits model parameters.

  • Validation set: Supports model or hyperparameter selection.

  • Test set: Remains untouched until all decisions are complete.

Use cross-validation for more stable estimates:

  • KFold suits independent, identically distributed observations.

  • GroupKFold keeps related records, such as patients from one hospital, in the same fold.

  • TimeSeriesSplit respects temporal order.

  • StratifiedKFold is for classification rather than ordinary continuous-target regression.

Fit every learned preprocessing step—including scaling, imputation, and feature selection—inside each training fold with a Pipeline. Impute missing values inside that pipeline rather than before splitting the data. Use fixed random seeds and document the split strategy.

Regression Metrics

Metric Formula Interpretation
MAE (1/n) * sum(abs(y_i - y_hat_i)) Average error in target units; relatively robust to large errors.
MSE (1/n) * sum((y_i - y_hat_i)^2) Heavily penalizes large errors; measured in squared target units.
RMSE sqrt(MSE) Expresses errors in target units while emphasizing large misses.
R² 1 - sum((y_i - y_hat_i)^2) / sum((y_i - y_mean)^2) Compares the model with a mean-prediction baseline; negative values indicate worse performance.
MAPE (100/n) * sum(abs((y_i - y_hat_i) / y_i)) Percentage error, but unreliable near zero targets.
Quantile loss max(alpha * (y - y_hat), (1 - alpha) * (y_hat - y)) Evaluates quantiles with asymmetric penalties controlled by alpha.

Here, n is the number of observations, y_i is an actual target, y_hat_i is its prediction, and y_mean is the mean actual target. alpha lies between 0 and 1; 0.5 represents the median and 0.9 an upper quantile.

Use MAE for equal unit costs, RMSE when large errors are especially harmful, and R² as a baseline-relative measure. Avoid MAPE for zero-heavy targets, and use quantile loss for probabilistic forecasts or asymmetric costs.

Code Example

import numpy as np

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import Ridge
from sklearn.metrics import (
    mean_absolute_error,
    mean_absolute_percentage_error,
    mean_squared_error,
    r2_score,
    root_mean_squared_error,
)
from sklearn.model_selection import KFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)

# Split before fitting the model or any preprocessing.
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=1 / 3, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("ridge", Ridge()),
])

# The pipeline is fitted independently inside every CV fold.
scoring = {
    "mae": "neg_mean_absolute_error",
    "mse": "neg_mean_squared_error",
    "rmse": "neg_root_mean_squared_error",
    "r2": "r2",
}
cv = KFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    model, X_train, y_train, cv=cv, scoring=scoring, n_jobs=-1
)

for metric in scoring:
    values = cv_results[f"test_{metric}"]
    mean = -values.mean() if metric != "r2" else values.mean()
    print(f"CV {metric.upper()}: {mean:.3f} +/- {values.std():.3f}")

# Fit on training data and inspect the separate validation set.
model.fit(X_train, y_train)
val_predictions = model.predict(X_val)

def report_metrics(y_true, predictions, label):
    print(f"{label} MAE: {mean_absolute_error(y_true, predictions):.3f}")
    print(f"{label} MSE: {mean_squared_error(y_true, predictions):.3f}")
    print(f"{label} RMSE: {root_mean_squared_error(y_true, predictions):.3f}")
    print(f"{label} R2: {r2_score(y_true, predictions):.3f}")
    print(
        f"{label} MAPE: "
        f"{100 * mean_absolute_percentage_error(y_true, predictions):.2f}%"
    )

report_metrics(y_val, val_predictions, "Validation")

# Refit on training plus validation data, then use the test set once.
model.fit(
    np.concatenate([X_train, X_val]),
    np.concatenate([y_train, y_val]),
)
test_predictions = model.predict(X_test)
report_metrics(y_test, test_predictions, "Test")

# Compare against a mean-prediction baseline fitted on the same data.
baseline = DummyRegressor(strategy="mean")
baseline.fit(
    np.concatenate([X_train, X_val]),
    np.concatenate([y_train, y_val]),
)
baseline_predictions = baseline.predict(X_test)
print(
    "Test baseline MAE: "
    f"{mean_absolute_error(y_test, baseline_predictions):.3f}"
)

Model Selection and Reporting

Tool How it works Strengths Weaknesses When to use Cost or scale
GridSearchCV Evaluates every candidate in a finite grid. Exhaustive and deterministic. Can become expensive as the grid grows. Small parameter grids. Cost grows with candidates, folds, and data size.
RandomizedSearchCV Samples candidates from specified distributions. Efficient for large or continuous spaces. May miss the best candidate. Broad search spaces. Cost is controlled mainly by n_iter and fold count.

Compare candidates using identical splits and metrics, preferably with cross-validation distributions, confidence intervals, or paired comparisons. When the reported estimate must include hyperparameter tuning, use nested cross-validation to avoid optimistic bias; a separate untouched test set is another valid final evaluation strategy. Compare against DummyRegressor and report dataset size, split strategy, preprocessing, uncertainty, and important subgroups.

Common Pitfalls

  • Data leakage: Fitting preprocessing on the full dataset exposes validation information.

  • Test-set tuning: Repeatedly using test results for model changes invalidates the test estimate.

  • Overfitting: Excessive tuning or overly complex models may fit noise rather than generalize.

  • Invalid splitting: Random splits can leak temporal or group-related information.

  • Missing values: Imputation must be learned within each training fold.

  • Misusing MAPE: Near-zero actual values make percentage errors unstable.

  • Ignoring baselines: A complex model must outperform a simple baseline to justify its complexity.

  • Ignoring business impact: Offline MAE or RMSE may not reflect revenue, operational cost, or user experience; validate important products with online A/B tests when possible.

12

Which classification metrics should you use to evaluate a classifier?

Classification metrics measure whether a classifier separates classes correctly, but the appropriate metric depends on the relative costs of false positives and false negatives. Accuracy is suitable for balanced problems with similar error costs, while precision, recall, F1, ROC-AUC, PR-AUC, and probability-calibration metrics reveal different aspects of performance. The main trade-off is choosing an overall measure versus metrics that emphasize the errors that matter most in the application.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Split the data into training, validation, and test sets before fitting any model or preprocessing step.

  • Fit and tune models on the training data, using cross-validation where appropriate. Use validation data to select thresholds or final settings, and keep the test set untouched until the final evaluation.

  • Use StratifiedKFold for ordinary classification so folds preserve class proportions.

  • Use GroupKFold when records from the same customer, patient, or device must remain in one fold.

  • Use TimeSeriesSplit for time-ordered data; random splits can leak future information.

  • Place scalers, imputers, encoders, and feature selection inside a scikit-learn Pipeline so preprocessing is fitted only on each training fold.

Classification Metrics

Let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives.

  • Accuracy = (TP + TN) / (TP + TN + FP + FN): the proportion of all predictions that are correct. It can be misleading when the positive class is rare.

  • Precision = TP / (TP + FP): the proportion of predicted positives that are truly positive. High precision means few false alarms.

  • Recall = TP / (TP + FN): the proportion of actual positives that the model detects. High recall means few missed positives; recall is also called sensitivity.

  • F1 score = 2 * precision * recall / (precision + recall): the harmonic mean of precision and recall. It is useful when both false positives and false negatives matter, but it ignores true negatives.

  • Specificity = TN / (TN + FP): the proportion of actual negatives correctly identified.

  • ROC curve: plots recall, TP / (TP + FN), against the false-positive rate, FP / (FP + TN), across thresholds.

  • ROC-AUC = P(s_positive > s_negative) + 0.5 * P(s_positive = s_negative): the probability that a random positive receives a higher score than a random negative, with ties receiving half credit. It measures ranking quality, not probability calibration.

  • Precision-recall curve: plots precision against recall across thresholds. Its area, PR-AUC, is generally more informative than ROC-AUC for rare positive classes.

  • Average precision = sum((recall_k - recall_(k-1)) * precision_k): a weighted sum of precision at successive recall levels. average_precision_score computes this quantity; it is related to, but not identical with, trapezoidal PR-AUC.

  • Log loss = -(1/n) * sum(y * log(p) + (1-y) * log(1-p)): here, n is the sample count, y is the binary true label, and p is the predicted positive probability. It strongly penalizes confident incorrect predictions.

  • Brier score = (1/n) * sum((p - y)^2): the mean squared difference between predicted probabilities and binary outcomes. Lower is better.

  • Reliability curve: bins predictions by confidence and plots the mean predicted probability in each bin against the observed positive rate. Predictions on the diagonal are calibrated; systematic deviation indicates over- or underconfidence.

Choosing Metrics

Business requirement Recommended metrics Reason
Balanced classes and equal error costs Accuracy All four error types contribute equally
Performance across all classes matters Macro-F1 Gives each class equal weight
False alarms are expensive Precision, PR-AUC Emphasizes the quality of positive predictions
Missing positives is expensive Recall Emphasizes coverage of actual positives
Rare positive class PR-AUC, recall More informative than accuracy or ROC-AUC alone
Ranking cases by risk ROC-AUC, PR-AUC Measures performance across decision thresholds
Probability quality matters Log loss, Brier score, reliability curve Evaluates predicted probability quality and calibration

Select an operational threshold on validation data using the metric or expected error cost relevant to the application. Then report the confusion matrix, precision, recall, F1, and other selected metrics on the untouched test set.

Model Selection and Baselines

  • Compare candidate models with GridSearchCV for small parameter grids or RandomizedSearchCV for larger searches.

  • Use nested cross-validation when tuning is extensive, because repeatedly selecting models on ordinary cross-validation scores can make the reported result optimistic.

  • Compare models using paired fold differences, bootstrap confidence intervals, or suitable statistical tests rather than relying only on one aggregate score.

  • Establish a baseline with DummyClassifier, such as predicting the majority class or class priors. A more complex model should provide meaningful, consistent improvement.

  • Use classification_report for precision, recall, F1, and support, and confusion_matrix to inspect the types of errors.

  • For multiclass classification, state whether precision, recall, and F1 use macro, weighted, or micro averaging.

Common Pitfalls

  • Data leakage: fitting preprocessing before splitting allows validation or test information to influence training.

  • Test-set tuning: repeatedly changing the model or threshold after inspecting test results invalidates the final estimate.

  • Misleading accuracy: a classifier can obtain high accuracy by ignoring a rare but important class.

  • Invalid splitting: random folds are inappropriate for grouped or time-ordered observations.

  • Overfitting the validation set: repeated manual tuning turns validation data into another training resource.

  • Metric–business mismatch: offline metrics do not replace an online A/B test when the real objective is revenue, retention, safety, or another product outcome.

13

How is Mean Squared Error used in regression models?

Mean Squared Error (MSE) measures the average squared difference between a regression model’s predictions and the actual target values. It is commonly used as both a training loss and an evaluation metric, with lower values indicating better predictions. Its main trade-off is that squaring makes MSE highly sensitive to large errors and outliers.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5+ and PyTorch 2.x (Python 3.12)

Evaluation Setup

Before using MSE to compare regression models, establish a reliable evaluation process:

  1. Training set: Fit the model’s parameters, including preprocessing steps, on this data.

  2. Validation set: Use MSE to select hyperparameters, regularization strengths, and candidate algorithms.

  3. Test set: Keep this data untouched until the final model and evaluation procedure have been selected.

  4. Cross-validation: For limited data, use KFold for ordinary regression. Use GroupKFold when observations from the same subject, customer, or device must remain together. Use TimeSeriesSplit for time-ordered data because random folds can leak future information into the past.

Report the mean and variability of validation or cross-validation MSE across folds. A single validation score can be unstable, especially with small datasets. For automated tuning, GridSearchCV or RandomizedSearchCV can select models using MSE; nested cross-validation provides an honest estimate of the complete tuning procedure.

Formula and Interpretation

For n observations:

MSE = (1/n) * sum((y_i - y_hat_i)^2)

The symbols mean:

  • n: Number of evaluated observations.

  • y_i: Actual target value for observation i.

  • y_hat_i: Model prediction for observation i.

  • y_i - y_hat_i: Prediction error, also called the residual.

  • Squaring ensures that positive and negative errors do not cancel and prevents an individual error from contributing negatively.

  • Dividing by n converts the sum of squared errors into an average error.

MSE is always nonnegative. A value of zero means the predictions exactly match the evaluated targets, although this is rarely achievable on real data. Because the errors are squared, MSE has the units of the target squared.

Using MSE for Training and Evaluation

MSE is frequently used as the optimization objective for regression models:

  • Linear regression minimizes the sum of squared residuals.

  • Neural networks commonly use MSELoss as their regression loss.

  • Model comparison: Validation or cross-validation MSE can identify which candidate model generalizes best.

Minimizing training MSE reduces the average squared training error, but it does not guarantee good test performance. A flexible model may achieve excellent training MSE while overfitting. Regularization, validation, and cross-validation are therefore still necessary.

MSE is also sensitive to the target scale. If the target is divided by 10, MSE is divided by 100. This means MSE values should generally be compared only for models predicting the same target in the same units.

Comparison with Regression Metrics

Let y_bar denote the mean actual target value.

Metric Formula Main property Useful when
MSE (1/n) * sum((y_i - y_hat_i)^2) Strongly penalizes large errors Large errors are especially undesirable
MAE (1/n) * sum(abs(y_i - y_hat_i)) More robust to outliers Outliers may be noise or unimportant
RMSE sqrt(MSE) Expresses error in the target’s original units Stakeholders need an interpretable error scale
R² 1 - sum((y_i - y_hat_i)^2) / sum((y_i - y_bar)^2) Measures variance explained relative to a mean baseline Comparing models on the same dataset
MAPE (100/n) * sum(abs((y_i - y_hat_i) / y_i)) Percentage-based error Targets are positive and far from zero
Quantile loss max(q * error, (1 - q) * -error) Penalizes under- and overprediction differently by quantile q Predicting conditional quantiles

MAPE is unstable when actual values are near zero and undefined when they are zero. Quantile loss is preferable when asymmetric costs matter, such as underestimating demand being more costly than overestimating it.

Code Example

import numpy as np
from sklearn.datasets import make_regression
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Create deterministic synthetic regression data.
X, y = make_regression(
    n_samples=300,
    n_features=5,
    noise=15.0,
    random_state=42,
)

# Reserve 20% as the untouched final test set.
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

# Split the remaining data into 60% training and 20% validation.
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    random_state=42,
)

models = {
    "linear_regression": Pipeline([
        ("scaler", StandardScaler()),
        ("regressor", LinearRegression()),
    ]),
    "mean_baseline": DummyRegressor(strategy="mean"),
}

cv = KFold(n_splits=5, shuffle=True, random_state=42)
validation_mse = {}

for name, model in models.items():
    # The scorer is negative because scikit-learn selects larger scores.
    cv_result = cross_validate(
        model,
        X_train,
        y_train,
        cv=cv,
        scoring="neg_mean_squared_error",
    )

    # The pipeline fits preprocessing only on each training fold.
    model.fit(X_train, y_train)
    validation_mse[name] = mean_squared_error(
        y_val, model.predict(X_val)
    )

    print(
        f"{name}: CV MSE = {-cv_result['test_score'].mean():.3f} "
        f"(SD {cv_result['test_score'].std():.3f}), "
        f"validation MSE = {validation_mse[name]:.3f}"
    )

# Select using validation MSE, then refit on training plus validation data.
best_name = min(validation_mse, key=validation_mse.get)
selected_model = models[best_name]
X_fit = np.vstack((X_train, X_val))
y_fit = np.concatenate((y_train, y_val))
selected_model.fit(X_fit, y_fit)

# Evaluate the final selection once on the untouched test set.
test_mse = mean_squared_error(
    y_test, selected_model.predict(X_test)
)
print(f"Selected model: {best_name}")
print(f"Final test MSE = {test_mse:.3f}")

Common Pitfalls

  • Outlier sensitivity: A few extreme errors can dominate MSE. Compare it with MAE when outliers represent valid but unusual observations.

  • Data leakage: Fit scalers, imputers, feature selection, and other learned preprocessing only on training folds. Use a scikit-learn Pipeline inside cross-validation.

  • Tuning on the test set: Repeatedly selecting models using test MSE turns the test set into a validation set and produces an optimistic estimate.

  • Incorrect splits: Random splitting is inappropriate when data have temporal order or meaningful groups.

  • Ignoring baselines: Always compare against a simple baseline, such as predicting the training-set mean. A complex model is not useful if it cannot outperform that baseline.

  • Choosing only one metric: MSE answers whether large errors matter, but MAE, RMSE, R², and business-specific metrics may reveal different aspects of performance.

14

What are the benefits of AUPRC for classification models?

Area Under the Precision-Recall Curve (AUPRC) summarizes a classifier’s precision-recall trade-off across all decision thresholds. It is especially useful when the positive class is rare because it focuses on the quality of positive predictions rather than being dominated by true negatives. Its main trade-off is that it depends on class prevalence and does not measure probability calibration or directly select a deployment threshold.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5+ metric definitions (Python 3.12)

Evaluation Setup

A reliable evaluation separates data into three roles:

  • Training set: Fits model parameters and preprocessing.

  • Validation set: Selects hyperparameters, thresholds, or calibration methods.

  • Test set: Provides the final estimate and remains untouched until evaluation is complete.

For classification, use a stratified split or StratifiedKFold to preserve class proportions. Use GroupKFold when related observations belong to the same group, and TimeSeriesSplit when ordering matters. Fit every scaler, encoder, or imputer only on each training fold, normally inside a scikit-learn Pipeline, to prevent leakage.

Use GridSearchCV or RandomizedSearchCV for tuning, and nested cross-validation when the tuning procedure must itself be included in an honest performance estimate. Compare models using paired fold-level results with confidence intervals or an appropriate paired test. Include a simple DummyClassifier baseline.

What AUPRC Measures

For a chosen threshold:

  • Precision = TP / (TP + FP): the proportion of predicted positives that are actually positive.

  • Recall = TP / (TP + FN): the proportion of actual positives that the model detects.

Here, TP means true positives, FP means false positives, and FN means false negatives. The precision-recall curve plots precision against recall as the decision threshold changes.

AUPRC = ∫ Precision® dRecall®

This integral means the area accumulated as recall changes; r denotes recall. Two related quantities are commonly reported:

  • Trapezoidal AUPRC: The area under the precision-recall curve, computable in scikit-learn with sklearn.metrics.auc.

  • Average precision (AP): A step-wise summary computed by sklearn.metrics.average_precision_score.

AP is often informally called “AUPRC,” but it need not equal the trapezoidal area. State which definition is being used.

Why It Helps

Metric Main emphasis Important limitation
Accuracy Overall correctness Can be misleading with imbalanced classes
ROC-AUC Ranking across all thresholds May appear strong because true negatives dominate
AUPRC / AP Positive-class precision and recall Depends on prevalence and the positive-class definition
F1 One selected precision-recall balance Depends strongly on the threshold

AUPRC is useful for comparing models across thresholds, identifying models that perform well on rare positives, and exposing trade-offs hidden by accuracy. For a random ranking, the expected AP equals the positive-class prevalence, so substantially higher AP is meaningful. AUPRC values should not be compared directly across datasets with different prevalences.

Related Metrics and Calibration

For binary classification, the principal metrics are:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN), where TN means true negatives.

  • F1 = 2 × Precision × Recall / (Precision + Recall), a harmonic mean of precision and recall.

  • Specificity = TN / (TN + FP), the proportion of actual negatives correctly identified.

  • ROC-AUC: Area under the curve of true-positive rate versus false-positive rate, where false-positive rate = FP / (FP + TN).

  • PR-AUC: Area under the precision-recall curve; generally more informative than ROC-AUC when positives are rare.

  • Log loss = -(1/N) × Σ[yᵢ log(pᵢ) + (1 - yᵢ) log(1 - pᵢ)], where N is the number of examples, yᵢ is the true binary label, and pᵢ is the predicted probability of the positive class. It penalizes confident incorrect predictions.

  • Brier score = (1/N) × Σ(pᵢ - yᵢ)², which summarizes squared probability error.

AUPRC does not establish calibration. A reliability curve, which compares mean predicted probabilities with observed event frequencies within probability bins, should be used when calibration matters.

Common Pitfalls

  • Do not choose a threshold or tune hyperparameters using the test set; perform selection inside validation or cross-validation.

  • Do not report only accuracy for imbalanced data; include precision, recall, F1, PR-AUC, and a confusion matrix.

  • Do not use random splits for temporal or grouped observations.

  • Do not compare AUPRC values across datasets without accounting for prevalence.

  • Do not treat AP and trapezoidal AUPRC as interchangeable.

  • Do not mistake a strong offline PR curve for guaranteed business value; validate the final policy with an A/B test when appropriate.

15

How do macro-average and micro-average differ in classification metrics?

Macro-average computes each class metric separately and averages them with equal class weight; micro-average pools class-level counts before computing the metric. For precision, recall, and F1, macro-averaging gives every class equal importance, while micro-averaging gives more influence to classes with more examples. The main trade-off is fairness to minority classes versus overall frequency-weighted performance.

Read full answer

Applies toclassification metrics across all versions (scikit-learn 1.5+ conventions, Python 3.12)

Evaluation Setup

Establish the evaluation protocol before comparing averaging methods:

  • Split data into training, validation, and test sets. Fit on training data, select models and thresholds on validation data, and keep the test set untouched until final evaluation.

  • Use KFold for general data, StratifiedKFold for imbalanced classification, GroupKFold when observations are grouped, and TimeSeriesSplit for temporal data.

  • For nested cross-validation, use inner folds for hyperparameter tuning and outer folds for an honest estimate that accounts for that tuning.

  • Compare against a baseline such as DummyClassifier.

Core Formulas

For class c, let TP_c and TN_c be true positives and true negatives, and FP_c and FN_c be false positives and false negatives. In one-vs-rest calculations:

  • Accuracy: Accuracy_c = (TP_c + TN_c) / (TP_c + TN_c + FP_c + FN_c)

  • Precision: Precision_c = TP_c / (TP_c + FP_c); measures how many predicted positives are correct.

  • Recall: Recall_c = TP_c / (TP_c + FN_c); measures how many actual positives were detected.

  • Specificity: Specificity_c = TN_c / (TN_c + FP_c); measures how well actual negatives were identified.

  • F1: F1_c = 2 × Precision_c × Recall_c / (Precision_c + Recall_c); the harmonic mean of precision and recall.

With C classes, macro-averaging gives each class equal weight:

  • Macro-Precision = sum(Precision_c) / C

  • Macro-Recall = sum(Recall_c) / C

  • Macro-F1 = sum(F1_c) / C

Micro-averaging combines counts first:

  • Micro-Precision = sum(TP_c) / sum(TP_c + FP_c)

  • Micro-Recall = sum(TP_c) / sum(TP_c + FN_c)

  • Micro-F1 is calculated from those micro-precision and micro-recall values.

Worked Example

Consider two classes with consistent one-vs-rest counts from a 160-example dataset:

Class TP FP FN TN Precision Recall F1
A 80 10 20 50 0.889 0.800 0.842
B 50 20 10 80 0.714 0.833 0.769

Class A contains 100 examples and class B contains 60. The macro-averaged F1 is:

(0.842 + 0.769) / 2 = 0.806

For micro-averaging, the pooled counts are TP = 130, FP = 30, and FN = 30:

  • Micro-Precision = 130 / (130 + 30) = 0.8125

  • Micro-Recall = 130 / (130 + 30) = 0.8125

  • Micro-F1 = 0.8125

Macro-averaging gives the smaller class B equal importance, while micro-averaging gives the larger class A more influence.

Other Classification Metrics

For binary classification, the ROC curve plots true-positive rate, TPR = TP / (TP + FN), against false-positive rate, FPR = FP / (FP + TN). ROC-AUC is the area under that curve.

The precision-recall curve plots precision and recall across thresholds. PR-AUC is generally more informative than ROC-AUC when the positive class is rare. In binary classification, evaluate PR-AUC for the positive class; in multiclass settings, state whether class-wise results are macro- or support-weighted.

For multiclass ROC-AUC in scikit-learn 1.5+, roc_auc_score(..., multi_class="ovr", average="macro") averages class-wise one-vs-rest AUC values; multiclass average="micro" is not supported.

Log loss penalizes incorrect probability estimates:

LogLoss = -(1 / N) × sum(y × log(p) + (1 - y) × log(1 - p))

Here, N is the number of observations, y is the true binary label, and p is the predicted probability. For multiclass classification, log loss uses categorical cross-entropy. The Brier score is:

BrierScore = (1 / N) × sum((p - y)^2)

Lower values are better. A reliability curve compares predicted probabilities with observed positive frequencies within probability bins; a poorly calibrated model’s scores may not represent real-world event probabilities.

Choosing an Average

Goal Preferred average Reason
Give every class equal importance Macro Exposes weaknesses in minority classes
Measure aggregate precision, recall, or F1 Micro Reflects instance frequency
Balance these concerns Weighted Weights classes by their support
Evaluate rare-positive detection Positive-class PR-AUC and class-wise precision/recall Focuses on the relevant class
Evaluate calibration Class-wise Brier scores and reliability curves Reveals systematic over- or under-confidence

Select any decision threshold using validation data or cross-validation, never the test set.

Common Pitfalls

  • Using only accuracy: a majority-class predictor can appear strong on imbalanced data.

  • Omitting the averaging convention: always report whether scores are macro, micro, or weighted and include class support when useful.

  • Tuning on the test set: model, threshold, and averaging-method choices must precede final test evaluation.

  • Data leakage: fit preprocessing within each training fold using Pipeline or ColumnTransformer.

  • Ignoring operational costs: offline F1 or accuracy may not match the business objective; use an A/B test when appropriate.

16

How should you read a model’s calibration curve?

A calibration curve, also called a reliability curve, compares predicted probabilities with the observed frequency of each outcome. A model is well calibrated in aggregate when its curve follows the diagonal y = x: predictions near 0.20 should produce positive outcomes about 20% of the time. The main trade-off is that detailed calibration estimates require many held-out observations, while fine-grained bins can be noisy; calibration also measures probability quality rather than ranking performance.

Read full answer

Applies tomachine learning concepts (scikit-learn 1.5+ with Python 3.12)

Evaluation Setup

  • Keep a test set untouched until model training, preprocessing, and threshold selection are complete.

  • Fit preprocessing only on training folds or the training set using a scikit-learn Pipeline to prevent data leakage.

  • Use StratifiedKFold for classification cross-validation, GroupKFold for grouped observations, and TimeSeriesSplit for time-ordered data.

  • Generate the calibration curve from held-out predictions, such as predictions on the test set or out-of-fold predictions.

  • A calibration curve does not show whether the model ranks examples well. A model can be calibrated but inaccurate, or highly discriminative but poorly calibrated.

Reading the Curve

The predicted probability is usually placed on the x-axis and the observed positive rate on the y-axis for each probability bin.

Curve position Interpretation Typical model behavior
Close to y = x Well calibrated in that region A predicted 30% risk occurs about 30% of the time
Above the diagonal Underconfident Observed outcomes are more frequent than predicted
Below the diagonal Overconfident Predicted probabilities are too high relative to observed outcomes
Near a horizontal prevalence line Little use of predicted probabilities The model may be calibrated on average but lack discrimination
Large bin-to-bin deviations Local miscalibration Some probability ranges are systematically unreliable

Perfect calibration does not require every bin to touch the diagonal exactly. Because outcomes are discrete, finite samples introduce sampling variation, especially for rare outcomes or narrow bins.

Calibration Metrics

  • Brier score: Measures the mean squared difference between predicted probabilities and binary outcomes:

Brier score = (1/n) * sum((p_i - y_i)^2)

Here, n is the number of observations, p_i is the predicted probability for observation i, and y_i is its binary outcome. Lower values are better. - Log loss: Penalizes confident incorrect predictions more heavily. It is useful for comparing probabilistic classifiers but is sensitive to extreme, incorrect probabilities. - Reliability diagram: Shows where miscalibration occurs and is more diagnostically useful than a single aggregate score. - Accuracy measures correct classifications rather than probability quality. A model with good accuracy can still assign misleading probabilities. - ROC-AUC measures ranking or discrimination, while PR-AUC is often more informative for rare positive classes. Neither directly establishes calibration.

Common Pitfalls

  • Using training-set predictions: Training predictions are usually overconfident and do not represent unseen-data behavior.

  • Choosing thresholds on the test set: Select thresholds using validation or out-of-fold predictions, then evaluate once on the test set.

  • Ignoring sample size: Sparse probability bins may appear miscalibrated because of random variation.

  • Interpreting calibration as accuracy: Calibration says whether stated probabilities are trustworthy; accuracy says whether class decisions are correct.

  • Comparing models with only one split: Prefer repeated or cross-validated estimates, baseline models such as DummyClassifier, and confidence intervals when comparing candidates.

  • Overfitting to validation data: Reusing the validation set extensively for tuning can bias its results; nested cross-validation provides a more honest estimate.

17

What is the Brier score, and when should you use it?

The Brier score measures the mean squared difference between predicted probabilities and observed outcomes. It is useful when probability quality matters, such as when forecasts drive risk estimates, expected benefits, or operational decisions. Its main trade-off is that it is simple and interpretable, but it combines different kinds of probability error into one number and is sensitive to class prevalence.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); the mathematical definition applies to all versions

Evaluation Setup

A sound evaluation procedure keeps the test data untouched until the final evaluation:

  1. Split the data into training, validation, and test sets. The test set estimates performance only after the modeling procedure is fixed.

  2. Fit preprocessing, feature selection, and the model using only the training portion. With cross-validation, perform these operations separately within each training fold.

  3. Use the validation data or cross-validation results to choose models, hyperparameters, probability thresholds, and calibration procedures.

  4. If probabilities are calibrated, fit the calibrator on a separate calibration set or out-of-fold predictions generated without access to the test set.

  5. Evaluate the final procedure on the test set once, after all choices have been made.

Use KFold for general regression, StratifiedKFold for classification with imbalanced class frequencies, GroupKFold when observations from the same group must remain together, and TimeSeriesSplit when temporal order matters.

How the Brier Score Works

For binary classification, the formula is:

Brier score = (1 / N) * sum((p_i - y_i)^2)

Here:

  • N is the number of evaluated observations.

  • i indexes an observation.

  • p_i is the predicted probability that observation i belongs to the positive class.

  • y_i is its observed label: 1 for a positive outcome and 0 for a negative outcome.

Lower scores are better. For binary outcomes, the score ranges from 0 to 2:

  • 0 means every predicted probability exactly matches its observed outcome.

  • 1 results from always predicting 0.5, regardless of the observation.

  • 2 results from completely confident predictions that are all wrong.

For multiclass classification, the multiclass form is:

Brier score = (1 / N) * sum(sum((p_ic - y_ic)^2))

Here:

  • i again indexes an observation.

  • c indexes one of the classes.

  • p_ic is the predicted probability assigned to class c for observation i.

  • y_ic is 1 when class c is the true class for observation i, and 0 otherwise.

The Brier score is a proper scoring rule: it rewards realistic probabilities and penalizes confident errors. It is not, however, a pure calibration diagnostic. A reliability curve is useful alongside it because it reveals how predicted probabilities compare with observed frequencies across probability ranges.

When to Use It

Use the Brier score when:

  • Probabilities drive decisions: For example, when probabilities feed an expected-benefit or expected-cost calculation.

  • Calibration matters: When assigning 0.8 should correspond to an approximately 80% event frequency.

  • Models are compared probabilistically: It provides one summary measure of squared probability error on the same evaluation set.

  • Calibration is being assessed: It can show whether post-hoc calibration improved probability accuracy.

  • Multiclass probabilities are evaluated: The multiclass form assesses the complete probability vector.

  • A baseline is needed: Compare against a prior-probability baseline, such as scikit-learn’s DummyClassifier(strategy="prior"), using the same held-out examples.

Interpret comparisons cautiously when class prevalence differs: the score depends on the event distribution, so scores from datasets with substantially different prevalences are not directly comparable. The Brier score is also less appropriate when only ranking quality matters and the probability magnitudes will not be used.

Comparison with Other Metrics

Metric What it measures Important limitation
Brier score Squared probability error and overall probabilistic quality Does not identify which observations caused errors and is sensitive to prevalence
Log loss Probability accuracy with strong penalties for confident mistakes Can be dominated by a few extreme errors
ROC-AUC Ranking positives above negatives Does not assess whether probabilities are realistic
PR-AUC Ranking performance for the positive class Does not directly measure calibration; generally more informative than ROC-AUC for rare positives
Accuracy Proportion of correctly classified labels Ignores confidence and can mislead with imbalanced classes
Reliability curve Observed frequencies within probability ranges Diagnostic rather than a single summary score

Accuracy can be misleading when the positive class is rare, while ROC-AUC can look optimistic in the same setting. PR-AUC is usually more informative for rare positives, and none of these ranking metrics replaces the Brier score when probability magnitudes matter.

Common Pitfalls

  • Using hard labels: Passing predict() output rather than predict_proba() discards the uncertainty the Brier score is intended to evaluate.

  • Ignoring the baseline: Compare against a prior-probability model on the same evaluation data.

  • Calibration leakage: Fit calibration on held-out or out-of-fold predictions rather than the test set.

  • Tuning on the test set: Selecting models, thresholds, or calibration methods using test results produces an optimistic estimate.

  • Ignoring prevalence: Compare scores only on data with a comparable target distribution unless prevalence shift is explicitly modeled.

  • Using invalid splits: Random splits can leak information in temporal or grouped datasets.

  • Confusing probability quality with ranking: A model can rank cases well but assign poor probabilities, or assign useful probabilities while ranking poorly.

  • Ignoring business costs: Offline metrics may not reflect the real objective; validate important changes with an A/B test when possible.

18

How would you use bootstrapping for model evaluation?

Bootstrapping estimates evaluation uncertainty by repeatedly resampling observations with replacement, refitting models, and evaluating the resulting predictions. It supports out-of-bag estimation, confidence intervals, and paired model comparisons, but ordinary bootstrapping mainly assumes independent, identically distributed observations.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12), NumPy 2.x

Evaluation Setup

  • Keep a final test set untouched. Fit preprocessing, tune hyperparameters, and perform internal validation using only the training portion, preferably with pipelines.

  • Use KFold for regression or ordinary classification, StratifiedKFold to preserve class proportions, GroupKFold for related observations, and TimeSeriesSplit for temporal data. Random bootstrapping is inappropriate for time-ordered or grouped observations without an adapted resampling scheme.

  • Use GridSearchCV or RandomizedSearchCV; use nested cross-validation when tuning occurs and the final performance estimate must remain honest.

  • Compare against a baseline such as DummyClassifier.

Bootstrap Procedure

  1. Draw a bootstrap sample with the same number of observations as the training data, sampling with replacement.

  2. Fit the complete training pipeline on that sample.

  3. Evaluate it on observations excluded from the sample, called out-of-bag (OOB) observations.

  4. Repeat and aggregate the metric estimates.

A percentile interval uses the 2.5th and 97.5th percentiles of the bootstrap estimates. For paired comparisons, bootstrap the same cases for both models and calculate each metric difference, preserving the pairing.

Bootstrapping a fixed model’s test predictions estimates finite-test-set sampling uncertainty only. Training uncertainty requires resampling the training data and refitting the models. Bagging training cost is approximately (B) times one model fit for (B) estimators and is usually CPU-bound and parallelizable.

Metrics and Calibration

Metric Formula
Accuracy (TP + TN) / N
Precision TP / (TP + FP)
Recall TP / (TP + FN)
F1 2 * Precision * Recall / (Precision + Recall)
Specificity TN / (TN + FP)
Log loss -1/N * sum(y*log(p) + (1-y)*log(1-p))
Brier score 1/N * sum((p-y)^2)

Here, TP and TN are true positives and true negatives, FP and FN are false positives and false negatives, y is the binary label, p is the predicted probability, and N is the number of observations.

  • ROC-AUC measures ranking across thresholds using true-positive rate versus false-positive rate.

  • PR-AUC or average precision is usually more informative for rare positives; accuracy can be misleading under imbalance.

  • A reliability curve compares predicted probabilities with observed frequencies. Bootstrap intervals for calibration metrics such as the Brier score assess their stability.

  • For regression, bootstrap MAE, MSE, RMSE, R², MAPE, and quantile or pinball loss. MAPE is unstable when actual values are near zero, so MAE or another scale-appropriate loss may be preferable.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import BaggingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Original target 0 is malignant; make malignant the positive class.
data = load_breast_cancer()
X = data.data
y = (data.target == 0).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

# Each bagging estimator fits its scaler only on its own bootstrap sample.
base_estimator = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

model = Pipeline([
    ("bagging", BaggingClassifier(
        estimator=base_estimator,
        n_estimators=200,
        bootstrap=True,
        max_samples=1.0,
        oob_score=True,
        random_state=42,
    ))
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring={
        "accuracy": "accuracy",
        "roc_auc": "roc_auc",
        "average_precision": "average_precision",
        "f1": "f1",
    },
)

for name, values in scores.items():
    if name.startswith("test_"):
        print(f"{name}: {values.mean():.3f} +/- {values.std():.3f}")

model.fit(X_train, y_train)
bagging = model.named_steps["bagging"]

# OOB predictions are used to develop the threshold, not as a final estimate.
oob_scores = bagging.oob_decision_function_[:, 1]
precision, recall, thresholds = precision_recall_curve(y_train, oob_scores)
denominator = precision[:-1] + recall[:-1]
f1_values = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
best_index = int(np.argmax(f1_values))
threshold = float(thresholds[best_index])

# Final evaluation on the untouched test set.
test_scores = model.predict_proba(X_test)[:, 1]
test_predictions = (test_scores >= threshold).astype(int)

print(classification_report(
    y_test, test_predictions, target_names=["benign", "malignant"]
))
print(confusion_matrix(y_test, test_predictions))
print("Test accuracy:", accuracy_score(y_test, test_predictions))
print("Test ROC-AUC:", roc_auc_score(y_test, test_scores))
print("Test PR-AUC (average precision):",
      average_precision_score(y_test, test_scores))

# Fixed-model bootstrap interval for test-set F1.
rng = np.random.default_rng(42)
bootstrap_f1 = []

for _ in range(2000):
    indices = rng.choice(len(y_test), size=len(y_test), replace=True)
    predictions = (test_scores[indices] >= threshold).astype(int)
    bootstrap_f1.append(
        f1_score(y_test[indices], predictions, zero_division=0)
    )

lower, upper = np.percentile(bootstrap_f1, [2.5, 97.5])
print(f"Bootstrap 95% F1 interval: [{lower:.3f}, {upper:.3f}]")

Common Pitfalls

  • Data leakage: Fit preprocessing within each training fold or bootstrap estimator.

  • Invalid independence assumptions: Adapt resampling for grouped or temporal observations.

  • Optimistic OOB results: Do not repeatedly tune against and report the same OOB predictions; use independent validation or nested procedures.

  • Wrong metric: Use PR-AUC, recall, or F1 for rare positives rather than accuracy alone.

  • Business mismatch: Validate offline metrics against online objectives, ideally through an A/B test.

19

When should the Matthews Correlation Coefficient be used?

The Matthews Correlation Coefficient (MCC) is a classification metric that uses all four confusion-matrix outcomes to measure the association between predicted and actual labels. It is especially useful for imbalanced binary classification, where accuracy can be misleading. Its main trade-off is that it provides a balanced summary but is less directly tied to business costs than precision or recall and still depends on the classification threshold.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Establish the evaluation design before calculating MCC:

  • Training set: Fit model parameters and preprocessing.

  • Validation set: Select hyperparameters and decision thresholds.

  • Test set: Use only once for the final unbiased evaluation.

  • For classification, use StratifiedKFold to preserve class proportions. Use GroupKFold for related observations and TimeSeriesSplit for temporal data.

  • Fit scalers, encoders, imputers, and feature selection inside a scikit-learn Pipeline so they are learned only from each training fold. Fitting them on the entire dataset causes leakage.

  • Compare the model with a baseline such as DummyClassifier. For model selection, tune inside cross-validation; use nested cross-validation when an honest estimate of the entire tuning procedure is required.

When MCC Is Appropriate

Use MCC when:

  • The positive class is rare: MCC considers true positives, true negatives, false positives, and false negatives instead of relying mainly on the dominant class.

  • You need one balanced metric: It reflects both types of classification errors rather than emphasizing only precision or recall.

  • Comparing binary classifiers across datasets or folds: Its standardized range makes comparisons easier than raw error counts.

  • Evaluating multiclass classification: MCC can aggregate the multiclass confusion matrix into one score, although this does not reveal which individual class caused errors.

  • Comparing fixed decision thresholds: MCC can be calculated from predictions produced by a threshold chosen on validation data.

Prefer other metrics when costs are asymmetric. For example, use recall when missing a positive case is much more costly, and precision when every predicted positive requires manual review. MCC does not directly encode these business costs.

How MCC Works

For binary classification:

MCC = (TP × TN − FP × FN) / sqrt((TP + FP) × (TP + FN) × (TN + FP) × (TN + FN))

Here:

  • TP is the number of true positives.

  • TN is the number of true negatives.

  • FP is the number of false positives.

  • FN is the number of false negatives.

The MCC ranges from -1 to +1:

  • +1: perfect prediction.

  • 0: no association between predictions and actual labels.

  • -1: complete disagreement.

The numerator rewards correct positive and negative predictions while penalizing the two kinds of errors. The denominator normalizes the result. If the denominator is zero, such as when predictions contain only one class, the coefficient is mathematically undefined; scikit-learn’s matthews_corrcoef returns 0.0 in that case.

Related Classification Metrics

Metric Formula or definition Main use
Accuracy (TP + TN) / N Overall correctness; misleading with imbalance
Precision TP / (TP + FP) Correctness of predicted positives
Recall TP / (TP + FN) Coverage of actual positives
Specificity TN / (TN + FP) Coverage of actual negatives
F1 2 × precision × recall / (precision + recall) Balance between precision and recall
MCC Confusion-matrix formula above Balanced evaluation, especially with imbalance
ROC-AUC Area under TPR versus FPR across thresholds Threshold-independent ranking quality
PR-AUC Area under precision versus recall across thresholds Often more informative for rare positives
Log loss -1/N × sum(y log(p) + (1-y) log(1-p)) Quality of predicted probabilities
Brier score 1/N × sum((p - y)^2) Probability accuracy and calibration

N is the number of samples, y is the actual binary label, and p is the predicted probability of the positive class. A reliability curve compares average predicted probabilities with observed positive frequencies within probability bins.

Accuracy can look excellent when a model predicts only the majority class. MCC avoids that specific distortion, but it does not assess probability calibration by itself; use log loss, the Brier score, and reliability curves for calibration.

Practical Guidance

  • Select a threshold using validation or out-of-fold predictions, not the test set. precision_recall_curve can support this decision.

  • Report MCC alongside precision, recall, F1, the confusion matrix, and the class distribution.

  • Compare MCC across folds and inspect the variation rather than relying on one aggregate number.

  • Use confidence intervals or paired comparisons when deciding whether model differences are meaningful.

  • For production decisions, verify that the offline metric aligns with the online objective, ideally through an A/B test.

Common Pitfalls

  • Using accuracy for imbalanced data: A high accuracy may hide complete failure on the minority class.

  • Tuning thresholds or hyperparameters on the test set: This contaminates the final estimate.

  • Leaking preprocessing information: Fit transformations inside each training fold.

  • Treating MCC as threshold-independent: It changes when the prediction threshold changes.

  • Ignoring probability quality: MCC evaluates hard class predictions, not calibrated probabilities.

  • Using random splits for grouped or temporal data: Use GroupKFold or TimeSeriesSplit instead.

  • Overlooking a degenerate predictor: Always inspect the confusion matrix and class distribution alongside MCC.

20

How do model evaluation metrics trade off against each other?

Model evaluation metrics measure different properties of a model’s errors, so the appropriate choice depends on the costs of false positives, false negatives, and regression errors. The main trade-off is that some metrics reflect threshold-dependent outcomes, while others assess ranking, probability quality, calibration, or overall numerical error.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12; NumPy 2.x)

Evaluation Setup

  • Train, validation, and test sets: Fit models on training data, use validation data or cross-validation for model and threshold selection, and keep the test set untouched until final evaluation.

  • Cross-validation: Use KFold for general data, StratifiedKFold for classification, GroupKFold when observations belong to related groups, and TimeSeriesSplit when temporal order matters.

  • Leakage prevention: Place scalers, imputers, encoders, and feature selection inside a Pipeline so they are fitted only on each training fold.

  • Baseline: Compare against a simple model such as DummyClassifier; a complex model is not useful unless it consistently outperforms the baseline.

Classification Metrics

Here, TP means true positives, TN true negatives, FP false positives, and FN false negatives.

  • Accuracy = (TP + TN) / (TP + TN + FP + FN): Measures overall correctness, but can be misleading when classes are imbalanced.

  • Precision = TP / (TP + FP): Measures how many predicted positives are correct. Favor it when false positives are expensive.

  • Recall = TP / (TP + FN): Measures how many actual positives are detected. Favor it when false negatives are expensive.

  • F1 score = 2 * precision * recall / (precision + recall): Balances precision and recall, but ignores true negatives and does not reflect probability quality.

  • Specificity = TN / (TN + FP): Measures how well negatives are identified; it is useful when false positives matter.

  • ROC curve and ROC-AUC: The ROC curve plots true-positive rate, or recall, against false-positive rate across classification thresholds. ROC-AUC summarizes the model’s ranking across thresholds, but a high value can obscure poor precision at operationally important thresholds in severely imbalanced data.

  • Precision-recall curve and PR-AUC: The precision-recall curve plots precision against recall across thresholds. PR-AUC focuses on positive-class performance and is generally more informative when positives are rare; average_precision_score summarizes this curve but is not identical to trapezoidal area under the curve.

  • Log loss = -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)): Here, n is the number of samples, y_i is the binary label, and p_i is the predicted positive probability. It strongly penalizes confident incorrect probabilities.

  • Brier score = (1/n) * sum((y_i - p_i)^2): Measures probability error; lower is better. Reliability curves compare predicted probabilities with observed positive rates.

Metric Best suited to Main limitation
Accuracy Balanced classes and equal error costs Hides minority-class failures
Precision False positives are costly Can achieve low recall
Recall False negatives are costly Can create many false positives
F1 Precision and recall both matter Ignores true negatives
ROC-AUC Comparing ranking quality May obscure rare-positive performance
PR-AUC Rare-positive detection Depends on prevalence and threshold range
Log loss Comparing probability quality Sensitive to confident mistakes
Brier score Assessing probability error Does not distinguish error types

Regression Metrics

For n observations, actual values y_i, and predictions y_hat_i:

  • MAE = (1/n) * sum(|y_i - y_hat_i|): Interpretable average error, but less sensitive to large outliers.

  • MSE = (1/n) * sum((y_i - y_hat_i)^2): Penalizes large errors heavily.

  • RMSE = sqrt(MSE): Expresses error in the target’s units and retains outlier sensitivity.

  • R² = 1 - SSE / SST: Compares performance with a baseline prediction. Here, SSE is residual error and SST is total target variation; R² can be negative.

  • MAPE = (100%/n) * sum(|y_i - y_hat_i| / |y_i|): Uses percentage errors but is unstable or undefined when actual values are near zero.

  • Quantile loss: For one observation, L_tau(y_i, y_hat_i) = max(tau * (y_i - y_hat_i), (1 - tau) * (y_hat_i - y_i)). Here, tau is the target quantile, y_i is the actual value, and y_hat_i is the prediction; losses are usually averaged across observations. Underprediction is weighted by tau, while overprediction is weighted by 1 - tau, supporting asymmetric quantile objectives.

Model Selection

  • GridSearchCV evaluates every specified combination and is appropriate for small search spaces.

  • RandomizedSearchCV samples configurations and is usually more efficient for large hyperparameter spaces.

  • Nested cross-validation uses an inner loop for tuning and an outer loop for unbiased performance estimation.

  • Compare candidates using paired fold-level differences, bootstrap confidence intervals, or suitable statistical tests rather than relying on one split.

Common Pitfalls

  • Selecting hyperparameters or thresholds using the test set causes optimistic bias.

  • Reporting accuracy alone can conceal poor minority-class performance.

  • Randomly splitting time-series or grouped data creates leakage.

  • Offline metrics may not represent online impact; validate important product changes with an A/B test when possible.

  • Interviewers commonly ask why a metric fits the business objective, how thresholds affect results, and how you would estimate uncertainty.

21

What does a p-value mean in model evaluation?

A p-value is the probability of observing a test statistic at least as extreme as the one measured, assuming the null hypothesis is true. In model evaluation, it helps assess whether an observed result—such as a performance difference or coefficient—is statistically distinguishable from random variation. Its main trade-off is that it measures evidence against the null hypothesis, not the size or practical importance of the effect.

Read full answer

Applies tostatistical inference in machine learning model evaluation (Python 3.12; scikit-learn 1.5+ workflows)

Evaluation Setup

Before interpreting a p-value, establish a sound evaluation design:

  • Training set: Used to fit the model and learn preprocessing steps.

  • Validation set: Used for hyperparameter selection, threshold selection, and model comparison.

  • Test set: Kept untouched until final evaluation.

  • Cross-validation: Uses KFold for general data, StratifiedKFold for imbalanced classification, GroupKFold when observations from the same group are related, and TimeSeriesSplit for time-ordered data.

A p-value cannot correct for leakage or an improperly designed split. For example, selecting a model using test-set results and then computing a p-value from those results does not produce an honest final estimate.

Metrics and Statistical Inference

Evaluation metrics describe performance, while p-values assess uncertainty under a stated null hypothesis.

Metric Formula Meaning
Accuracy (TP + TN) / (TP + TN + FP + FN) Fraction classified correctly; misleading with imbalanced classes
Precision TP / (TP + FP) Fraction of predicted positives that are truly positive
Recall TP / (TP + FN) Fraction of actual positives detected
F1 2 * precision * recall / (precision + recall) Harmonic mean of precision and recall
Specificity TN / (TN + FP) Fraction of actual negatives correctly identified
ROC-AUC Area under the TPR-versus-FPR curve Ranking ability across thresholds
PR-AUC Area under the precision-versus-recall curve Often more informative when positives are rare
Log loss -1/n * sum(y * log(p) + (1 - y) * log(1 - p)) Penalizes confident incorrect probabilities
Brier score 1/n * sum((y - p)^2) Mean squared probability error; useful for calibration

Here, TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives. n is the number of examples, y is the true binary label, and p is the predicted probability.

Meaning and Formula

For a test statistic T, the p-value is:

p-value = P(T is at least as extreme as the observed value | H0 is true)

P denotes probability, T is the test statistic, H0 is the null hypothesis, and “at least as extreme” includes outcomes in the direction specified by the test, or both directions for a two-sided test.

If p-value <= alpha, the result is statistically significant at level alpha, commonly 0.05. This does not mean there is an alpha probability that H0 is true. It means that, if H0 were true, obtaining a result this extreme would be relatively unusual.

Applying P-Values to Models

P-values may be used to test specific statistical claims, such as:

  • Feature significance: H0: the coefficient for a feature is zero.

  • Model comparison: H0: model A and model B have equal expected performance.

  • Threshold or intervention effects: H0: the change has no effect on the target metric.

For model comparisons, evaluate both models on the same folds or examples and use an appropriate paired test, bootstrap, or permutation procedure. Cross-validation folds are related, so their scores should not automatically be treated as independent observations. A confidence interval is usually more informative than a p-value because it communicates the estimated effect and its uncertainty.

Limitations and Alternatives

  • A p-value does not measure effect size: a very small p-value can accompany a practically negligible improvement.

  • Statistical significance does not guarantee deployment value; consider business impact, cost, latency, and robustness.

  • Many p-values increase false-positive risk when many features, models, or metrics are tested. Use multiple-testing controls such as the Benjamini-Hochberg procedure when appropriate.

  • For predictive modeling, report cross-validated metrics with uncertainty intervals rather than relying only on a single significance test.

  • Calibration should be checked with reliability curves and probability metrics such as the Brier score.

Common Pitfalls

  • Interpreting p = 0.03 as “there is a 3% probability the null hypothesis is true.”

  • Using accuracy alone on severely imbalanced data.

  • Performing preprocessing before splitting the data, causing data leakage.

  • Tuning hyperparameters or thresholds on the test set.

  • Using random splits for time-series or grouped observations.

  • Reporting a p-value without the effect size, confidence interval, metric, and experimental design.

  • Assuming a significant result is operationally important.

Interviewers commonly ask: “What was your null hypothesis?”, “How did you avoid leakage?”, “Why is this metric appropriate?”, and “How would you distinguish statistical significance from business significance?”

22

How does an ROC curve evaluate classifier discrimination?

A receiver operating characteristic (ROC) curve evaluates classifier discrimination by plotting the true-positive rate against the false-positive rate as the decision threshold changes. It measures ranking quality across thresholds rather than probability calibration; its main limitation is that ROC-AUC can obscure poor performance at a required operating threshold and can appear overly optimistic under severe class imbalance.

Read full answer

Applies toscikit-learn 1.5.1+ with NumPy 2.x (Python 3.12)

Evaluation Setup

  • Reserve a final test set and keep it untouched until model and threshold selection are complete.

  • Use cross-validation within the development set: KFold generally, StratifiedKFold for classification, GroupKFold for grouped observations, and TimeSeriesSplit for time-ordered data.

  • For classification, stratification helps preserve class proportions. Random splitting is inappropriate when future observations or repeated subjects could leak into training folds.

How the ROC Curve Works

For binary labels, define TP, TN, FP, and FN as true positives, true negatives, false positives, and false negatives. At each classification threshold:

TPR = TP / (TP + FN) and FPR = FP / (FP + TN)

Plotting TPR against FPR produces the ROC curve. The diagonal represents random ranking, so points above it indicate better-than-random discrimination.

ROC-AUC summarizes this ordering:

ROC-AUC = P(score of a randomly selected positive > score of a randomly selected negative)

Ties receive half credit. Thus, 0.5 represents random ranking and 1.0 perfect ranking. ROC-AUC does not measure calibration or guarantee useful performance at a particular threshold.

Metrics and Calibration

Let N be the number of observations, y_i the binary label, and p_i the predicted probability of the positive class.

Metric Formula or definition Main question
Accuracy (TP + TN) / N Are predictions correct overall?
Precision TP / (TP + FP) How many predicted positives are correct?
Recall TP / (TP + FN) How many actual positives were found?
Specificity TN / (TN + FP) How many actual negatives were rejected?
F1 2 * precision * recall / (precision + recall) How well do precision and recall balance?
PR-AUC Area under the precision-recall curve How useful are ranked predictions for the positive class?
Log loss -1/N * sum(y_i * log(p_i) + (1-y_i) * log(1-p_i)) How accurate are predicted probabilities?
Brier score 1/N * sum((p_i - y_i)^2) How close are predicted probabilities to outcomes?

A precision-recall curve plots precision against recall and is often more informative for rare positive classes. average_precision_score computes average precision, a commonly used PR summary that is related to, but not identical to, the trapezoidal area under the PR curve.

Calibration is evaluated separately using the Brier score and reliability curves, which compare predicted probabilities with observed event frequencies.

Model Selection

  • Compare candidates with GridSearchCV or RandomizedSearchCV, placing preprocessing in a Pipeline.

  • Include a DummyClassifier baseline.

  • Use nested cross-validation when tuning models and estimating the complete selection procedure without optimistic bias.

  • Select the operating threshold using validation predictions and the business requirement.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    auc,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# The test set remains untouched until final evaluation.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.30, stratify=y, random_state=42
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
])

cv_results = cross_validate(
    model,
    X_dev,
    y_dev,
    cv=cv,
    scoring={
        "accuracy": "accuracy",
        "roc_auc": "roc_auc",
        "average_precision": "average_precision",
        "neg_log_loss": "neg_log_loss",
    },
)

# Out-of-fold probabilities provide validation predictions for tuning.
oof_probabilities = cross_val_predict(
    model, X_dev, y_dev, cv=cv, method="predict_proba"
)[:, 1]

precision, recall, thresholds = precision_recall_curve(y_dev, oof_probabilities)
denominator = precision[:-1] + recall[:-1]
f1_scores = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator != 0,
)
best_threshold = thresholds[np.argmax(f1_scores)]

model.fit(X_dev, y_dev)
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

fpr, tpr, _ = roc_curve(y_test, test_probabilities)
test_precision, test_recall, _ = precision_recall_curve(
    y_test, test_probabilities
)
test_pr_auc = auc(test_recall[::-1], test_precision[::-1])

print("Mean CV accuracy:", cv_results["test_accuracy"].mean())
print("Mean CV ROC-AUC:", cv_results["test_roc_auc"].mean())
print("Mean CV average precision:", cv_results["test_average_precision"].mean())
print("Mean CV log loss:", -cv_results["test_neg_log_loss"].mean())
print("Selected threshold:", best_threshold)
print("ROC curve points:", len(fpr))
print("Test ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("Test PR-AUC:", test_pr_auc)
print("Test average precision:", average_precision_score(y_test, test_probabilities))
print("Confusion matrix:\n", confusion_matrix(y_test, test_predictions))
print(classification_report(y_test, test_predictions, digits=3))

Common Pitfalls

  • Leakage: Fit scalers, encoders, imputers, and feature selection inside a pipeline so they use training folds only.

  • Wrong metric: Accuracy can mislead with imbalanced classes; inspect precision, recall, F1, PR-AUC, and the confusion matrix.

  • Threshold confusion: A high ROC-AUC does not establish performance at the required operating point.

  • Discrimination versus calibration: A model may rank cases well while its probability magnitudes remain inaccurate.

  • Test-set reuse: Tuning against test results turns the test set into a validation set.

  • Invalid validation design: Use grouped or time-based splitting when random folds would leak information.

  • Business mismatch: Confirm that offline metrics reflect the intended online outcome, using controlled experiments such as A/B tests when appropriate.

23

How do you test statistical significance of model performance differences?

Paired statistical testing compares models by treating their predictions on the same observations or resamples as dependent pairs, then quantifying the uncertainty of the performance difference. It solves the problem of deciding whether an observed improvement is distinguishable from sampling variation; the main trade-off is that simple paired tests can be anti-conservative when folds or training runs are dependent.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Evaluation Setup

  • Split data into training, validation, and test sets. Fit on training data, select the model and threshold on validation data, and keep the test set untouched until final evaluation.

  • Use StratifiedKFold for classification, GroupKFold for dependent groups, and TimeSeriesSplit for ordered data.

  • Tune with GridSearchCV or RandomizedSearchCV inside nested cross-validation, and compare candidates with a DummyClassifier baseline. Pipelines ensure preprocessing is fitted only on each training fold.

  • Give competing models identical splits so their results are paired. Scale features for logistic regression; tree models generally do not require scaling.

Metrics and Test Selection

Let N be the number of cases, TP, FP, TN, and FN the confusion-matrix counts, y_i the true outcome, p_i its predicted probability, and e_i = y_i - y_hat_i the regression error.

Setting Formulas
Binary classification Accuracy = (TP + TN) / N; precision = TP / (TP + FP); recall = TP / (TP + FN); F1 = 2TP / (2TP + FP + FN); specificity = TN / (TN + FP).
Ranking The ROC curve plots (FPR, TPR), where FPR = FP/(FP+TN) and TPR = TP/(TP+FN), and ROC-AUC is its area. The PR curve plots (recall, precision) as the threshold decreases; average precision is sum((R_k - R_(k-1)) * P_k). PR-AUC is especially informative for rare positives.
Probability quality Log loss = -(1/N) * sum(y_i*log(p_i) + (1-y_i)*log(1-p_i)); Brier score = (1/N) * sum((p_i-y_i)^2). A reliability curve compares mean predicted probabilities with observed frequencies within probability bins.
Regression MAE = mean(abs(e_i)); MSE = mean(e_i^2); RMSE = sqrt(MSE); R² = 1 - SSE/SST, where SSE = sum(e_i^2) and SST = sum((y_i-mean(y))^2). MAPE = 100 * mean(abs(e_i/y_i)) is unstable near zero. Quantile loss is mean(max(tau*e_i, (tau-1)*e_i)), where tau is the target quantile.

For two classifiers on the same held-out examples, McNemar’s test compares paired correctness decisions. Let b be cases where A is correct and B is wrong, c where A is wrong and B is correct, and n = b + c. Under the null, b follows a binomial distribution with probability 0.5; the exact two-sided p-value is min(1, 2 * sum(C(n,k) for k from 0 to min(b,c)) / 2^n).

Report the effect size and a confidence interval as well as the p-value. Use a paired bootstrap for an accuracy or loss difference, DeLong’s test or a paired score bootstrap for ROC-AUC, and a corrected paired test or hierarchical bootstrap when comparing cross-validation folds. A paired test or bootstrap of per-example regression losses can compare regression models.

Code Example

from math import comb

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.50,
    stratify=y_train_val, random_state=42
)

models = {
    "logistic_regression": Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
    ]),
    "random_forest": Pipeline([
        ("classifier", RandomForestClassifier(
            n_estimators=300, random_state=42
        )),
    ]),
    "dummy_baseline": Pipeline([
        ("classifier", DummyClassifier(strategy="prior", random_state=42)),
    ]),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "accuracy": "accuracy",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "pr_auc": "average_precision",
}

thresholds = {}
test_probabilities = {}

for name, model in models.items():
    cv_results = cross_validate(
        model, X_train, y_train, cv=cv, scoring=scoring
    )
    print(
        name,
        {key: round(values.mean(), 3)
         for key, values in cv_results.items()
         if key.startswith("test_")},
    )

    model.fit(X_train, y_train)
    validation_probabilities = model.predict_proba(X_val)[:, 1]
    precision, recall, candidate_thresholds = precision_recall_curve(
        y_val, validation_probabilities
    )

    denominator = precision[:-1] + recall[:-1]
    f1 = np.divide(
        2 * precision[:-1] * recall[:-1],
        denominator,
        out=np.zeros_like(denominator),
        where=denominator != 0,
    )
    threshold = float(candidate_thresholds[np.argmax(f1)])
    thresholds[name] = threshold
    test_probabilities[name] = model.predict_proba(X_test)[:, 1]

    predictions = (test_probabilities[name] >= threshold).astype(int)
    print(classification_report(y_test, predictions, target_names=data.target_names))
    print(confusion_matrix(y_test, predictions))
    print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities[name]):.4f}")
    print(
        "Average precision: "
        f"{average_precision_score(y_test, test_probabilities[name]):.4f}"
    )

predictions_a = (
    test_probabilities["logistic_regression"]
    >= thresholds["logistic_regression"]
).astype(int)
predictions_b = (
    test_probabilities["random_forest"]
    >= thresholds["random_forest"]
).astype(int)

correct_a = predictions_a == y_test
correct_b = predictions_b == y_test
b = int(np.sum(correct_a & ~correct_b))
c = int(np.sum(~correct_a & correct_b))
n_discordant = b + c

if n_discordant:
    lower_tail = sum(
        comb(n_discordant, k) for k in range(min(b, c) + 1)
    ) / (2 ** n_discordant)
    mcnemar_p = min(1.0, 2 * lower_tail)
else:
    mcnemar_p = 1.0

difference = correct_a.astype(float) - correct_b.astype(float)
observed_difference = float(difference.mean())

rng = np.random.default_rng(42)
bootstrap_differences = np.empty(5_000)
for i in range(bootstrap_differences.size):
    indices = rng.integers(0, difference.size, difference.size)
    bootstrap_differences[i] = difference[indices].mean()

ci_lower, ci_upper = np.quantile(
    bootstrap_differences, [0.025, 0.975]
)

print(f"Accuracy difference: {observed_difference:.4f}")
print(f"95% paired bootstrap CI: ({ci_lower:.4f}, {ci_upper:.4f})")
print(f"Exact McNemar p-value: {mcnemar_p:.4f}")

Interpreting Results

At significance level alpha = 0.05, reject the null only when p < alpha. A statistically significant difference may still be too small to justify deployment; compare its confidence interval and effect size with operational costs and business impact.

Common Pitfalls

  • Leakage: Do not tune models, thresholds, or preprocessing on the test set.

  • Invalid independence assumptions: Ordinary t-tests on overlapping CV folds often underestimate uncertainty.

  • Imbalanced classes: Accuracy can hide poor minority-class recall; inspect precision, recall, F1, PR-AUC, and the confusion matrix.

  • Repeated validation use: Excessive tuning against one validation set gradually overfits it.

  • Multiple comparisons: Adjust the procedure or significance level when testing many models.

  • Offline versus online impact: Statistical significance does not establish business value; confirm important changes with an A/B test.

24

What is the role of confidence intervals in model evaluation?

A confidence interval quantifies uncertainty around an estimated evaluation metric, such as accuracy or RMSE. It indicates whether measured performance is precise enough to support a decision and whether an observed difference between models is meaningful. The main trade-off is that a higher confidence level generally produces a wider interval.

Read full answer

Applies tomachine-learning concepts and scikit-learn 1.5+ (Python 3.12)

Evaluation Setup

A reliable evaluation separates the data into three roles:

  • Training set: Used to fit model parameters.

  • Validation set: Used for model selection and hyperparameter tuning.

  • Test set: Remains untouched until the final evaluation.

Within the development data, use KFold for independent, identically distributed data, StratifiedKFold for classification, GroupKFold when observations belong to dependent groups, and TimeSeriesSplit for time-ordered observations. Fit preprocessing inside a scikit-learn Pipeline so that each fold learns scaling, imputation, encoding, and feature selection only from its training portion.

What Confidence Intervals Show

Under an independence approximation, an approximate interval for the mean of k validation scores m_1,...,m_k is:

CI = mean(m) ± t × (s / sqrt(k))

Here, mean(m) is the average validation score, s is the sample standard deviation of the fold scores, sqrt(k) is the square root of the fold count, and t is the critical value for the desired confidence level with k - 1 degrees of freedom.

A 95% confidence interval means that, under repeated application of the same sampling procedure, approximately 95% of intervals constructed this way would contain the target population performance. It does not mean that the true performance has a 95% probability of lying in that particular interval.

Cross-validation scores are not strictly independent because their training sets overlap. Consequently, the ordinary t interval can be unreliable when differences are small; paired bootstrap procedures or corrected repeated cross-validation are often preferable. A confidence interval concerns uncertainty in an estimated population metric, not uncertainty around an individual prediction.

Metrics to Estimate

Choose metrics that match the problem and estimate uncertainty for the same metric. Define TP as true positives, TN as true negatives, FP as false positives, FN as false negatives, and n = TP + TN + FP + FN.

Metric Formula Primary use
Accuracy (TP + TN) / n Overall correctness; misleading with imbalanced classes
Precision TP / (TP + FP) Correctness of positive predictions
Recall TP / (TP + FN) Coverage of actual positives
F1 2TP / (2TP + FP + FN) Balance between precision and recall
Specificity TN / (TN + FP) Correct rejection of negatives

Additional classification measures include:

  • ROC curve: Plots true-positive rate TPR = TP / (TP + FN) against false-positive rate FPR = FP / (FP + TN) across classification thresholds. ROC-AUC can be expressed as P(S+ > S-) + 0.5P(S+ = S-), where S+ and S- are scores for positive and negative cases.

  • Precision-recall curve and PR-AUC: Focus on the positive class and are generally more informative for rare positives. One common PR-AUC convention is average precision, AP = sum_j (R_j - R_(j-1)) × P_j, where P_j and R_j are precision and recall at successive thresholds and R_0 = 0; trapezoidal PR-AUC is another convention.

  • Log loss: -1/n × sum(y_i × log(p_i) + (1-y_i) × log(1-p_i)), where y_i is a binary label and p_i is its predicted probability. Lower is better.

  • Brier score: 1/n × sum((p_i-y_i)^2), which measures probability accuracy and is lower when better. Reliability curves compare predicted probabilities with observed positive rates for calibration assessment.

For regression, let y_i be observed values, yhat_i their predictions, n the number of observations, and y_bar the mean observed value:

  • MAE: 1/n × sum(abs(y_i-yhat_i)).

  • MSE: 1/n × sum((y_i-yhat_i)^2).

  • RMSE: sqrt(MSE); penalizes large errors more strongly.

  • R²: 1 - sum((y_i-yhat_i)^2) / sum((y_i-y_bar)^2).

  • MAPE: 100/n × sum(abs(y_i-yhat_i) / abs(y_i)); undefined when an actual value is zero and unstable near zero.

  • Quantile loss: 1/n × sum(rho_tau(y_i-yhat_i)), where u = y_i-yhat_i is the residual, tau is the target quantile in [0,1], and rho_tau(u) = u × (tau - 1[u < 0]); the indicator distinguishes underprediction from overprediction.

Estimation and Model Comparison

  • Bootstrap intervals: Resample observations, recompute the metric, and construct percentile or BCa endpoints. For comparing models, resample the same examples and compute paired metric differences.

  • Analytical intervals: Wilson intervals are preferable to naïve normal approximations for proportions such as accuracy.

  • Paired comparison: Use the same folds or examples for both models. Overlapping confidence intervals alone do not establish that one model is better; examine the interval for the paired difference.

  • Scikit-learn evaluation: cross_validate with a Pipeline and several scorers collects fold results. classification_report, confusion_matrix, roc_auc_score, and average_precision_score provide classification summaries.

  • Model selection: Use GridSearchCV or RandomizedSearchCV for tuning. Use nested cross-validation when an honest estimate is needed after tuning, and compare against a DummyClassifier baseline.

Common Pitfalls

  • Tuning against the test set invalidates its unbiased purpose.

  • Reporting only a point estimate hides sampling uncertainty.

  • Using accuracy with severe class imbalance can make a weak model appear strong.

  • Treating dependent cross-validation folds as independent can misstate uncertainty.

  • Selecting a threshold repeatedly against the test set turns it into a validation set.

  • Interpreting a wider interval as necessarily meaning a worse model; it may reflect a small or variable dataset.

  • Assuming an offline metric improvement guarantees business improvement; validate important changes with online experiments such as A/B tests.

25

How do Bayesian methods support model evaluation?

Bayesian model evaluation uses prior beliefs and observed data to quantify parameter uncertainty, predictive uncertainty, and the relative plausibility of candidate models. It helps with small or noisy datasets by making regularization explicit, at the cost of prior sensitivity and potentially expensive posterior computation.

Read full answer

Applies toBayesian model evaluation concepts; scikit-learn 1.6+ and NumPy 2.x (Python 3.12)

Evaluation Setup

Keep three evaluation roles separate:

  • Training set: Fit model parameters.

  • Validation set: Select models, thresholds, or calibration methods.

  • Test set: Remain untouched until final evaluation.

Use KFold for general data, StratifiedKFold for classification, GroupKFold for related observations, and TimeSeriesSplit for temporal data. Put learned preprocessing inside a Pipeline so it is fitted only on each training fold. Use nested cross-validation when hyperparameter selection must be evaluated honestly, and compare against a simple DummyClassifier baseline.

Tool How it works Strength Weakness When to use
GridSearchCV Tests every point in a grid Exhaustive and deterministic Expensive for large grids Small discrete search spaces
RandomizedSearchCV Samples candidates from a distribution Handles larger spaces May miss the optimum Large or mostly continuous spaces
Bayesian optimization Uses a probabilistic surrogate to select trials Can be evaluation-efficient Adds optimization overhead and modeling assumptions Costly black-box evaluations

Metrics to Evaluate

For binary classification, let y be the binary label, p the predicted probability of the positive class, and n the sample count. TP, TN, FP, and FN are confusion-matrix counts.

Metric Definition
Accuracy (TP + TN) / n
Precision TP / (TP + FP)
Recall TP / (TP + FN)
F1 2TP / (2TP + FP + FN)
Specificity TN / (TN + FP)
ROC-AUC Area under the true-positive-rate versus false-positive-rate curve; under random tie-breaking, the probability that a positive outranks a negative
PR-AUC Area under the precision-recall curve; preferable for rare positives
Log loss -(1/n) * sum(y*log(p) + (1-y)*log(1-p))
Brier score (1/n) * sum((p-y)^2)

Accuracy can mislead with imbalanced classes. Log loss and Brier score evaluate probability quality, while reliability curves compare predicted probabilities with observed frequencies.

For regression, y is the observed value, y_hat the prediction, and mean(y) the observed mean:

Metric Definition
MAE (1/n) * sum(abs(y-y_hat))
MSE (1/n) * sum((y-y_hat)^2)
RMSE sqrt(MSE)
R² 1 - sum((y-y_hat)^2) / sum((y-mean(y))^2)
MAPE (100/n) * sum(abs(y-y_hat)/abs(y)); unstable near zero
Quantile loss mean(u * (tau - I(u < 0))), where u=y-y_hat, tau is the target quantile, and I(u < 0) is 1 when u<0 and 0 otherwise

Bayesian Methods

Bayesian inference combines the likelihood of the observed data with prior beliefs:

posterior(parameters | data) ∝ likelihood(data | parameters) * prior(parameters)

The posterior predictive distribution integrates predictions over parameter uncertainty:

p(y_new | data) = integral p(y_new | parameters) * p(parameters | data) d(parameters)

These foundations support:

  • Posterior predictive checks: Compare simulated replicated data with observations to detect inadequate model assumptions.

  • Uncertainty evaluation: Distinguish credible intervals for parameters from predictive intervals for new observations.

  • Bayesian model averaging: Weight predictions by each model’s posterior model probability rather than selecting one model outright.

  • Bayesian optimization: Use a posterior-based surrogate to choose promising expensive evaluations, often requiring fewer trials than exhaustive search.

  • Bayes factors: Compare models through their relative predictive support; resulting posterior odds also depend on prior model odds.

Code Example

GaussianNB is a naive Bayes classifier combining class priors with fitted Gaussian class-conditionals. Its predict_proba gives posterior class probabilities under those fixed assumptions; it does not include uncertainty in the estimated parameters.

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.metrics import (
    average_precision_score,
    brier_score_loss,
    classification_report,
    confusion_matrix,
    log_loss,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target

# Keep the test set untouched during model and threshold selection.
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    stratify=y_train_val,
    random_state=42,
)

# Scaling is not required by GaussianNB, but the pipeline demonstrates
# leakage-safe preprocessing. Imputation should also be placed in it if needed.
model = make_pipeline(StandardScaler(), GaussianNB())
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}

# Each fold fits preprocessing only on its training partition.
cv_results = cross_validate(
    model, X_train, y_train, cv=cv, scoring=scoring
)
print("CV means:", {
    name: round(np.mean(values), 3)
    for name, values in cv_results.items()
})

model.fit(X_train, y_train)
validation_probabilities = model.predict_proba(X_val)[:, 1]

# Select an F1 threshold using validation predictions, not test predictions.
precision, recall, thresholds = precision_recall_curve(
    y_val, validation_probabilities
)
denominator = precision[:-1] + recall[:-1]
f1_values = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
best_threshold = thresholds[np.argmax(f1_values)]

# Perform the final evaluation once on the untouched test set.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

baseline = DummyClassifier(strategy="prior").fit(X_train, y_train)
baseline_probabilities = baseline.predict_proba(X_test)[:, 1]

print("Selected threshold:", round(best_threshold, 3))
print(classification_report(y_test, test_predictions))
print("Confusion matrix:\n", confusion_matrix(y_test, test_predictions))
print("ROC-AUC:", round(roc_auc_score(y_test, test_probabilities), 3))
print("PR-AUC:", round(average_precision_score(y_test, test_probabilities), 3))
print("Log loss:", round(log_loss(y_test, test_probabilities), 3))
print("Brier score:", round(brier_score_loss(y_test, test_probabilities), 3))
print("Dummy ROC-AUC:", round(roc_auc_score(y_test, baseline_probabilities), 3))

Common Pitfalls

  • Leakage: Fit scalers, imputers, or feature selectors outside a pipeline and contaminate evaluation.

  • Test-set tuning: Reuse the test set for thresholds, model changes, or preprocessing decisions.

  • Imbalance: Accuracy may hide poor minority-class performance; inspect precision, recall, F1, PR-AUC, and confusion matrices.

  • Invalid splits: Random splits can leak information in temporal or grouped observations.

  • Poor uncertainty calibration: High ROC-AUC does not guarantee calibrated probabilities; check reliability curves and prior sensitivity.

  • Metric mismatch: Offline predictive metrics may not represent business value; validate important decisions through A/B tests when possible.

Interviewers commonly ask how Bayesian uncertainty differs from bootstrap uncertainty, why PR-AUC is preferred for rare positives, and when posterior predictive checks are most useful.

26

How to compare multiple machine learning models?

Model comparison is the systematic process of evaluating multiple machine learning models on identical data splits to determine which one generalizes best. It addresses the problem of model selection by providing a fair, metric-driven assessment that accounts for randomness in data splitting. The main trade-off is between the depth of comparison (number of models, metrics, statistical tests) and the computational cost and complexity of the analysis.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12) for code examples; the concepts apply to any ML library.

Evaluation Setup

  • Train/Validation/Test split: Reserve a test set that is never used during model selection or hyperparameter tuning. Use the training portion for cross-validation to compare models.

  • Cross-validation: Apply the same cross-validation strategy (e.g., StratifiedKFold for classification, TimeSeriesSplit for time series) to every candidate model. This ensures each model sees exactly the same data folds, making scores directly comparable.

  • Pipelines: Wrap preprocessing (scaling, encoding, imputation) inside a Pipeline so that preprocessing is fitted on each training fold only, preventing data leakage.

  • Baseline: Always include a simple baseline (e.g., DummyClassifier) to gauge whether your models are better than a trivial predictor.

Choosing Comparison Metrics

Select metrics that reflect the business objective. Using the wrong metric can make a poor model look superior.

Task Metric Formula When to Use
Classification Accuracy (TP+TN)/(TP+TN+FP+FN) Balanced classes only
Classification F1-score 2·(precision·recall)/(precision+recall) Imbalanced classes, need balance
Classification ROC-AUC Area under ROC curve General, threshold-independent
Classification PR-AUC Area under precision-recall curve Rare positives (imbalanced)
Regression RMSE √(mean((y-ŷ)²)) Sensitive to large errors
Regression MAE mean(|y-ŷ|) Robust to outliers
Regression R² 1 - SS_res/SS_tot Proportion of variance explained

For imbalanced datasets, accuracy can be misleading; prefer PR-AUC or F1-score.

Statistical Comparison of Models

Point estimates (e.g., mean accuracy) can be misleading because of variance across folds. Use statistical tests to assess whether observed differences are likely real. - Per-fold scores: From cross_validate, collect the metric values on each fold. This gives a distribution for each model. - Paired tests: Since the same folds are used, scores are paired. Apply the paired t-test or Wilcoxon signed-rank test to test the null hypothesis that the two models have equal mean performance. A p-value < 0.05 is conventionally considered significant. - Nested cross-validation: When you need an honest estimate of the final model’s performance, use an outer loop for evaluation and an inner loop for hyperparameter tuning. This prevents optimistic bias from tuning on the same data. - Confidence intervals: Compute the standard error of the mean score (std / √n_folds) to build a confidence interval. Overlapping intervals suggest no significant difference.

Code Example: Comparing Multiple Models

The following example loads the breast cancer dataset, defines three classifiers, evaluates them with 5-fold stratified cross-validation using three scoring metrics, and prints the results.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from scipy import stats

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# Split into train and test (test kept untouched until final evaluation)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Define models (with scaling where needed)
models = {
    'Logistic Regression': Pipeline([
        ('scaler', StandardScaler()),
        ('clf', LogisticRegression(max_iter=1000, random_state=42))
    ]),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'SVM': Pipeline([
        ('scaler', StandardScaler()),
        ('clf', SVC(probability=True))
    ])
}

# Cross-validation strategy
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Metrics to compute
scoring = ['accuracy', 'f1', 'roc_auc']

results = {}
for name, model in models.items():
    scores = cross_validate(
        model, X_train, y_train,
        cv=cv,
        scoring=scoring,
        return_train_score=False,
        n_jobs=-1
    )
    results[name] = {
        'test_accuracy': scores['test_accuracy'],
        'test_f1': scores['test_f1'],
        'test_roc_auc': scores['test_roc_auc']
    }

# Print mean and standard deviation for each model
print("Cross-validation results (mean +/- std):")
for name, metrics in results.items():
    print(f"\n{name}:")
    for metric_name, values in metrics.items():
        print(f"  {metric_name}: {values.mean():.4f} (+/- {values.std():.4f})")

# Optional: paired t-test between Logistic Regression and Random Forest
lr_acc = results['Logistic Regression']['test_accuracy']
rf_acc = results['Random Forest']['test_accuracy']
t_stat, p_value = stats.ttest_rel(lr_acc, rf_acc)
print(f"\nPaired t-test (LR vs RF) on accuracy: t={t_stat:.3f}, p={p_value:.3f}")

The script prints each model’s mean and standard deviation per metric, then the paired t-test. On a dataset this easy all three models usually score within a point or two of each other, so a p-value above 0.05 is common. Treat the t-test on five folds as a rough check: the folds share training data, so the test tends to understate uncertainty, and a corrected resampled t-test or repeated cross-validation gives a fairer comparison.

Common Pitfalls

  • Data leakage: Using the same data for preprocessing and evaluation without pipelines. Always fit transformers on training folds only.

  • Different splits for different models: Never compare scores obtained with different random seeds or fold assignments. Use a single cv object for all models.

  • Tuning on the test set: Hyperparameter tuning should be done on validation folds, not the test set. The test set must remain untouched until the final model is chosen.

  • Relying on a single metric: Accuracy on imbalanced data can be deceptive. Always look at multiple metrics (F1, PR-AUC, ROC-AUC).

  • Ignoring statistical significance: A model may appear better by a small margin, but the difference could be due to random chance. Always back up your choice with a statistical test or confidence intervals.

  • Overfitting to the validation set: When you repeatedly choose the model based on validation performance, you are implicitly tuning on that set. Use nested cross-validation or a separate hold-out set for final model selection.

Interview Tips: Be ready to explain why you chose a particular metric, how you ensured a fair comparison, and how you would handle the case where two models have very similar mean scores but different variances. Also, be prepared to discuss the computational cost of comparing many models and strategies to reduce it (e.g., random search for hyperparameters, early stopping for iterative algorithms).

27

How do AIC and BIC guide statistical model selection?

Akaike’s Information Criterion (AIC) and Bayesian Information Criterion (BIC) select models by balancing statistical fit against parameter complexity. Lower values are preferred: AIC generally targets predictive accuracy, while BIC uses a stronger complexity penalty and favors more parsimonious models. The central trade-off is that a better fit is accepted only when it justifies the additional parameters.

Read full answer

Applies tostatistical model selection concepts (all software versions)

Evaluation Setup

AIC and BIC compare candidate models evaluated on the same response variable and the same observations. Their likelihoods must also use a consistent scale; otherwise, the resulting values are not directly comparable.

During model development:

  1. Reserve a test set before fitting models if a final unbiased performance estimate is required.

  2. Fit each candidate model to the training data.

  3. Calculate AIC and BIC for those models.

  4. Prefer the model with the lower criterion, while examining the size of the difference.

  5. Evaluate the selected model once on the untouched test set.

For purely predictive work, cross-validation may be more appropriate because information criteria depend on distributional assumptions. A repeated random split should not be used for time-ordered or grouped observations.

How AIC and BIC Work

Let:

  • n = number of observations.

  • k = number of estimated model parameters, including the intercept and usually the residual variance when it is estimated.

  • L_hat = model’s maximized likelihood.

The formulas are:

  • AIC = 2k - 2 ln(L_hat)

  • BIC = ln(n)k - 2 ln(L_hat)

In both formulas, -2 ln(L_hat) rewards better likelihood. The penalty discourages unnecessary complexity:

  • AIC penalty: 2k

  • BIC penalty: ln(n)k

Because ln(n) grows with sample size, BIC penalizes additional parameters more heavily as the dataset becomes larger. Both criteria are relative: only their differences between models on the same dataset are meaningful.

AIC Versus BIC

Criterion Main objective Complexity penalty Typical behavior Best suited for
AIC Minimize estimated predictive error 2k Often selects a larger model when models fit similarly Prediction and model averaging
BIC Identify the most parsimonious adequate model ln(n)k Favors simpler models, especially with large n Variable selection and parsimony
AICc Correct AIC for finite samples AIC penalty plus a sample-size correction Performs better than ordinary AIC when n is small relative to k Small or moderate samples
Cross-validation Estimate out-of-sample predictive performance No explicit AIC/BIC penalty Directly measures generalization, but requires repeated training Strong emphasis on prediction

AIC is approximately optimal for minimizing expected out-of-sample prediction error under suitable assumptions, but the selected model is generally not guaranteed to be the true model. BIC has a stronger consistency property: under regularity conditions and when the true model is included among the candidates, its probability of selecting the true model approaches one as the sample grows.

Interpreting Differences

A lower criterion is better, but the magnitude of the difference matters. Common interpretations include:

  • AIC difference of 2 or less: the models have similar support.

  • AIC difference of 4–6: support is limited or moderate.

  • AIC difference above 10: strong support for the lower-AIC model.

  • BIC differences above 10: generally strong evidence against the higher-BIC model.

These are heuristics, not formal significance tests. AIC and BIC do not calculate the probability that a model is “true.” If model weights are used, they are also model-dependent and should not be interpreted as absolute truth probabilities.

Practical Guidance

  • Compare nested or otherwise meaningful candidate models using identical observations.

  • Count parameters consistently, especially in models with interactions, splines, random effects, or estimated variances.

  • Use AICc instead of AIC when the sample size is small relative to the number of parameters.

  • Prefer AIC or AIC model averaging when predictive uncertainty is the main concern.

  • Prefer BIC when parsimony and consistent variable selection are priorities.

  • Inspect residuals, calibration, and domain plausibility; a low criterion cannot compensate for a fundamentally misspecified model.

  • Use nested cross-validation when an honest comparison of the entire tuning procedure is needed.

Common Pitfalls

  • Comparing AIC or BIC across different datasets: the values are not portable.

  • Counting parameters inconsistently: this unfairly penalizes some models.

  • Treating the lowest value as proof: information criteria provide relative evidence under their assumptions.

  • Ignoring assumptions: non-normal errors, dependent observations, and poorly specified likelihoods can invalidate the comparison.

  • Tuning against the test set: repeated model selection contaminates the final evaluation.

  • Using in-sample fit alone: a high R^2 or low training error may simply indicate overfitting.

Interviewers commonly ask why BIC penalizes complexity more strongly, whether AIC guarantees selection of the true model, how AIC differs from cross-validation, and how the number of parameters affects the choice between AIC and AICc.

28

When should AIC be preferred over BIC for model selection?

AIC (Akaike Information Criterion) is generally preferred when prediction is the main objective, while BIC (Bayesian Information Criterion) places greater emphasis on parsimony, especially in large datasets. Both balance goodness of fit against model complexity, but they penalize additional parameters differently. The main trade-off is that AIC often permits more complex models, whereas BIC increasingly favors simpler models as the sample size grows.

Read full answer

Applies tostatistical model selection concepts (all versions)

Evaluation Setup

AIC and BIC compare candidate models fitted to the same observations and response variable; their absolute values are not directly comparable across unrelated datasets or outcomes.

A sound evaluation workflow is:

  1. Keep a final test set untouched.

  2. Fit candidate models on the training set.

  3. Calculate AIC and BIC from the training-set likelihoods to compare those candidates.

  4. Use validation data or cross-validation for hyperparameter tuning and predictive evaluation.

  5. Evaluate the final selected model once on the test set.

Use StratifiedKFold for classification, TimeSeriesSplit for time-ordered observations, and GroupKFold when observations from the same group must remain together. AIC and BIC do not replace held-out evaluation because both rely on in-sample likelihood.

For hyperparameter tuning, GridSearchCV or RandomizedSearchCV can search candidate settings. If the same cross-validation results are also used to report final predictive performance, use an outer cross-validation loop—nested cross-validation—to avoid optimistic bias. When predictive differences are close, use outer-fold paired differences to construct a confidence interval or perform an appropriate paired test; an AIC or BIC difference is not a confidence interval for test error.

How AIC and BIC Work

For a candidate model:

  • AIC = 2k - 2 * ln(L_hat)

  • BIC = k * ln(n) - 2 * ln(L_hat)

In these formulas:

  • k is the number of estimated parameters.

  • n is the number of observations.

  • L_hat is the model’s maximized likelihood.

  • ln is the natural logarithm.

Lower AIC or BIC indicates a more favorable balance between fit and complexity, provided the models use the same data and response. The negative sign before the log-likelihood makes better-fitting, higher-likelihood models receive lower scores.

AIC is motivated as a proxy for expected out-of-sample predictive performance. BIC can also be interpreted through marginal likelihood under a prior model and, under suitable assumptions, can consistently identify the true model as the sample size becomes large.

Choosing AIC Over BIC

Prefer AIC when:

  • The model will mainly be used for prediction.

  • Additional complexity may capture genuine predictive structure.

  • The sample is small relative to the number of parameters, making AICc more appropriate than ordinary AIC.

  • The goal is predictive performance rather than identifying the simplest “true” model.

  • A small-sample correction is needed because AIC’s usual derivation is asymptotic.

AICc adds a correction to AIC:

AICc = AIC + 2k(k + 1) / (n - k - 1)

The correction becomes substantial when n is small relative to k, discouraging unnecessarily complex models.

Prefer BIC when:

  • Interpretability and parsimony are priorities.

  • The dataset is large relative to the number of parameters.

  • Strong shrinkage toward simpler models is desirable.

  • The candidates are well specified and marginal-likelihood interpretation is valuable.

  • Model simplicity should outweigh modest improvements in fit.

For example, BIC may favor a model with 10 parameters over one with 12 when the likelihood advantage is modest, while AIC may retain the more complex model. BIC’s penalty, k * ln(n), generally grows faster with sample size than AIC’s fixed 2k penalty.

Practical Comparison

Criterion Main objective Complexity penalty Typical behavior Best suited for
AIC Predictive performance 2k Often selects more complex models Prediction and moderate datasets
AICc Predictive performance with small-sample correction AIC plus correction Penalizes high parameter counts more strongly Small samples
BIC Fit, parsimony, and marginal likelihood k * ln(n) Often selects simpler models as n grows Large datasets and interpretable models

Check the selected model against simple baselines, such as DummyClassifier for classification, and against cross-validated predictive performance when accuracy is the main concern.

Common Pitfalls

  • Comparing scores across different outcomes: AIC and BIC are meaningful for models fitted to the same observations and response.

  • Assuming the lower score guarantees better test performance: These criteria can still select an overfit model, so evaluate the final model on untouched data.

  • Ignoring the definition of k: Count all parameters estimated from the data, including variance parameters when applicable.

  • Using ordinary AIC for very small samples: Prefer AICc when the sample size is limited relative to the parameter count.

  • Treating a small AIC/BIC difference as conclusive: Assess practical predictive differences and their uncertainty.

  • Overfitting through repeated testing: Modifying candidates based on the test set makes the final estimate biased.

Interview Tips

Explain that AIC favors predictive fit, while BIC imposes stronger parsimony, especially for large n. Common follow-up questions concern BIC’s sample-size-dependent penalty, the difference between AIC and AICc, whether either criterion is universally “more correct,” and how these criteria complement rather than replace cross-validation and final test evaluation.

29

How does the Elbow Method select clusters in K-means?

The Elbow Method chooses the number of clusters, k, by plotting K-means’ within-cluster sum of squares (WCSS), also called inertia, for several candidate values of k. The selected k is near the bend where additional clusters produce only a marginal reduction in WCSS; K-means itself determines the cluster assignments and centroids. The main trade-off is model simplicity versus fit quality because WCSS normally decreases as k increases, while the elbow’s location is subjective.

Read full answer

Applies toscikit-learn 1.5+ with `KMeans` (Python 3.12)

Evaluation Setup

Clustering is unsupervised, so there are no class labels for a conventional train/validation/test split. When held-out evaluation is required, however, the data should still be divided before fitting any preprocessing or clustering model:

  1. Training partition: Fit candidate K-means models and calculate their elbow curves.

  2. Validation partition: Compare candidate values of k using validation metrics, stability checks, and domain knowledge.

  3. Test partition: Keep it untouched until k, preprocessing, and the clustering procedure have been finalized.

For exchangeable observations, repeated resampling can provide a more stable assessment. Use group-preserving splits such as GroupKFold when observations from the same entity are correlated, and time-ordered splits such as TimeSeriesSplit for temporal data. StratifiedKFold is not applicable to ordinary unlabeled clustering because it relies on class labels. Fit scaling, imputation, and K-means only on the relevant training data to prevent leakage.

How It Works

For a candidate number of clusters k, K-means minimizes:

WCSS(k) = sum from i = 1 to n of ||x_i - mu_c(i,k)||^2

Here:

  • n is the number of observations.

  • x_i is the position of observation i in feature space.

  • c(i,k) is the cluster assigned to observation i for a model with k clusters.

  • mu_c(i,k) is that cluster’s centroid.

  • ||x_i - mu_c(i,k)||^2 is the squared Euclidean distance from the observation to its centroid.

  • The outer sum adds those squared distances across all observations.

In scikit-learn, this value is exposed as KMeans.inertia_ after fitting. Plot inertia against each candidate k and identify the point after which the curve begins to flatten. Equivalently, examine the reduction between neighboring values using gain(k) = WCSS(k) - WCSS(k + 1); the elbow is where these marginal gains become relatively small. The optimal WCSS is non-increasing as k increases, but an empirical curve can contain flat sections or variations caused by local optima.

Interpreting and Supporting the Elbow

Method What it evaluates Strength Limitation
Elbow curve Reduction in inertia as k increases Simple and widely used The elbow is subjective
Silhouette score Within-cluster cohesion relative to between-cluster separation Does not require labels Can favor geometrically compact clusters
Adjusted Rand Index or normalized mutual information Agreement with known reference labels Useful when trustworthy labels exist Cannot be used without suitable labels
Stability analysis Consistency of assignments across samples or random seeds Reveals unstable solutions Requires repeated clustering runs

Treat the elbow as a candidate rather than an automatic answer. A choice near k = 4, for example, is stronger when silhouette score, repeated-run stability, and domain interpretation also support four groups. Silhouette score is commonly considered alongside the elbow, but it is not a universal replacement because its preferred k can differ.

Practical Considerations

  • Scale features: K-means uses Euclidean distances, so features with larger numeric scales can dominate. Fit a StandardScaler or MinMaxScaler on training data only.

  • Check assumptions: K-means tends to work best for roughly spherical clusters with similar sizes and densities. Density-based methods such as DBSCAN may be more suitable for irregular shapes or varying densities.

  • Handle missing values and outliers: K-means does not accept NaNs, and extreme values can strongly affect centroids and WCSS.

  • Control initialization sensitivity: Multiple initializations reduce the chance of selecting a poor local optimum. In scikit-learn, set n_init explicitly when reproducibility is important.

  • Connect to the application: Offline cluster quality does not guarantee business value. Where appropriate, validate the downstream effect through an online A/B test.

Common Pitfalls

  • Treating the elbow as objective: Plotting choices, scaling, and local optima can move the apparent bend; report supporting evidence.

  • Using the test set during selection: Inspecting test performance for each k turns the test set into another validation set.

  • Assuming more clusters are always better: Larger k lowers inertia but can overfit, fragment meaningful groups, and make interpretation harder.

  • Ignoring data structure: Random splits can leak group information or break temporal ordering.

  • Failing to check stability: If small changes in the sample or random seed produce different partitions, the discovered clusters are not dependable.

A common interview follow-up is whether to choose the smallest k because it has the best inertia. It does not: the smallest k has the largest WCSS, and the elbow instead balances that fit improvement against model complexity.

30

What does Gini coefficient measure in classification model evaluation?

Gini is used in two distinct ways in classification: Gini impurity measures class mixing for a decision-tree split, while normalized Gini summarizes a binary classifier’s ranking quality as 2 * ROC-AUC - 1. The distinction matters because impurity guides training whereas normalized Gini evaluates predictions. Gini impurity is computationally cheap but does not guarantee generalization; normalized Gini is threshold-independent but ignores calibration and the costs of a particular operating threshold.

Read full answer

Applies tomachine learning concepts (all versions); scikit-learn 1.5+ for named APIs (Python 3.12)

Evaluation Setup

  • Train, validation, and test sets: Fit models on training data, use validation data or cross-validation to select hyperparameters, and keep the test set untouched until final evaluation.

  • Cross-validation: Use KFold for general data, StratifiedKFold to preserve class proportions in classification, GroupKFold when related observations must remain together, and TimeSeriesSplit for time-ordered observations.

  • Leakage prevention: Put preprocessing inside a scikit-learn Pipeline so it is fitted only on each training fold.

  • Final baseline: Compare a tuned model with DummyClassifier to determine whether it provides useful predictive skill.

Two Common Interpretations

  • Gini impurity: Used by decision trees to choose candidate splits. It measures how many classes are mixed within a node, not how accurately the completed model will generalize.

  • Normalized Gini: Used to summarize how well a binary classifier ranks positive examples above negative examples. It is derived from ROC-AUC and describes ranking quality rather than probability calibration.

  • The economic Gini coefficient used for income or wealth inequality is a different concept.

Formulas and Interpretation

For a node containing K classes, Gini impurity is:

G = 1 - sum(p_i^2)

Here, G is the node impurity, p_i is the fraction of samples belonging to class i, and i ranges over the K classes. A pure node has G = 0. A maximally mixed binary node has G = 0.5; with K classes, the maximum is 1 - 1/K.

After a proposed split, the weighted impurity is:

G_split = sum((n_j / n) * G_j)

Here, n_j is the number of samples in child node j, n is the original node’s sample count, and G_j is that child’s impurity. Decision trees generally prefer the split with the lowest weighted impurity.

For binary ranking evaluation:

Gini_normalized = 2 * ROC_AUC - 1

Here, ROC_AUC is the area under the receiver operating characteristic curve. A normalized Gini of 1 represents perfect ranking, 0 represents random ranking, and -1 represents completely reversed ranking.

Related Classification Metrics

For binary labels, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives:

  • Accuracy: (TP + TN) / (TP + TN + FP + FN); it can be misleading when classes are imbalanced.

  • Precision: TP / (TP + FP); it measures how many predicted positives are correct.

  • Recall: TP / (TP + FN); it measures how many actual positives are detected.

  • Specificity: TN / (TN + FP); it measures how well actual negatives are identified.

  • F1: 2 * Precision * Recall / (Precision + Recall); it balances precision and recall.

  • ROC curve: Plots true-positive rate, TP / (TP + FN), against false-positive rate, FP / (FP + TN), across thresholds. ROC-AUC summarizes the resulting ranking performance.

  • Precision-recall curve: Plots precision against recall across thresholds. PR-AUC is particularly informative when the positive class is rare.

  • Log loss: -1/n * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where y_i is the true binary label, p_i is the predicted positive probability, and n is the number of samples. It penalizes confident incorrect predictions.

  • Brier score: 1/n * sum((p_i - y_i)^2) for binary labels; lower values indicate better probability estimates. Reliability curves compare mean predicted probabilities with observed positive frequencies across probability bins.

Model Selection and Practical Use

  • Tune tree depth, minimum leaf size, and related parameters with GridSearchCV or RandomizedSearchCV.

  • Use nested cross-validation when an honest performance estimate is needed: an inner loop selects hyperparameters and an outer loop estimates generalization.

  • Compare models using paired cross-validation results, paired resampling, or confidence intervals rather than treating every fold comparison as independent.

  • Use classification_report, confusion_matrix, roc_auc_score, average_precision_score, and cross_validate for complementary evaluation views.

  • Select an operating threshold with precision_recall_curve when the relative costs of false positives and false negatives matter.

  • Feature scaling is unnecessary for a standard decision tree, but class imbalance, overfitting, and random-state reproducibility still matter.

Common Pitfalls

  • Confusing impurity with performance: Gini impurity evaluates candidate training splits, whereas validation metrics evaluate the fitted model.

  • Treating normalized Gini as accuracy: It describes ranking, not the fraction of correct predictions or probability quality.

  • Using accuracy alone: A high accuracy can conceal poor minority-class recall.

  • Tuning on the test set: Repeated test-set feedback leaks information and produces optimistic results.

  • Using inappropriate splits: Random splits can invalidate results for grouped or time-ordered observations.

  • Ignoring business costs: Offline metrics may not reflect online outcomes; validate important changes with an A/B test when possible.

31

How to compute F1-score from precision and recall in Python?

The F1-score is the harmonic mean of precision and recall, a key metric for binary and multiclass classification, especially when classes are imbalanced. It balances the trade-off between false positives and false negatives into a single number, making it more informative than accuracy when one class is rare. The main trade-off is that it weights precision and recall equally, which may not align with business costs of different errors.

Read full answer

Applies toNumPy 2.x (Python 3.12)

Formula

The F1-score is defined as:

F1 = 2 * (precision * recall) / (precision + recall)

where precision = TP / (TP + FP) and recall = TP / (TP + FN), with TP, FP, FN standing for true positives, false positives, and false negatives. If precision + recall = 0, the F1-score is defined as 0 (the harmonic mean is undefined, but 0 is the conventional choice).

Implementation

The function should accept scalar or array-like inputs (for per-class scores in multiclass problems) and return the same shape. Using NumPy allows vectorization and safe handling of division by zero.

Code Example

import numpy as np
from sklearn.metrics import f1_score as sk_f1_score

def f1_score(precision, recall):
    """
    Compute the F1-score given precision and recall.

    Parameters
    ----------
    precision : float or array-like
        Precision value(s). TP / (TP + FP)
    recall : float or array-like
        Recall value(s). TP / (TP + FN)

    Returns
    -------
    float or ndarray
        F1-score(s). Returns 0 where precision + recall == 0.
    """
    p = np.asarray(precision, dtype=np.float64)
    r = np.asarray(recall, dtype=np.float64)
    denom = p + r
    # Use np.where to avoid division by zero
    f1 = np.where(denom > 0, 2 * p * r / denom, 0.0)
    # Return scalar if input was scalar
    if f1.shape == ():
        return f1.item()
    return f1

# Example usage:
if __name__ == "__main__":
    # Single value
    p = 0.8
    r = 0.7
    print(f"F1 for precision={p}, recall={r}: {f1_score(p, r):.4f}")
    # Expected around 0.7467

    # Array input (e.g., per-class scores for a 3-class problem)
    precisions = np.array([0.9, 0.8, 0.7])
    recalls    = np.array([0.8, 0.7, 0.6])
    f1s = f1_score(precisions, recalls)
    print("Per-class F1:", f1s)
    # To get macro F1, take the average:
    macro_f1 = np.mean(f1s)
    print("Macro F1:", macro_f1)

    # Using scikit-learn's built-in f1_score (recommended for real projects)
    # Requires true labels and predictions, not precomputed precision/recall
    y_true = np.array([0, 1, 2, 0, 1, 2])
    y_pred = np.array([0, 2, 1, 0, 0, 1])
    print("sklearn macro F1:", sk_f1_score(y_true, y_pred, average='macro'))
    print("sklearn per-class F1:", sk_f1_score(y_true, y_pred, average=None))

Best Practices

  • Use np.asarray to handle lists, tuples, or scalars uniformly.

  • Always guard against division by zero; np.where is clean and vectorized.

  • For multiclass problems, compute per-class F1 and then average according to the desired averaging strategy (macro, micro, weighted). The function above returns per-class values; np.mean gives macro F1.

  • In production code, prefer sklearn.metrics.f1_score which handles averaging strategies (macro, micro, weighted, samples) and the zero_division parameter directly.

Common Pitfalls

  • Forgetting to handle the precision + recall = 0 case, which raises a ZeroDivisionError in pure Python.

  • Using integer division (//) or not converting to float, which can silently produce 0.

  • Confusing F1 with F0.5 or F2 scores, which weight recall or precision differently.

  • Applying F1 to regression problems where it is not defined.

Interview Tips

Interviewers may ask you to derive the formula from the confusion matrix or to explain why F1 is the harmonic mean rather than arithmetic mean. Be ready to show that the harmonic mean penalizes low values more heavily, which is desirable when both precision and recall must be high. They might also ask about macro/micro/weighted F1 for multiclass settings and when each is appropriate.

32

How do I compute a two-class confusion matrix in Python?

A confusion matrix compares predicted labels with true labels to show correct predictions, false positives, and false negatives. For two-class evaluation, it exposes errors hidden by aggregate accuracy, but the interpretation depends on the chosen threshold and class prevalence.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Split the data into training, validation, and test sets before fitting preprocessing or a model. Fit each learned preprocessing step and the model on training data or training folds only, use validation data for model or threshold selection, and keep the test set untouched until final evaluation. Use StratifiedKFold for imbalanced classification, GroupKFold for grouped observations, and TimeSeriesSplit for time-ordered data.

Confusion Matrix and Metrics

For class 1 as the positive class, the matrix contains true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). Its rows represent true labels and its columns represent predicted labels.

  • Accuracy = (TP + TN) / (TP + TN + FP + FN)

  • Precision = TP / (TP + FP): the proportion of predicted positives that are correct.

  • Recall = TP / (TP + FN): the proportion of actual positives detected.

  • F1 = 2 * precision * recall / (precision + recall)

  • Specificity = TN / (TN + FP): the proportion of actual negatives correctly identified.

The true-positive rate is TPR = TP / (TP + FN), and the false-positive rate is FPR = FP / (TN + FP). The ROC curve plots (FPR, TPR) across thresholds, while ROC-AUC is the area under that curve and measures ranking quality rather than calibration. The precision-recall curve plots (recall, precision), and PR-AUC is its area; it is often more informative for rare positive classes.

For n samples indexed by i, true binary label y_i, and predicted positive probability p_i:

  • Log loss = -(1/n) * sum_i(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where log is the natural logarithm.

  • Brier score = (1/n) * sum_i((y_i - p_i)^2).

Both assess probability quality; a reliability curve compares mean predicted probabilities with observed positive rates across probability bins.

Thresholding and Model Selection

A probability cutoff of 0.5 is only a convention, not necessarily the best operating point. Use precision_recall_curve on validation probabilities to select a threshold—for example, the threshold maximizing F1—and then evaluate it on the test set.

Use GridSearchCV or RandomizedSearchCV with a pipeline for hyperparameter selection. Use nested cross-validation when the final performance estimate must remain unbiased, compare models using paired folds or confidence intervals, and include a DummyClassifier baseline.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    auc,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Targets in this dataset: 0 = malignant, 1 = benign.
X, y = load_breast_cancer(return_X_y=True)

# Create train, validation, and test sets before fitting anything.
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=42
)

# The pipeline ensures the scaler is fitted only on each training fold.
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=2000)),
])

X_dev = np.concatenate([X_train, X_val])
y_dev = np.concatenate([y_train, y_val])

scorers = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    pipeline, X_dev, y_dev, cv=cv, scoring=scorers
)

print("Cross-validation means:")
print({
    name: round(cv_results[f"test_{name}"].mean(), 3)
    for name in scorers
})

# Fit on training data and select an F1-maximizing validation threshold.
pipeline.fit(X_train, y_train)
val_probabilities = pipeline.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_val, val_probabilities
)

# Each threshold corresponds to precision[:-1] and recall[:-1].
denominator = precision[:-1] + recall[:-1]
f1_scores = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
best_index = int(np.argmax(f1_scores))
best_threshold = float(thresholds[best_index])

# Refit on development data, then evaluate the test set once.
pipeline.fit(X_dev, y_dev)
test_probabilities = pipeline.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

matrix = confusion_matrix(y_test, test_predictions, labels=[0, 1])

print(f"\nSelected validation threshold: {best_threshold:.3f}")
print("\nConfusion matrix (rows=true, columns=predicted):")
print(matrix)
print("\nClassification report:")
print(classification_report(
    y_test,
    test_predictions,
    labels=[0, 1],
    target_names=["malignant (0)", "benign (1)"],
    zero_division=0,
))

print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")

# Build the PR curve from test probabilities; reverse it because recall is returned in descending order.
test_precision, test_recall, _ = precision_recall_curve(y_test, test_probabilities)
print(f"PR-AUC: {auc(test_recall[::-1], test_precision[::-1]):.3f}")
print(
    "Average precision: "
    f"{average_precision_score(y_test, test_probabilities):.3f}"
)

Common Pitfalls

  • Data leakage: Do not fit scalers, imputers, encoders, or feature selection on validation or test data.

  • Tuning on the test set: Select the model and threshold without using test results.

  • Ignoring imbalance: A model predicting only the majority class can have high accuracy but zero recall.

  • Wrong split strategy: Use grouped or temporal splitting when observations are related or ordered.

  • Metric confusion: ROC-AUC measures ranking, while log loss, Brier score, and reliability curves assess calibration.

33

How do I implement k-fold cross-validation in Python?

K-fold cross-validation repeatedly trains a model on k - 1 folds and evaluates it on the remaining fold. It reduces dependence on one particular train-validation split, at the cost of roughly k model fits and increased computation.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Evaluation Setup

Reserve a final test set and keep it untouched until model selection and threshold selection are complete. Use the remaining training data for cross-validation, with each held-out fold serving as a validation set.

  • Classification: StratifiedKFold preserves class proportions as closely as integer fold sizes permit.

  • Ordinary data: KFold is suitable when observations are independent.

  • Grouped data: GroupKFold prevents related observations from appearing in both training and validation folds.

  • Time series: TimeSeriesSplit respects temporal order.

Fit preprocessing inside each training fold. A scikit-learn Pipeline ensures that scaling, imputation, encoding, and feature selection do not learn from validation data.

How It Works

For k folds:

  1. Hold out one fold for validation.

  2. Fit the estimator on the other k - 1 folds.

  3. Compute the chosen metric on the held-out fold.

  4. Repeat for all folds, then summarize the mean and variability.

Use identical folds when comparing models so that paired per-fold differences are meaningful.

Metrics and Formulas

For binary classification, let TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives; N is the sample count; y_i is the binary target; and p_i is its predicted positive probability.

Metric Formula and purpose
Accuracy (TP + TN) / N; overall correctness, but misleading for imbalanced classes
Precision TP / (TP + FP); proportion of predicted positives that are correct
Recall TP / (TP + FN); proportion of actual positives detected
F1 2 * precision * recall / (precision + recall); harmonic mean of precision and recall
Specificity TN / (TN + FP); proportion of actual negatives correctly identified
ROC-AUC With TPR = TP / (TP + FN) and FPR = FP / (FP + TN), the ROC curve plots (FPR, TPR); ROC-AUC is its area
PR-AUC The PR curve plots (recall, precision); PR-AUC is its area and is usually more informative for rare positives
Log loss -(1/N) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)); penalizes incorrect probability estimates
Brier score (1/N) * sum((p_i - y_i)^2); mean squared probability error

A reliability curve compares predicted probability bins with their observed positive rates.

For regression, y_i is the actual value, y_hat_i is the prediction, and N is the sample count:

  • MAE: (1/N) * sum(abs(y_i - y_hat_i)); interpretable and less sensitive to large errors.

  • MSE: (1/N) * sum((y_i - y_hat_i)^2); penalizes large errors.

  • RMSE: sqrt(MSE); uses the target’s units.

  • R²: 1 - SSE / SST, where SSE is the sum of squared errors and SST is the total sum of squares around the target mean.

  • MAPE: (100/N) * sum(abs((y_i - y_hat_i) / y_i)); unstable when targets are near zero.

  • Quantile loss: max(q * r, (q - 1) * r), where r = y_i - y_hat_i and q is the target quantile.

Model Selection

GridSearchCV evaluates an exhaustive parameter grid, while RandomizedSearchCV samples specified parameter distributions; both can receive the same cross-validation splitter and pipeline. In nested cross-validation, an inner loop selects hyperparameters and an untouched outer fold estimates the selected procedure. Compare candidates using paired fold differences, a confidence interval, or a paired statistical test, and compare them with a DummyClassifier or DummyRegressor baseline.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data,
    data.target,
    test_size=0.20,
    stratify=data.target,
    random_state=42,
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
    "neg_log_loss": "neg_log_loss",
}

# Evaluate several metrics on training data only.
results = cross_validate(
    model, X_train, y_train,
    cv=cv,
    scoring=scoring,
    n_jobs=-1,
    error_score="raise",
)
for name, values in results.items():
    print(f"{name}: mean={values.mean():.3f}, std={values.std():.3f}")

# Select the F1 threshold from out-of-fold training predictions.
oof_probability = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

precision, recall, thresholds = precision_recall_curve(y_train, oof_probability)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) != 0,
)
best_threshold = float(thresholds[np.argmax(f1)])

# Fit once and evaluate the untouched test set.
model.fit(X_train, y_train)
test_probability = model.predict_proba(X_test)[:, 1]
test_prediction = (test_probability >= best_threshold).astype(int)

print(classification_report(y_test, test_prediction, digits=3))
print("Confusion matrix:", confusion_matrix(y_test, test_prediction))
print("ROC-AUC:", roc_auc_score(y_test, test_probability))
print("PR-AUC:", average_precision_score(y_test, test_probability))
print("Selected threshold:", best_threshold)

Common Pitfalls

  • Leakage: Fitting preprocessing before splitting contaminates validation results.

  • Wrong splitter: Random folds are unsuitable for time-ordered or grouped observations.

  • Imbalanced classes: Accuracy can conceal poor minority-class performance; inspect precision, recall, F1, PR-AUC, and the confusion matrix.

  • Repeated tuning: Reusing the same folds for many decisions eventually overfits them; nested CV gives a more honest estimate.

  • Weak baseline: Confirm that a complex model improves on a dummy estimator.

  • Preprocessing: Scaling is needed for k-NN, SVM, and PCA but usually unnecessary for trees; keep every learned transform in the pipeline.

  • Business mismatch: Offline metrics may not represent online outcomes; validate important changes with A/B tests.

Interviewers commonly ask why the test set remains untouched, why StratifiedKFold is preferred for classification, and how nested cross-validation separates tuning from final evaluation.

34

How can validation curves reveal overfitting in scikit-learn?

Validation-curve overfitting detection compares training and cross-validation performance across model-complexity levels. Overfitting is visible when training error continues decreasing while validation error stops improving or starts increasing. The main trade-off is fitting the training data more closely versus maintaining generalization to unseen data.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Evaluation Setup

A reliable evaluation pipeline separates three responsibilities:

  • Training set: Fits model parameters and learns preprocessing steps.

  • Validation folds: Estimate performance for each candidate complexity and select the model.

  • Test set: Remains untouched until final evaluation.

In this example, polynomial regression simulates increasing model complexity. Lower degrees underfit the nonlinear relationship, while excessive degrees can fit noise. validation_curve trains and scores each degree on five training folds; the separate test set is used only after selecting the best degree.

For regression, the validation metric is:

RMSE = sqrt((1 / n) * sum((y_i - y_hat_i)^2))

Here, n is the number of observations, y_i is the actual value, and y_hat_i is the predicted value. Lower RMSE is better.

How Overfitting Appears

A typical validation curve has three regions:

  • Underfitting: Both training and validation errors are high.

  • Good fit: Training error decreases while validation error reaches a minimum.

  • Overfitting: Training error keeps decreasing, but validation error rises.

The widening gap between the two curves indicates that the model is memorizing training-specific noise. The selected complexity should come from the lowest validation error, not the lowest training error.

Code Example

import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold, train_test_split, validation_curve
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(42)

# Synthetic nonlinear relationship with random noise.
X = rng.uniform(-2.0, 2.0, size=(600, 1))
y = (
    0.7 * X[:, 0] ** 2
    + np.sin(2.0 * X[:, 0])
    + rng.normal(0.0, 0.35, size=600)
)

# The test set is reserved for final evaluation.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

degrees = np.arange(1, 13)

# Preprocessing is fitted separately inside each cross-validation fold.
pipeline = Pipeline([
    ("poly", PolynomialFeatures(include_bias=False)),
    ("scale", StandardScaler()),
    ("model", LinearRegression()),
])

train_scores, validation_scores = validation_curve(
    estimator=pipeline,
    X=X_train,
    y=y_train,
    param_name="poly__degree",
    param_grid=[{"poly__degree": degree} for degree in degrees],
    cv=KFold(n_splits=5, shuffle=True, random_state=42),
    scoring="neg_mean_squared_error",
    n_jobs=-1,
)

# scikit-learn returns negative MSE because larger scores are better.
train_rmse = np.sqrt(-train_scores)
validation_rmse = np.sqrt(-validation_scores)

mean_train_rmse = train_rmse.mean(axis=1)
mean_validation_rmse = validation_rmse.mean(axis=1)

# Select complexity using validation performance only.
best_index = np.argmin(mean_validation_rmse)
best_degree = int(degrees[best_index])

# Fit the selected model on the complete training set.
final_model = Pipeline([
    ("poly", PolynomialFeatures(include_bias=False)),
    ("scale", StandardScaler()),
    ("model", LinearRegression()),
])
final_model.set_params(poly__degree=best_degree)
final_model.fit(X_train, y_train)

test_predictions = final_model.predict(X_test)
test_rmse = np.sqrt(mean_squared_error(y_test, test_predictions))

print(f"Selected polynomial degree: {best_degree}")
print(f"Mean training RMSE: {mean_train_rmse[best_index]:.3f}")
print(f"Mean validation RMSE: {mean_validation_rmse[best_index]:.3f}")
print(f"Final test RMSE: {test_rmse:.3f}")

plt.figure(figsize=(8, 5))
plt.plot(degrees, mean_train_rmse, marker="o", label="Training RMSE")
plt.plot(
    degrees,
    mean_validation_rmse,
    marker="o",
    label="Validation RMSE",
)
plt.axvline(best_degree, color="gray", linestyle="--", label="Selected degree")
plt.xlabel("Polynomial degree")
plt.ylabel("RMSE")
plt.title("Validation Curve for Overfitting Detection")
plt.legend()
plt.tight_layout()
plt.show()

Interpreting the Result

The printed test RMSE provides an unbiased final estimate because the test data did not influence preprocessing, degree selection, or model fitting. The exact selected degree and scores can vary slightly across scikit-learn versions, but the characteristic pattern remains: excessive polynomial degrees reduce training RMSE while worsening validation RMSE.

For classification, the same technique can use neg_log_loss, ROC-AUC, or another suitable scorer. In that case, look for improving training scores accompanied by degrading validation scores.

Common Pitfalls

  • Data leakage: Fit scaling, encoding, imputation, and feature selection only on training folds. A Pipeline prevents this within validation_curve.

  • Tuning on the test set: Do not repeatedly evaluate different models on the test set; it stops being an unbiased estimate.

  • Choosing by training error: Very low training error does not indicate the best model.

  • Invalid splitting: Use StratifiedKFold for classification, GroupKFold for grouped observations, and TimeSeriesSplit for time-ordered data.

  • Ignoring variance: One validation split can be misleading; use several folds and inspect the mean and variability of scores.

  • Unrealistic data size: Extremely high-degree models may become unstable with small datasets, so compare only meaningful complexity levels.

35

How do you plot an ROC curve and calculate AUC?

An ROC curve shows how a classifier’s true-positive rate changes with its false-positive rate as the decision threshold varies. ROC-AUC summarizes that curve with one number: 1.0 represents perfect ranking, while 0.5 represents random ranking. Its main trade-off is that it measures threshold-independent discrimination but does not identify the best threshold and can appear overly optimistic for severely imbalanced datasets.

Read full answer

Applies toscikit-learn 1.5+ with Python 3.12

Evaluation Setup

Split the data into training, validation, and test sets. Fit the model on the training set, select its threshold on the validation set, and keep the test set untouched until final evaluation.

Use cross-validation on the training portion when appropriate:

  • KFold: general cross-validation.

  • StratifiedKFold: classification when preserving class proportions.

  • GroupKFold: observations from the same group remain together.

  • TimeSeriesSplit: time-ordered data; do not use random splits.

Use GridSearchCV or RandomizedSearchCV for tuning, and nested cross-validation when the reported tuning score must remain an honest estimate. Compare models using paired fold scores, confidence intervals, or paired tests, and include a DummyClassifier baseline.

ROC Curve and AUC

For each decision threshold, the ROC curve contains a point made from:

  • True-positive rate (TPR) = TP / (TP + FN): the proportion of actual positives detected.

  • False-positive rate (FPR) = FP / (FP + TN): the proportion of actual negatives incorrectly classified as positive.

The AUC is the area under the curve: ROC-AUC = ∫[0,1] TPR(FPR) d(FPR), where the integral denotes the accumulated area as FPR moves from 0 to 1. It is also the probability that a randomly selected positive is ranked above a randomly selected negative, with a tied pair contributing one half.

Related Metrics

Let TP, TN, FP, and FN be confusion-matrix counts, n the number of examples, y the binary true label, and p the predicted probability of the positive class:

  • Accuracy = (TP + TN) / n; it can be misleading with imbalanced classes.

  • Precision = TP / (TP + FP); the proportion of positive predictions that are correct.

  • Recall = TP / (TP + FN); the proportion of positives detected.

  • Specificity = TN / (TN + FP); the proportion of negatives correctly identified.

  • F1 = 2 * precision * recall / (precision + recall).

  • PR curve: plots precision against recall across thresholds; PR-AUC is the accumulated area under that curve. average_precision_score computes average precision, a commonly used non-interpolated PR summary, and is often more informative for rare positives.

  • Log loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p)); it penalizes confident incorrect probabilities.

  • Brier score = (1/n) * sum((p - y)^2); it measures probability calibration.

  • Reliability curve: plots observed positive frequency against mean predicted probability in each bin; the diagonal represents perfect calibration.

Code Example

import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# In this dataset, 1 = malignant and 0 = benign.
data = load_breast_cancer()
X, y = data.data, data.target

# Reserve the test set for final evaluation only.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25, stratify=y_dev, random_state=42
)

# Preprocessing is fitted separately within every training fold.
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=2000, random_state=42)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}

cv_results = cross_validate(
    pipeline, X_train, y_train, cv=cv, scoring=scoring, n_jobs=-1
)
for metric, scores in cv_results.items():
    print(f"{metric}: {scores.mean():.3f}")

# Select an F1-oriented threshold using validation PR data.
model = pipeline.fit(X_train, y_train)
val_probability = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_probability)

f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) > 0,
)
selected_threshold = thresholds[np.argmax(f1)]

# Keep the same fitted model so the validation threshold remains paired
# with the model evaluated on the untouched test set.
test_probability = model.predict_proba(X_test)[:, 1]
fpr, tpr, _ = roc_curve(y_test, test_probability)
auc = roc_auc_score(y_test, test_probability)
average_precision = average_precision_score(y_test, test_probability)
test_prediction = (test_probability >= selected_threshold).astype(int)

print(f"Selected threshold: {selected_threshold:.3f}")
print(f"Test ROC-AUC: {auc:.3f}")
print(f"Test average precision: {average_precision:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_prediction):.3f}")
print(classification_report(
    y_test, test_prediction, target_names=["benign", "malignant"]
))
print(confusion_matrix(y_test, test_prediction))

plt.figure(figsize=(6, 5))
plt.plot(fpr, tpr, label=f"ROC curve (AUC = {auc:.3f})")
plt.plot([0, 1], [0, 1], linestyle="--", color="gray", label="Random")
plt.xlabel("False-positive rate")
plt.ylabel("True-positive rate")
plt.title("ROC curve")
plt.legend()
plt.tight_layout()
plt.show()

Common Pitfalls

  • Using hard predictions: Thresholded labels discard score information; use probabilities or continuous decision scores.

  • Data leakage: Fit scalers, encoders, and imputers only on training folds using a Pipeline.

  • Test-set tuning: Do not select the threshold, model, or hyperparameters using test results.

  • Ignoring imbalance: Compare PR-AUC or average precision, precision, and recall alongside ROC-AUC.

  • Wrong split strategy: Do not randomly split time-ordered or grouped observations.

  • Confusing ranking with calibration: A high ROC-AUC does not guarantee calibrated probabilities.

  • Ignoring business outcomes: Offline metrics may not reflect online impact; validate important product decisions with A/B tests.

36

How should you evaluate a time-series forecasting model?

Evaluate a time-series forecasting model with a strictly chronological validation setup, realistic forecast horizons, and metrics tied to the cost of forecast errors. This prevents future information from leaking into model selection and provides an honest estimate of operational performance. The main trade-off is realism: rolling or fixed-origin multi-step evaluations are more representative but harder to implement and compute than random splits or one-step tests.

Read full answer

Applies toNumPy 2.x, pandas 2.2+, and scikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Train, validation, and test splits: Split observations by time, with training data first, validation data next, and the test period last. Keep the test set untouched until the model, hyperparameters, and preprocessing are finalized.

  • Cross-validation: Use TimeSeriesSplit, which trains on earlier observations and validates on later ones without shuffling. Set gap or otherwise exclude training targets that overlap the forecast horizon.

  • Forecast horizon: Evaluate one-step-ahead and operationally relevant multi-step forecasts separately. Supplying actual observations during the test period is appropriate only for rolling one-step forecasting; fixed-origin multi-step forecasting must use only information available at each forecast origin.

  • Grouped or multiple series: Preserve both temporal ordering and group boundaries. GroupKFold keeps groups together but does not guarantee chronological ordering, so a group-aware temporal design may be necessary.

  • Model selection: Use GridSearchCV or RandomizedSearchCV with TimeSeriesSplit. Use nested time-series cross-validation when the reported cross-validation score must also account honestly for hyperparameter selection.

  • Baseline: Compare against a naive forecast, such as the previous observed value. Compare models using paired fold-level error differences or bootstrap confidence intervals rather than relying on one lucky split.

Metrics and Selection

For n paired forecasts, let i index a forecast, y_i be its actual value, y_hat_i its forecast, and y_bar the mean actual value:

  • MAE: MAE = sum(abs(y_i - y_hat_i)) / n. It uses the target’s units and is relatively robust to outliers.

  • MSE: MSE = sum((y_i - y_hat_i)^2) / n. It penalizes large errors more heavily.

  • RMSE: RMSE = sqrt(MSE). It remains in the target’s units and emphasizes large errors.

  • R²: R² = 1 - sum((y_i - y_hat_i)^2) / sum((y_i - y_bar)^2). It measures improvement over a constant-mean predictor but can be misleading for nonstationary series.

  • MAPE: MAPE = 100 * sum(abs((y_i - y_hat_i) / y_i)) / n. It is scale-free but unstable or undefined when actual values are near zero.

  • Quantile loss: loss = max(q * (y_i - y_hat_i), (1 - q) * (y_hat_i - y_i)), where q is the target quantile. It evaluates probabilistic forecasts and penalizes under- and over-forecasting differently.

Report results by horizon. For probabilistic forecasts, also examine interval coverage and width; for business decisions, map offline errors to actual costs and validate important changes with an online A/B test where possible.

Code Example

This example evaluates one-step-ahead forecasting. At origin t, lag_1 is y_t and the target is y_(t+1), so no future value is used as a feature.

import numpy as np
import pandas as pd

from sklearn.base import clone
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import (
    mean_absolute_error,
    r2_score,
    root_mean_squared_error,
)
from sklearn.model_selection import GridSearchCV, TimeSeriesSplit
from sklearn.pipeline import Pipeline

rng = np.random.default_rng(42)

# Synthetic series with trend, seasonality, and reproducible noise.
n = 240
time = np.arange(n)
series = (
    10
    + 0.08 * time
    + 2 * np.sin(2 * np.pi * time / 12)
    + rng.normal(0, 0.5, n)
)
data = pd.DataFrame({"y": series})

# Features are measured backward from the forecast origin t.
for lag in (1, 2, 3, 12):
    data[f"lag_{lag}"] = data["y"].shift(lag - 1)

data["target"] = data["y"].shift(-1)
data = data.dropna().reset_index(drop=True)

features = ["lag_1", "lag_2", "lag_3", "lag_12"]
X = data[features]
y = data["target"]

# Chronological train, validation, and test periods.
train_end = int(len(data) * 0.70)
validation_end = int(len(data) * 0.85)

X_train, y_train = X.iloc[:train_end], y.iloc[:train_end]
X_validation, y_validation = (
    X.iloc[train_end:validation_end],
    y.iloc[train_end:validation_end],
)
X_test, y_test = X.iloc[validation_end:], y.iloc[validation_end:]

model = Pipeline([
    ("regressor", HistGradientBoostingRegressor(random_state=42)),
])

search = GridSearchCV(
    estimator=model,
    param_grid={
        "regressor__learning_rate": [0.03, 0.1],
        "regressor__max_iter": [100, 200],
    },
    scoring="neg_mean_absolute_error",
    cv=TimeSeriesSplit(n_splits=3),
    n_jobs=-1,
)
search.fit(X_train, y_train)

def regression_metrics(actual, predicted):
    return {
        "MAE": mean_absolute_error(actual, predicted),
        "RMSE": root_mean_squared_error(actual, predicted),
        "R2": r2_score(actual, predicted),
    }

# Evaluate the selected configuration on validation data.
validation_model = clone(search.best_estimator_).fit(X_train, y_train)
validation_predictions = validation_model.predict(X_validation)

# The naive forecast predicts that the next value equals the current value.
naive_validation_predictions = X_validation["lag_1"]

print("Best parameters:", search.best_params_)
print("Validation model:", regression_metrics(y_validation, validation_predictions))
print("Validation naive:", regression_metrics(y_validation, naive_validation_predictions))

# Refit on train plus validation only after model selection.
final_model = clone(search.best_estimator_).fit(
    pd.concat([X_train, X_validation], ignore_index=True),
    pd.concat([y_train, y_validation], ignore_index=True),
)
test_predictions = final_model.predict(X_test)
naive_test_predictions = X_test["lag_1"]

print("Test model:", regression_metrics(y_test, test_predictions))
print("Test naive:", regression_metrics(y_test, naive_test_predictions))

Common Pitfalls

  • Temporal leakage: Randomly shuffled splits can train on future observations.

  • Test-set tuning: Repeatedly inspecting test results turns the test set into an implicit validation set.

  • Leaky features: Do not use future values, future-derived statistics, or unavailable target information.

  • Wrong horizon: One-step results can overstate performance for multi-step operational use.

  • Ignoring baselines and uncertainty: Compare with naive forecasts and evaluate prediction intervals when decisions depend on uncertainty.

  • Metric mismatch: Offline MAE or RMSE may not reflect business impact; validate important decisions with online experiments where possible.

37

How should models be evaluated on imbalanced datasets?

Imbalanced-dataset evaluation should use stratified validation, metrics that expose minority-class errors, and thresholds chosen for the real costs of false positives and false negatives. The main trade-off is precision versus recall: reducing false alarms can reduce the number of true positives detected.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

Use a three-way train/validation/test split and keep the test set untouched until model and threshold selection are complete.

  • Use StratifiedKFold to preserve approximately the same class proportions in each fold.

  • Use GroupKFold when rows from the same patient, customer, or other entity must not cross folds.

  • Use TimeSeriesSplit for temporal data so future observations never enter earlier training folds.

  • Fit scaling, imputation, feature selection, and resampling only on training folds. Put preprocessing in a Pipeline to prevent leakage.

  • Report cross-validation means with variability, such as standard deviations or confidence intervals. Use nested cross-validation when the inner loop selects hyperparameters and the outer loop estimates performance.

Classification and Probability Metrics

Let TP, TN, FP, and FN be true positives, true negatives, false positives, and false negatives. Let n be the number of examples.

Metric Formula Interpretation
Accuracy (TP + TN) / n Fraction of all predictions that are correct
Precision TP / (TP + FP) Fraction of predicted positives that are correct
Recall TP / (TP + FN) Fraction of actual positives detected
F1 2 * (precision * recall) / (precision + recall) Harmonic mean of precision and recall; use 0 when both are zero
Specificity TN / (TN + FP) Fraction of actual negatives correctly identified

Accuracy can hide failure on the minority class: with 100 examples and 5 positives, always predicting negative gives 95% accuracy and zero recall. Macro F1 weights every class equally, while weighted F1 weights classes by frequency.

  • ROC curve: plots true-positive rate TP / (TP + FN) against false-positive rate FP / (TN + FP) across thresholds. ROC-AUC is the probability that a positive is scored above a negative, with half credit for ties; it can look optimistic under severe imbalance because the false-positive rate is diluted.

  • Precision-recall curve: plots precision against recall across thresholds. PR-AUC, or average precision, is usually more informative for rare positives. Average precision is sum((R_i - R_(i-1)) * P_i), where R_i is recall at threshold i, P_i is precision there, and R_0 = 0.

  • For probabilities, let y_i be an actual binary label and p_i its predicted positive probability. Log loss is -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where log is the natural logarithm. Brier score is (1/n) * sum((p_i - y_i)^2); lower values indicate better probability accuracy.

  • Use reliability curves to compare predicted probabilities with observed positive rates. Good discrimination does not guarantee good calibration.

For regression, let y_i be targets, f_i predictions, y_bar the target mean, and q a quantile. MAE is (1/n) * sum(|y_i - f_i|); MSE is (1/n) * sum((y_i - f_i)^2); RMSE is sqrt(MSE); and R² is 1 - SSE / SST, where SSE = sum((y_i - f_i)^2) and SST = sum((y_i - y_bar)^2). MAPE is (100/n) * sum(|y_i - f_i| / |y_i|) but is unstable near zero. Quantile loss is (1/n) * sum(max(q * (y_i - f_i), (q - 1) * (y_i - f_i))).

Thresholds, Baselines, and Model Selection

  • A fixed 0.5 cutoff is often unsuitable. Select a threshold on validation or cross-validated predictions using precision_recall_curve, then freeze it before testing.

  • Lower the threshold when recall is more valuable; raise it when false positives are more costly.

  • Compare against DummyClassifier(strategy="prior") or strategy="most_frequent".

  • Use GridSearchCV for small searches and RandomizedSearchCV for larger ones. Compare models on identical splits with paired confidence intervals or suitable paired tests.

  • Offline metrics may not match deployed behavior. Validate important changes through an A/B test or another controlled online experiment.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# Keep the test set separate; use validation data to choose the threshold.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25, stratify=y_dev, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
])

scoring = {
    "roc_auc": "roc_auc",
    "pr_auc": "average_precision",
    "f1_macro": "f1_macro",
    "balanced_accuracy": "balanced_accuracy",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

model_cv = cross_validate(
    model, X_train, y_train, cv=cv, scoring=scoring, n_jobs=-1
)

baseline = Pipeline([
    ("classifier", DummyClassifier(strategy="prior"))
])
baseline_cv = cross_validate(
    baseline, X_train, y_train, cv=cv, scoring=scoring, n_jobs=-1
)

# Select a threshold that maximizes positive-class F1 on validation data.
model.fit(X_train, y_train)
val_probability = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_probability)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) != 0,
)
threshold = thresholds[np.argmax(f1)]

# Evaluate once on the untouched test set.
test_probability = model.predict_proba(X_test)[:, 1]
prediction = (test_probability >= threshold).astype(int)

print("Model CV:", {k: round(model_cv[f"test_{k}"].mean(), 3) for k in scoring})
print("Baseline CV:", {k: round(baseline_cv[f"test_{k}"].mean(), 3) for k in scoring})
print("Chosen threshold:", threshold)
print(classification_report(y_test, prediction, zero_division=0))
print("Confusion matrix:", confusion_matrix(y_test, prediction, labels=[0, 1]))
print("ROC-AUC:", roc_auc_score(y_test, test_probability))
print("Average precision:", average_precision_score(y_test, test_probability))

Common Pitfalls

  • Reporting accuracy alone on a severely imbalanced dataset.

  • Tuning hyperparameters, features, or thresholds on the test set.

  • Randomly splitting time-ordered or grouped observations.

  • Resampling before splitting, which leaks validation information.

  • Hiding minority-class failures behind micro-averaged metrics.

  • Comparing models with different splits, preprocessing, or evaluation protocols.

  • Ignoring calibration, operational costs, or differences between offline and business metrics.

38

How do you validate a natural language processing model?

Model validation checks whether an NLP model meets its task and error-cost requirements on unseen, representative data. It combines leakage-free splitting, cross-validation, task-appropriate metrics, human review where needed, and online experiments, trading evaluation thoroughness for time and cost.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Evaluation Setup

  • Define the evaluation unit: use the natural unit of the task, such as a document, dialogue, document pair, or user interaction, and define the target label or score.

  • Reserve a test set: use training data for fitting, validation data for hyperparameters and thresholds, and evaluate the test set only after all decisions are final.

  • Match splitting to the data: use KFold for regression or sufficiently large balanced datasets, StratifiedKFold for classification, GroupKFold to keep users or conversations in one fold, and TimeSeriesSplit for temporal order.

  • Prevent leakage: split before fitting tokenizers, vocabularies, normalizers, imputers, or feature selectors. Put text preprocessing and the estimator in a scikit-learn Pipeline so cross-validation fits each fold independently.

Metrics

Let n be the number of examples, y the true binary label, and p the predicted probability.

  • Classification: accuracy = (TP + TN) / n; precision = TP / (TP + FP); recall = TP / (TP + FN); F1 = 2 * precision * recall / (precision + recall); specificity = TN / (TN + FP). TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives. Accuracy can hide poor minority-class performance.

  • ROC-AUC: the ROC curve plots TPR = TP / (TP + FN) against FPR = FP / (FP + TN). ROC-AUC is the probability that a random positive receives a higher score than a random negative, with ties counted as one half.

  • PR-AUC: the precision-recall curve plots precision against recall and is generally more informative for rare positives. A common summary is average precision, AP = sum((R_n - R_(n-1)) * P_n), where R_n and P_n are recall and precision at ordered threshold points.

  • Log loss: -(1/n) * sum(y * log(p) + (1-y) * log(1-p)); lower is better, and confident incorrect predictions receive large penalties.

  • Calibration: Brier score = mean((y-p)^2); lower is better. A reliability curve compares mean predicted probability with the observed positive frequency within each probability bin.

  • Regression: MAE = mean(abs(y-y_hat)); MSE = mean((y-y_hat)^2); RMSE = sqrt(MSE); R² = 1 - SSE/SST; MAPE = mean(abs(y-y_hat)/abs(y)) * 100, which is unstable near zero; quantile loss is max(tau*(y-y_hat), (tau-1)*(y-y_hat)). Here y_hat is the prediction, tau the quantile level, and SSE and SST the error and total sums of squares.

  • Generative NLP: combine exact match, ROUGE, BLEU, or semantic similarity with blinded human ratings of relevance, factuality, fluency, and safety.

Model Selection and Production Validation

  • Tune with GridSearchCV or RandomizedSearchCV inside the leakage-safe pipeline. Use nested cross-validation when estimating performance after hyperparameter selection, and compare candidates using paired fold-level differences or bootstrap confidence intervals.

  • Compare against DummyClassifier and a simple linear or keyword baseline. For grouped or temporal data, ensure every fold respects group and time boundaries.

  • Validate offline metrics against an online business metric such as successful-task rate, then use an A/B test to measure production impact. Record package versions, seeds, and serving hardware, and benchmark latency on the target CPU or GPU.

Code Example

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate, train_test_split, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.metrics import (
    average_precision_score, classification_report, confusion_matrix,
    precision_recall_curve, roc_auc_score,
)

rng = np.random.default_rng(42)

positive = np.array([
    f"{adjective} {noun}"
    for adjective in "great excellent amazing wonderful strong lovely".split()
    for noun in "movie story performance acting direction".split()
])
negative = np.array([
    f"{adjective} {noun}"
    for adjective in "awful terrible horrible dull bad disappointing".split()
    for noun in "movie story performance acting direction".split()
])
rng.shuffle(positive)
rng.shuffle(negative)

texts = np.concatenate([positive, negative])
labels = np.r_[np.ones(positive.size, dtype=int), np.zeros(negative.size, dtype=int)]

X_train, X_temp, y_train, y_temp = train_test_split(
    texts, labels, test_size=0.40, random_state=42, stratify=labels
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
scores = cross_validate(model, X_train, y_train, cv=cv, scoring=scoring)

model.fit(X_train, y_train)
val_probabilities = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_probabilities)
f1_values = (
    2 * precision[:-1] * recall[:-1]
    / np.maximum(precision[:-1] + recall[:-1], 1e-12)
)
best_threshold = thresholds[np.argmax(f1_values)]

test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)
baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)

print("Cross-validation means:", {
    name: round(float(scores[f"test_{name}"].mean()), 3) for name in scoring
})
print("Selected threshold:", round(float(best_threshold), 3))
print(classification_report(y_test, test_predictions, zero_division=0))
print("Confusion matrix:\n", confusion_matrix(y_test, test_predictions))
print("ROC-AUC:", round(roc_auc_score(y_test, test_probabilities), 3))
print("PR-AUC (average precision):",
      round(average_precision_score(y_test, test_probabilities), 3))
print("Dummy accuracy:", round(baseline.score(X_test, y_test), 3))

Common Pitfalls

  • Leakage: fitting preprocessing before splitting exposes test information.

  • Test-set tuning: repeatedly using test results turns the test set into another validation set.

  • Invalid splits: random splits can leak users, conversations, or future observations.

  • Wrong metric or data quality: account for class imbalance, missing or empty documents, and overfitting.

  • Misleading probability estimates: high AUC does not imply calibration or strong production performance.

39

How do you evaluate a recommendation system effectively?

Offline evaluation measures recommendation quality on data not used to fit the model, while online A/B testing measures causal product impact with live users. The strongest approach combines ranking metrics, business metrics, and long-term outcomes. The main trade-off is that offline evaluation is inexpensive but sensitive to observational bias, whereas online experiments are realistic but costly and require careful randomization.

Read full answer

Applies tomachine learning concepts (Python 3.12; NumPy 2.x, pandas 2.2+, and scikit-learn 1.5+)

Evaluation Setup

  • Split interactions chronologically into train, validation, and test sets. Fit on train, tune on validation, and evaluate on test only once.

  • Use rolling origins or TimeSeriesSplit for time order. Use KFold only for exchangeable observations, StratifiedKFold for classification-label proportions, and GroupKFold to keep sessions or repeated users together.

  • Prevent leakage from future interactions, post-click behavior, full-data popularity statistics, and features unavailable at recommendation time.

  • Evaluate cold-start users and new items separately, and compare against popularity, random, and business-rule baselines.

Metrics That Matter

Metric Meaning Important limitation
Precision@K Relevant recommendations among the first K Penalizes systems returning fewer than K results
Recall@K Relevant items found divided by known relevant items Distorted by incomplete labels and many observed positives
NDCG@K Discounted ranking quality Needs meaningful binary or graded relevance
MAP@K Average precision over relevant-item positions Sensitive to relevant items appearing early
Hit rate@K Users receiving at least one relevant item Ignores the positions of other results
Coverage Catalog fraction appearing in recommendations Can be high despite poor relevance
Diversity and novelty Variety and unfamiliarity of results Requires defensible definitions

For binary relevance, NDCG@K = DCG@K / IDCG@K. DCG@K sums gain / log2(rank + 1), where gain is 1 for a relevant item and 0 otherwise, and rank is one-based; IDCG@K places the known relevant items first.

When training a binary candidate classifier, with true-positive, true-negative, false-positive, and false-negative counts denoted TP, TN, FP, and FN, use accuracy = (TP + TN) / M, precision = TP / (TP + FP), recall = TP / (TP + FN), F1 = 2TP / (2TP + FP + FN), and specificity = TN / (TN + FP), where M is the number of examples. A ROC curve plots TPR = TP / (TP + FN) against FPR = FP / (FP + TN) over thresholds, and ROC-AUC is its area. A precision-recall curve plots precision against recall; PR-AUC is usually more informative for rare positives. For label y and probability p, log loss is -sum(y*log(p) + (1-y)*log(1-p))/M, and Brier score is sum((p-y)^2)/M; a reliability curve compares mean predicted probability with observed frequency.

For rating prediction, with target y, prediction yhat, and mean target ybar, use MAE = sum(|y-yhat|)/M, MSE = sum((y-yhat)^2)/M, RMSE = sqrt(MSE), and R^2 = 1 - sum((y-yhat)^2)/sum((y-ybar)^2). MAPE = sum(|y-yhat|/|y|)/M is unstable near zero; quantile loss is the corresponding pinball loss.

Code Example

The first example selects K on validation data and evaluates a popularity baseline once on the test set. Test-time history includes validation interactions.

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n_users, n_items, n_sessions = 200, 30, 8

# Each user has unique items, avoiding ambiguous duplicate held-out events.
items = np.vstack([
    rng.choice(n_items, size=n_sessions, replace=False)
    for _ in range(n_users)
])
data = pd.DataFrame({
    "user_id": np.repeat(np.arange(n_users), n_sessions),
    "item_id": items.ravel(),
    "timestamp": (
        np.tile(np.arange(n_sessions), n_users) * n_users
        + np.repeat(np.arange(n_users), n_sessions)
    ),
})
data["rank_from_end"] = data.groupby("user_id").cumcount(ascending=False)
data["split"] = np.select(
    [data["rank_from_end"].eq(0), data["rank_from_end"].eq(1)],
    ["test", "validation"],
    default="train",
)

train = data[data["split"].eq("train")]
popularity = train["item_id"].value_counts().sort_values(
    ascending=False, kind="stable"
)

def recommend(user_id, k, history):
    seen = set(history.loc[history["user_id"].eq(user_id), "item_id"])
    return [int(i) for i in popularity.index if i not in seen][:k]

def evaluate(split, k):
    past = ["train"] if split == "validation" else ["train", "validation"]
    history = data[data["split"].isin(past)]
    target = data[data["split"].eq(split)]
    results = []

    for user_id, rows in target.groupby("user_id"):
        recommended = recommend(user_id, k, history)
        relevant = set(rows["item_id"])
        hits = sum(item in relevant for item in recommended)
        dcg = sum(
            1.0 / np.log2(position + 2)
            for position, item in enumerate(recommended)
            if item in relevant
        )
        idcg = sum(
            1.0 / np.log2(position + 2)
            for position in range(min(len(relevant), k))
        )
        results.append({
            "precision": hits / k,
            "recall": hits / len(relevant),
            "hit_rate": float(hits > 0),
            "ndcg": dcg / idcg if idcg else 0.0,
        })

    return pd.DataFrame(results).mean()

validation_scores = {k: evaluate("validation", k) for k in (5, 10)}
best_k = max(validation_scores, key=lambda k: validation_scores[k]["ndcg"])
test_scores = evaluate("test", best_k)

print(f"Selected K={best_k} using validation NDCG")
print(test_scores.round(4).to_dict())

For pointwise binary diagnostics, preprocess inside a pipeline and select the classification threshold on validation data, not test data.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score, classification_report, confusion_matrix,
    precision_recall_curve, roc_auc_score,
)
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=42
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000)),
])
scoring = {
    "accuracy": "accuracy", "precision": "precision", "recall": "recall",
    "f1": "f1", "roc_auc": "roc_auc", "average_precision": "average_precision",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(pipeline, X_train, y_train, cv=cv, scoring=scoring)
print({name: round(float(values.mean()), 3) for name, values in scores.items()})

pipeline.fit(X_train, y_train)
val_probability = pipeline.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_probability)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1], precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]), where=(precision[:-1] + recall[:-1]) != 0,
)
threshold = float(thresholds[int(np.argmax(f1))])

pipeline.fit(np.concatenate([X_train, X_val]), np.concatenate([y_train, y_val]))
test_probability = pipeline.predict_proba(X_test)[:, 1]
prediction = (test_probability >= threshold).astype(int)
print(classification_report(y_test, prediction, zero_division=0))
print(confusion_matrix(y_test, prediction))
print("ROC-AUC:", roc_auc_score(y_test, test_probability))
print("PR-AUC:", average_precision_score(y_test, test_probability))

Model Selection and Production Checks

Tune candidate generators, ranking models, and rerankers separately with GridSearchCV or RandomizedSearchCV inside pipelines. Use nested cross-validation when tuning and reporting honest performance, and compare models on identical folds with confidence intervals or paired tests. Use DummyClassifier for pointwise baselines and popularity or business rules for ranking baselines. In production, measure p95 latency, memory, throughput, catalog freshness, diversity, novelty, and GPU-versus-CPU cost.

Common Pitfalls

  • Treating unobserved interactions as negatives despite exposure bias.

  • Tuning thresholds, hyperparameters, or features on the test set.

  • Using random splits for temporal or grouped interactions.

  • Relying on accuracy for imbalanced data or ignoring cold-start segments.

  • Optimizing popularity or short-term clicks while harming diversity or retention.

  • Assuming offline gains will appear online; validate them with sufficiently long user-level A/B tests and guardrail metrics.

40

How can clustering algorithm performance be evaluated?

A clustering algorithm can be evaluated using internal validity metrics, external ground-truth metrics, and stability analysis. Internal metrics measure cluster cohesion and separation without labels; external metrics measure agreement with known classes; stability analysis checks whether similar structure appears across resamples. The main trade-off is that internal metrics are practical but depend heavily on the data geometry, while external metrics require meaningful labels and may not represent the structure you actually want to discover.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

A typical evaluation workflow is:

  1. Split the data into training, validation, and test sets before fitting any transformations or models.

  2. Fit preprocessing and the clustering algorithm on the training set only.

  3. Use the validation set to select the number of clusters, distance metric, or algorithm.

  4. Optionally refit the final model on the training and validation sets.

  5. Evaluate once on the untouched test set.

Cluster labels produced by the algorithm are not used to fit it. When ground-truth labels exist only for external evaluation, they may be used to balance the evaluation split, but they must not be passed to clustering or used in silhouette-based model selection.

KFold is common for label-free resampling, GroupKFold keeps related observations in the same fold, and TimeSeriesSplit respects time order. StratifiedKFold is normally used for supervised classification with class labels rather than for label-free clustering.

Evaluation Metrics

Approach Examples What it measures Important caveat
Internal validity Silhouette score, Davies-Bouldin score, Calinski-Harabasz score Cluster compactness and separation Results depend on distance scale, cluster count, and outliers
External validity Adjusted Rand Index, normalized mutual information, homogeneity, completeness Agreement with known labels Labels must represent the structure of interest
Stability Reclustering after resampling or perturbation Reproducibility of discovered groups A stable but uninformative clustering is still possible

For the silhouette score, each observation receives:

silhouette(i) = (b(i) - a(i)) / max(a(i), b(i))

Here, a(i) is the average distance between observation i and other observations in its cluster, while b(i) is the smallest average distance between observation i and observations in another cluster. Values range from -1 to 1; higher values generally indicate better-defined clusters. In scikit-learn, an observation in a singleton cluster receives 0 because its within-cluster average distance is undefined.

The Davies-Bouldin score measures within-cluster scatter relative to separation between clusters, so lower is better. The Calinski-Harabasz score compares between-cluster separation with within-cluster variation, so higher is better. Neither should be interpreted in isolation.

For stability analysis, rerun clustering on bootstrap or subsampled data, or after small perturbations, and compare the resulting partitions with the original partition. Adjusted Rand Index or normalized mutual information can quantify agreement because they are invariant to arbitrary cluster-label renumbering.

Code Example

The following example selects the number of clusters using validation-set silhouette scores, then evaluates the final model on an untouched test set. The scaler is fitted inside a Pipeline, preventing preprocessing leakage.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import (
    adjusted_rand_score,
    completeness_score,
    homogeneity_score,
    normalized_mutual_info_score,
    silhouette_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Generate reproducible synthetic data with three known groups.
X, y = make_blobs(
    n_samples=300,
    centers=3,
    cluster_std=1.0,
    random_state=42,
)

# Split before fitting the scaler or clustering model.
X_train, X_temp, _, y_temp = train_test_split(
    X, y, test_size=0.40, random_state=42, stratify=y
)
X_val, X_test, _, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp
)


def build_model(n_clusters):
    return Pipeline(
        [
            ("scaler", StandardScaler()),
            ("kmeans", KMeans(
                n_clusters=n_clusters,
                n_init=10,
                random_state=42,
            )),
        ]
    )


# Select k using internal validation performance only.
validation_results = []
for n_clusters in range(2, 6):
    model = build_model(n_clusters)
    model.fit(X_train)

    validation_labels = model.predict(X_val)
    validation_features = model.named_steps["scaler"].transform(X_val)
    validation_results.append({
        "n_clusters": n_clusters,
        "silhouette": silhouette_score(
            validation_features, validation_labels
        ),
    })

best_result = max(
    validation_results,
    key=lambda result: result["silhouette"],
)
best_k = best_result["n_clusters"]

# Refit on training and validation data, keeping the test set untouched.
X_train_val = np.vstack([X_train, X_val])
final_model = build_model(best_k)
final_model.fit(X_train_val)

test_labels = final_model.predict(X_test)
test_features = final_model.named_steps["scaler"].transform(X_test)

# Internal and external test evaluation.
print("Selected number of clusters:", best_k)
print("Test silhouette score:",
      round(silhouette_score(test_features, test_labels), 3))
print("Test adjusted Rand index:",
      round(adjusted_rand_score(y_test, test_labels), 3))
print("Test normalized mutual information:",
      round(normalized_mutual_info_score(y_test, test_labels), 3))
print("Test homogeneity:",
      round(homogeneity_score(y_test, test_labels), 3))
print("Test completeness:",
      round(completeness_score(y_test, test_labels), 3))

Choosing and Comparing Algorithms

Compare algorithms using the same splits, preprocessing, metric suite, runtime measurements, and stability tests.

Algorithm How it works Strengths Weaknesses When to use Typical complexity
KMeans Assigns points to centroids and iteratively updates them Fast and scalable Requires k; sensitive to initialization, outliers, and cluster shape Compact, roughly spherical clusters O(k * n * d) per iteration
Agglomerative clustering Repeatedly merges the closest clusters into a hierarchy Shows nested cluster structure High memory use; can be sensitive to linkage choice Small or medium datasets, hierarchical analysis Typically O(n²) memory
DBSCAN Groups density-connected points and identifies noise Finds arbitrary shapes and outliers Sensitive to eps and feature scale; weaker in high dimensions Irregular clusters and explicit noise detection O(n²) worst case; faster with spatial indexing

Do not rely on a single score. A method with a slightly lower silhouette score may be preferable if it is more stable, interpretable, better suited to expected cluster sizes, or more useful for the downstream objective.

Common Pitfalls

  • Data leakage: Fitting a scaler, imputer, or feature selector on the entire dataset exposes test information to model selection. Use a Pipeline and fit it only on training data.

  • Ignoring feature scale: Distance-based methods such as KMeans and DBSCAN can produce misleading results when features have incompatible units.

  • Choosing only the best internal score: A metric may favor a particular geometry rather than meaningful clusters. Combine internal, external, and stability evidence.

  • Overfitting the validation set: Repeatedly trying many algorithms and thresholds can make validation performance optimistic. Preserve the test set for final evaluation.

  • Ignoring outliers and cluster size: Outliers can distort centroids, while algorithms such as KMeans assume approximately compact clusters.

  • Confusing offline and business metrics: A statistically separable clustering may not improve recommendations, fraud detection, or customer activation. Validate downstream usefulness with operational metrics and, where possible, an A/B test.

Common interviewer follow-ups include: “How would you choose the number of clusters?”, “What if no ground-truth labels exist?”, “How would you detect unstable clusters?”, and “Why might silhouette score favor the wrong solution?”

41

How do you implement StratifiedKFold evaluation for imbalanced classification?

StratifiedKFold cross-validation evaluates a classifier using folds that approximately preserve each class’s proportion, so minority-class examples are not concentrated in one split. It stabilizes comparison and threshold selection for imbalanced classification, but it does not rebalance the data or replace an untouched test set. The main trade-off is additional computation because the pipeline is trained once per fold.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

First reserve a stratified test set for final evaluation, then perform cross-validation only on the development data:

  1. Split development and test data with train_test_split(..., stratify=y).

  2. Create StratifiedKFold validation folds on the development data.

  3. Fit preprocessing and the classifier inside each fold using a Pipeline.

  4. Tune hyperparameters and the decision threshold using development results only.

  5. Fit the final pipeline on all development data and evaluate it once on the test set.

Unlike StratifiedKFold, plain KFold does not preserve class proportions. Use GroupKFold for grouped observations, StratifiedGroupKFold when grouped observations also need class balancing, and TimeSeriesSplit when temporal order must be preserved.

Metrics for Imbalanced Classification

Let TP, TN, FP, and FN mean true positives, true negatives, false positives, and false negatives.

Metric Formula Meaning
Accuracy (TP + TN) / (TP + TN + FP + FN) Correct predictions; misleading when the majority class dominates
Precision TP / (TP + FP) Fraction of predicted positives that are correct
Recall TP / (TP + FN) Fraction of actual positives detected
F1 2 * precision * recall / (precision + recall) Harmonic mean of precision and recall; use zero when their sum is zero
Specificity TN / (TN + FP) Fraction of actual negatives correctly rejected
ROC-AUC Area under the ROC curve Ranking performance across decision thresholds
PR-AUC/AP Area under or average precision of the precision-recall curve Usually more informative when the positive class is rare

True-positive rate is TPR = TP / (TP + FN), equal to recall. False-positive rate is FPR = FP / (FP + TN). The ROC curve plots TPR against FPR while varying the threshold; the precision-recall curve plots precision against recall.

For predicted probabilities p_i, binary labels y_i, and N observations:

  • Log loss = -(1/N) * sum(y_i * log(p_i) + (1-y_i) * log(1-p_i)) measures probability quality and strongly penalizes confident mistakes.

  • Brier score = (1/N) * sum((p_i - y_i)^2) measures squared probability error.

  • Reliability curves compare mean predicted probability with observed positive frequency within probability bins, exposing overconfidence or underconfidence.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


data = load_breast_cancer()
X = data.data
y = data.target  # 0 = benign, 1 = malignant

X_dev, X_test, y_dev, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(
        solver="liblinear",
        class_weight="balanced",
        max_iter=1000,
        random_state=42,
    )),
])

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "pr_auc": "average_precision",
}

cv_scores = cross_validate(
    pipeline,
    X_dev,
    y_dev,
    cv=cv,
    scoring=scoring,
    error_score="raise",
)

oof_probability = cross_val_predict(
    pipeline,
    X_dev,
    y_dev,
    cv=cv,
    method="predict_proba",
)[:, 1]

precision, recall, thresholds = precision_recall_curve(
    y_dev, oof_probability
)
candidate_precision = precision[:-1]
candidate_recall = recall[:-1]
denominator = candidate_precision + candidate_recall
f1_scores = np.divide(
    2 * candidate_precision * candidate_recall,
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
best_index = int(np.argmax(f1_scores))
selected_threshold = float(thresholds[best_index])

model = pipeline.fit(X_dev, y_dev)
test_probability = model.predict_proba(X_test)[:, 1]
test_predictions = (
    test_probability >= selected_threshold
).astype(int)

print("Cross-validation means:")
for metric, values in cv_scores.items():
    print(f"{metric}: {values.mean():.3f}")

print("Selected threshold:", selected_threshold)
print("Test confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("Test classification report:")
print(classification_report(
    y_test,
    test_predictions,
    target_names=["benign", "malignant"],
    zero_division=0,
))
print("Test ROC-AUC:", roc_auc_score(y_test, test_probability))
print("Test PR-AUC:", average_precision_score(y_test, test_probability))

Model Selection

  • Compare the pipeline with a DummyClassifier baseline.

  • Use GridSearchCV or RandomizedSearchCV for tuning. Use nested cross-validation when the final performance estimate must remain honest.

  • Compare models on identical folds with paired tests and report uncertainty, such as a confidence interval for the mean paired difference.

  • Select thresholds using business costs or required recall rather than maximizing F1 automatically. Threshold metrics calculated on the same OOF predictions used to choose the threshold are optimistically biased.

  • class_weight="balanced" changes training weights, not the class distribution. If using SMOTE or another resampler, place it inside the fold-specific pipeline.

Common Pitfalls

  • Leakage: Fit scaling, imputation, resampling, and feature selection only within training folds.

  • Wrong split: Random splits are inappropriate for time-dependent or grouped observations.

  • Misleading accuracy: Evaluate precision, recall, F1, ROC-AUC, and PR-AUC as appropriate.

  • Test-set tuning: Repeatedly inspecting the test set turns it into a validation set.

  • Probability misuse: Class weighting can affect calibration; inspect reliability when probabilities drive decisions.

  • Offline versus online performance: Cross-validation metrics may not reflect the business objective; validate important changes through an A/B test when possible.

42

How do I plot learning curves for an estimator in Python?

A learning curve plots model performance against the number of training examples, usually showing both training and cross-validation scores. It helps diagnose underfitting, overfitting, and whether obtaining more data is likely to help. Its main trade-off is that it provides training diagnostics rather than a final performance estimate from an untouched test set.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12), NumPy 2.x, Matplotlib 3.x

Evaluation Setup

  • Split the data into training and test sets before fitting any model. Keep the test set untouched until model and threshold selection are complete.

  • Use cross-validation on the training set to generate the validation scores:

  • KFold for general regression or balanced classification data.

  • StratifiedKFold for classification to preserve class proportions.

  • GroupKFold when observations from the same subject or entity must remain together.

  • TimeSeriesSplit for time-ordered data; never use a random split.

  • A separate validation split is useful for repeated manual tuning, but cross-validation can provide the validation signal instead.

  • Place scalers, imputers, encoders, and feature selectors inside a Pipeline so they are fitted only on each fold’s training data.

Metrics and Scoring

The scoring argument can use any compatible scikit-learn scorer. For binary classification:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN).

  • Precision = TP / (TP + FP); how many predicted positives are correct.

  • Recall = TP / (TP + FN); how many actual positives are found.

  • F1 = 2 * precision * recall / (precision + recall).

  • Specificity = TN / (TN + FP).

  • Log loss = -1/n * sum(y * log(p) + (1 - y) * log(1 - p)).

  • Brier score = 1/n * sum((p - y)^2); lower values indicate better probability calibration.

Here, TP, TN, FP, and FN are confusion-matrix counts, n is the number of samples, y is the true binary label, and p is the predicted positive probability. The ROC curve plots true-positive rate, TPR = TP / (TP + FN), against false-positive rate, FPR = FP / (FP + TN), over all thresholds; ROC-AUC is its area under the curve. The precision-recall curve plots precision against recall, and PR-AUC summarizes its area. PR-AUC is usually more informative for rare positive classes. Reliability curves compare predicted probabilities with observed positive rates.

For regression:

  • MAE = mean(abs(y - y_hat)).

  • MSE = mean((y - y_hat)^2).

  • RMSE = sqrt(MSE).

  • R² = 1 - sum_i((y_i - y_hat_i)^2) / sum_i((y_i - y_bar)^2).

  • MAPE = 100/n * sum_i(abs((y_i - y_hat_i) / y_i)); it is unstable or undefined near zero.

  • Quantile loss = max(tau * (y - y_hat), (tau - 1) * (y - y_hat)).

Here, y_hat is the prediction, y_bar is the mean true value, i indexes samples, and tau is the target quantile between zero and one. Accuracy can mislead on imbalanced data because a high majority-class rate can hide substantial minority-class errors.

Code Example

import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    learning_curve,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def plot_learning_curves(
    estimator,
    X,
    y,
    *,
    cv,
    scoring="accuracy",
    title="Learning Curve",
    n_points=8,
):
    """Plot training and cross-validation scores for an estimator."""
    train_sizes, train_scores, validation_scores = learning_curve(
        estimator,
        X,
        y,
        train_sizes=np.linspace(0.1, 1.0, n_points),
        cv=cv,
        scoring=scoring,
        n_jobs=-1,
        return_train_score=True,
        error_score="raise",
    )

    train_mean = train_scores.mean(axis=0)
    train_std = train_scores.std(axis=0)
    validation_mean = validation_scores.mean(axis=0)
    validation_std = validation_scores.std(axis=0)

    fig, ax = plt.subplots(figsize=(8, 5))
    ax.plot(train_sizes, train_mean, marker="o", label="Training score")
    ax.plot(train_sizes, validation_mean, marker="o", label="Validation score")

    ax.fill_between(
        train_sizes,
        train_mean - train_std,
        train_mean + train_std,
        alpha=0.15,
    )
    ax.fill_between(
        train_sizes,
        validation_mean - validation_std,
        validation_mean + validation_std,
        alpha=0.15,
    )

    ax.set(
        title=title,
        xlabel="Number of training examples",
        ylabel=scoring,
        xticks=train_sizes,
    )
    ax.legend()
    ax.grid(alpha=0.3)
    fig.tight_layout()
    plt.show()

    return train_sizes, train_scores, validation_scores


data = load_breast_cancer()
X, y = data.data, data.target

# The test set remains untouched during training and model selection.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    stratify=y,
    random_state=42,
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

estimator = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
])

plot_learning_curves(
    estimator,
    X_train,
    y_train,
    cv=cv,
    scoring="accuracy",
    title="Logistic Regression Learning Curve",
)

cv_results = cross_validate(
    estimator,
    X_train,
    y_train,
    cv=cv,
    scoring={
        "accuracy": "accuracy",
        "roc_auc": "roc_auc",
        "average_precision": "average_precision",
        "f1": "f1",
    },
    n_jobs=-1,
)

cv_summary = {
    name.removeprefix("test_"): round(values.mean(), 3)
    for name, values in cv_results.items()
    if name.startswith("test_")
}
print("Cross-validation mean scores:", cv_summary)

# Select the decision threshold using out-of-fold training predictions only.
oof_probabilities = cross_val_predict(
    estimator,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

precision, recall, thresholds = precision_recall_curve(y_train, oof_probabilities)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) > 0,
)
best_index = int(np.argmax(f1))
best_threshold = thresholds[best_index]

estimator.fit(X_train, y_train)
test_probabilities = estimator.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)

print(classification_report(y_test, test_predictions))
print("Confusion matrix:\n", confusion_matrix(y_test, test_predictions))
print("ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("Average precision (PR-AUC summary):",
      average_precision_score(y_test, test_probabilities))
print("Selected threshold:", best_threshold)
print("Baseline accuracy:", accuracy_score(y_test, baseline.predict(X_test)))

held_out_accuracy = accuracy_score(y_test, test_predictions)
print("Held-out accuracy:", held_out_accuracy)
# Held-out accuracy is around 0.95 (varies slightly by version).

Model Selection and Best Practices

  • Use GridSearchCV for small discrete search spaces and RandomizedSearchCV for larger or continuous parameter spaces.

  • Use nested cross-validation when estimating the performance of the entire tuning process; the inner loop selects hyperparameters and the outer loop evaluates them.

  • Compare models using identical folds and out-of-fold predictions, preferably with paired confidence intervals or a paired statistical test.

  • Establish a baseline with DummyClassifier or DummyRegressor.

  • Scale features for distance-based or gradient-based models; scaling is generally unnecessary for tree ensembles.

  • For imbalanced classification, use metrics such as PR-AUC, recall, or F1 rather than accuracy alone.

  • Set random seeds and record library versions for reproducibility.

Common Pitfalls

  • Data leakage: fitting preprocessing before splitting or outside a Pipeline leaks information across folds.

  • Test-set tuning: tuning hyperparameters, features, or thresholds on the test set produces an optimistic estimate.

  • Wrong cross-validation: random splits can leak temporal or grouped information.

  • Misreading the gap: a large training-validation gap indicates overfitting; both low scores may indicate underfitting.

  • Ignoring uncertainty: shaded standard-deviation bands reveal variability across folds.

  • Business mismatch: offline metrics may not represent user behavior; validate important product changes with A/B tests.

43

How do I simulate and evaluate model performance with Monte Carlo cross-validation?

Monte Carlo cross-validation repeatedly creates random training/holdout splits and measures performance on each holdout, producing estimates of average performance and variability. It is useful when random-split assumptions are reasonable, but overlapping holdouts make the resulting scores correlated.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Evaluation Setup

Reserve a final test set and keep it untouched until the model, hyperparameters, preprocessing, and decision threshold are frozen. Use the development set for cross-validation and model selection, fitting every learned preprocessing step inside a Pipeline so it sees only each fold’s training data.

Splitting method Main consideration Typical use
Random stratified splits Preserve class proportions IID classification
GroupKFold Keep related samples together Repeated patients or customers
TimeSeriesSplit Preserve temporal order Forecasting and time series

Monte Carlo CV and Metrics

Monte Carlo CV changes the random partition between iterations rather than using the same folds as KFold. Stratify classification splits, and use fixed seeds for reproducibility.

For binary classification, TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives; N is the number of examples.

Metric Formula or meaning
Accuracy (TP + TN) / N; can hide poor minority-class performance
Precision TP / (TP + FP)
Recall TP / (TP + FN)
F1 2TP / (2TP + FP + FN)
Specificity TN / (TN + FP)
ROC-AUC Ranking performance across thresholds, using false-positive rate and true-positive rate as the axes
PR-AUC Precision-recall performance; generally more informative for rare positives
Log loss -1/N * sum_i(y_i*ln(p_i) + (1-y_i)*ln(1-p_i))
Brier score 1/N * sum_i((p_i-y_i)^2); a proper probability-scoring rule where lower is better

Here i indexes examples, y_i is the binary outcome, and p_i is the predicted probability of the positive class. A reliability curve compares mean predicted probabilities with observed positive frequencies within probability bins.

For regression, with n observations, actual values y_i, predictions y_hat_i, and mean outcome y_bar: MAE = (1/n) * sum_i(abs(y_i-y_hat_i)); MSE = (1/n) * sum_i((y_i-y_hat_i)^2); RMSE = sqrt(MSE); and R² = 1 - sum_i((y_i-y_hat_i)^2) / sum_i((y_i-y_bar)^2). MAPE = (100/n) * sum_i(abs(y_i-y_hat_i)/abs(y_i)), but becomes unstable near zero. Quantile loss at quantile tau is (1/n) * sum_i(max(tau*(y_i-y_hat_i), (tau-1)*(y_i-y_hat_i))), where tau lies between zero and one.

Model Selection

Use GridSearchCV for small grids and RandomizedSearchCV for larger spaces. Use nested cross-validation when many configurations are compared, and compare the selected model with a DummyClassifier baseline. Compare candidates on identical splits using paired differences; estimate uncertainty with a paired test or bootstrap over independent data units rather than treating overlapping holdout scores as independent.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def choose_f1_threshold(y_true, probabilities):
    precision, recall, thresholds = precision_recall_curve(y_true, probabilities)
    denominator = precision[:-1] + recall[:-1]
    f1 = np.divide(
        2 * precision[:-1] * recall[:-1],
        denominator,
        out=np.zeros_like(denominator),
        where=denominator != 0,
    )
    return thresholds[np.argmax(f1)]


data = load_breast_cancer()
X, y = data.data, data.target

# Keep the final test set untouched.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])
baseline = Pipeline([
    ("classifier", DummyClassifier(strategy="prior", random_state=42)),
])

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Compare the model and a probability baseline on identical folds.
for estimator, name in [(model, "model"), (baseline, "dummy")]:
    results = cross_validate(
        estimator, X_dev, y_dev, cv=cv, scoring=scoring, n_jobs=-1
    )
    for metric, values in results["test_score"].items():
        print(f"{name} {metric}: {values.mean():.3f} +/- {values.std():.3f}")

# Monte Carlo CV with independent threshold-tuning and evaluation subsets.
scores = {name: [] for name in scoring}
for iteration in range(30):
    seed = 1000 + iteration
    X_train, X_rest, y_train, y_rest = train_test_split(
        X_dev, y_dev, test_size=0.30, stratify=y_dev, random_state=seed
    )
    X_threshold, X_eval, y_threshold, y_eval = train_test_split(
        X_rest, y_rest, test_size=0.50, stratify=y_rest, random_state=seed
    )

    model.fit(X_train, y_train)
    threshold = choose_f1_threshold(
        y_threshold, model.predict_proba(X_threshold)[:, 1]
    )
    probabilities = model.predict_proba(X_eval)[:, 1]
    predictions = (probabilities >= threshold).astype(int)

    scores["accuracy"].append(accuracy_score(y_eval, predictions))
    scores["precision"].append(precision_score(y_eval, predictions, zero_division=0))
    scores["recall"].append(recall_score(y_eval, predictions, zero_division=0))
    scores["f1"].append(f1_score(y_eval, predictions, zero_division=0))
    scores["roc_auc"].append(roc_auc_score(y_eval, probabilities))
    scores["average_precision"].append(average_precision_score(y_eval, probabilities))

for name, values in scores.items():
    print(f"Monte Carlo {name}: {np.mean(values):.3f} +/- {np.std(values):.3f}")

# Select the final threshold from out-of-fold development predictions.
oof_probabilities = cross_val_predict(
    model, X_dev, y_dev, cv=cv, method="predict_proba"
)[:, 1]
final_threshold = choose_f1_threshold(y_dev, oof_probabilities)

# Freeze the model and evaluate the final test set once.
model.fit(X_dev, y_dev)
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= final_threshold).astype(int)
print(classification_report(y_test, test_predictions, digits=3))
print(confusion_matrix(y_test, test_predictions))

Common Pitfalls

  • Leakage: Fit scaling, imputation, feature selection, and thresholds only on development or training data.

  • Test-set tuning: Repeated test inspection converts the test set into a validation set.

  • Invalid splits: Random splits can violate temporal, group, or dependency assumptions.

  • Misleading accuracy: Check precision, recall, F1, PR-AUC, and business costs for imbalanced data.

  • False precision: A standard deviation across overlapping Monte Carlo splits is not a confidence interval for the mean.

  • Metric mismatch: Offline CV metrics may not predict online outcomes; use A/B tests when available.

Common follow-ups concern why PR-AUC is preferred for rare positives, how thresholds affect precision and recall, when nested CV is needed, and how paired comparisons account for shared folds.

44

How do you calculate specificity and sensitivity from a confusion matrix?

Sensitivity measures the proportion of actual positive cases detected by a model, while specificity measures the proportion of actual negative cases correctly rejected. Both are calculated from the true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in a confusion matrix. Their main trade-off is that changing the classification threshold can increase sensitivity at the expense of specificity, or vice versa.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Metric Definitions

For binary classification:

Metric Formula Meaning
Accuracy (TP + TN) / (TP + TN + FP + FN) Fraction of all predictions that are correct. It can be misleading with imbalanced classes.
Sensitivity, recall, TPR TP / (TP + FN) Fraction of actual positives correctly detected.
Specificity, TNR TN / (TN + FP) Fraction of actual negatives correctly rejected.
Precision TP / (TP + FP) Fraction of predicted positives that are truly positive.
F1 score 2 * precision * recall / (precision + recall) Harmonic mean of precision and recall.
FPR FP / (FP + TN) Fraction of actual negatives incorrectly classified as positive.
ROC-AUC Area under the curve of TPR versus FPR Ranking quality across classification thresholds.

Here, TP is true positives, TN is true negatives, FP is false positives, and FN is false negatives.

Additional useful metrics include:

  • PR-AUC: Area under the precision-recall curve; usually more informative than ROC-AUC when the positive class is rare.

  • Log loss: -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)). Here, n is the number of samples, y_i is the true binary label, and p_i is the predicted probability of the positive class.

  • Brier score: (1/n) * sum((p_i - y_i)^2). It measures probability calibration and is useful with reliability curves.

Evaluation Setup

  • Split data into training, validation, and test sets. Fit model parameters on training data, select hyperparameters or thresholds using validation data, and keep the test set untouched until final evaluation.

  • Use StratifiedKFold for imbalanced classification, GroupKFold when observations from the same subject or entity are grouped, and TimeSeriesSplit when temporal ordering must be preserved.

  • Keep preprocessing inside a scikit-learn Pipeline so scalers, encoders, and imputers are fitted only on each training fold.

  • Compare models against a DummyClassifier baseline. Use GridSearchCV or RandomizedSearchCV for tuning and nested cross-validation when an honest final performance estimate is required.

  • Compare validation scores with confidence intervals or paired statistical tests rather than relying on one small score difference. Finally, confirm that offline metrics correspond to business outcomes through online A/B testing when appropriate.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def calculate_specificity_sensitivity(
    confusion_matrix: np.ndarray,
) -> dict[str, float]:
    """Calculate sensitivity and specificity from a binary confusion matrix.

    Rows represent actual classes and columns represent predicted classes.
    The matrix must use the order [negative, positive].
    """
    matrix = np.asarray(confusion_matrix)

    if matrix.shape != (2, 2):
        raise ValueError("The confusion matrix must have shape (2, 2).")
    if not np.issubdtype(matrix.dtype, np.integer):
        raise ValueError("Confusion matrix entries must be integers.")
    if np.any(matrix < 0):
        raise ValueError("Confusion matrix entries cannot be negative.")

    tn, fp = matrix[0]
    fn, tp = matrix[1]

    if tp + fn == 0:
        raise ValueError("Sensitivity is undefined because there are no actual positives.")
    if tn + fp == 0:
        raise ValueError("Specificity is undefined because there are no actual negatives.")

    return {
        "sensitivity": float(tp / (tp + fn)),
        "specificity": float(tn / (tn + fp)),
    }


# Use malignant as the positive class.
data = load_breast_cancer()
X = data.data
y = (data.target == 0).astype(np.int64)

# Reserve the test set for final evaluation.
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

# Further split training data into training and validation sets.
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    stratify=y_train_val,
    random_state=42,
)

model = Pipeline(
    [
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
    ]
)

# Each fold fits the scaler only on its own training partition.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_validate(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring={
        "accuracy": "accuracy",
        "precision": "precision",
        "recall": "recall",
        "f1": "f1",
        "roc_auc": "roc_auc",
    },
)

model.fit(X_train, y_train)

# Select a threshold on validation data using the PR curve.
validation_probabilities = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_val, validation_probabilities
)
f1_values = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) != 0,
)
best_index = int(np.argmax(f1_values))
selected_threshold = float(thresholds[best_index])

# Evaluate the selected model and threshold only once on the test set.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= selected_threshold).astype(np.int64)
test_matrix = confusion_matrix(y_test, test_predictions, labels=[0, 1])
rates = calculate_specificity_sensitivity(test_matrix)

print("Cross-validation F1:", cv_scores["test_f1"])
print("Selected threshold:", selected_threshold)
print("Confusion matrix:\n", test_matrix)
print("Sensitivity:", rates["sensitivity"])
print("Specificity:", rates["specificity"])
print("ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("PR-AUC:", average_precision_score(y_test, test_probabilities))
print(
    classification_report(
        y_test,
        test_predictions,
        target_names=["benign", "malignant"],
        zero_division=0,
    )
)

Common Pitfalls

  • Reversing matrix orientation: Treat rows as actual classes and columns as predictions; otherwise TP, TN, FP, and FN are assigned incorrectly.

  • Ignoring the positive class: Always state which class is positive, since sensitivity and precision are class-dependent.

  • Using accuracy on imbalanced data: A model predicting only the majority class can appear highly accurate while failing to detect positives.

  • Leakage: Do not fit scalers, encoders, feature selection, or models on validation or test data.

  • Tuning on the test set: Selecting thresholds or hyperparameters using test results turns the test set into another validation set.

  • Ignoring undefined cases: Sensitivity is undefined without actual positives, and specificity is undefined without actual negatives.

45

How do I compare two models with a paired t-test?

A paired t-test determines whether the mean performance difference between two models is statistically significant. It is appropriate when both models are evaluated on the same observations, because pairing removes much of the variation caused by difficult or easy examples. The main trade-off is that it assumes the paired differences are approximately independent and normally distributed—assumptions that can be imperfect for classification accuracy because each observation contributes a binary correct/incorrect value.

Read full answer

Applies toPython 3.12, scikit-learn 1.5+, and SciPy 1.13+

Evaluation Setup

A reliable evaluation separates three purposes:

  • Training data: Fits the model parameters.

  • Validation data: Selects hyperparameters or compares candidate models.

  • Test data: Estimates final generalization performance and remains untouched until evaluation ends.

Cross-validation is useful when data is limited. KFold supports general resampling, StratifiedKFold preserves class proportions for classification, GroupKFold keeps related observations together, and TimeSeriesSplit respects temporal order. Every preprocessing step must be fitted only on the training portion of each fold.

The example below uses StratifiedKFold to generate out-of-fold predictions for two models. It then performs a paired t-test over their per-example correctness scores before evaluating either model on the final test set.

Paired t-Test

For paired observations, calculate the difference:

d_i = correct_i(model_A) - correct_i(model_B)

The test statistic is:

t = mean(d) / (standard_deviation(d) / sqrt(n))

Here:

  • d_i is the performance difference for observation i.

  • mean(d) is the average difference between the models.

  • standard_deviation(d) measures variation in the paired differences.

  • n is the number of evaluated observations.

  • Degrees of freedom equal n - 1.

A small p-value indicates that the observed difference is unlikely under the null hypothesis that the models have equal mean performance. It does not show that the difference is practically or commercially important.

Code Example

import numpy as np
from scipy.stats import t, ttest_rel
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

RANDOM_STATE = 42
ALPHA = 0.05

# Bundled binary-classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the final test set separate from all model comparison work.
X_dev, X_test, y_dev, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=RANDOM_STATE,
)

models = {
    "logistic_regression": Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=2000)),
    ]),
    "random_forest": Pipeline([
        ("classifier", RandomForestClassifier(
            n_estimators=300,
            random_state=RANDOM_STATE,
        )),
    ]),
}

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE,
)

# Every observation receives an out-of-fold prediction.
correctness = {}
for name, model in models.items():
    predictions = cross_val_predict(
        model,
        X_dev,
        y_dev,
        cv=cv,
        method="predict",
        n_jobs=-1,
    )
    correctness[name] = (predictions == y_dev).astype(float)

names = list(models)
score_a = correctness[names[0]]
score_b = correctness[names[1]]

result = ttest_rel(score_a, score_b)
mean_a = score_a.mean()
mean_b = score_b.mean()
mean_difference = mean_a - mean_b

standard_error = np.std(score_a - score_b, ddof=1) / np.sqrt(len(y_dev))
critical_value = t.ppf(1 - ALPHA / 2, df=len(y_dev) - 1)
confidence_interval = (
    mean_difference - critical_value * standard_error,
    mean_difference + critical_value * standard_error,
)

print("Cross-validated results")
print(f"{names[0]} accuracy: {mean_a:.4f}")
print(f"{names[1]} accuracy: {mean_b:.4f}")
print(f"Mean paired difference: {mean_difference:.4f}")
print(f"95% confidence interval: {confidence_interval}")
print(f"T-statistic: {result.statistic:.4f}")
print(f"P-value: {result.pvalue:.6f}")
print(f"Statistically significant at alpha={ALPHA}: {result.pvalue < ALPHA}")

# Final evaluation happens only after the model comparison.
for name, model in models.items():
    model.fit(X_dev, y_dev)
    test_predictions = model.predict(X_test)
    test_probabilities = model.predict_proba(X_test)[:, 1]

    print(f"\nFinal test results: {name}")
    print(classification_report(
        y_test,
        test_predictions,
        target_names=data.target_names,
        zero_division=0,
    ))
    print("Confusion matrix:")
    print(confusion_matrix(y_test, test_predictions))
    print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.4f}")
    print(
        "PR-AUC: "
        f"{average_precision_score(y_test, test_probabilities):.4f}"
    )

Metrics and Interpretation

  • Accuracy = (TP + TN) / n; it can be misleading with imbalanced classes.

  • Precision = TP / (TP + FP); it measures positive-prediction correctness.

  • Recall = TP / (TP + FN); it measures detected positives.

  • F1 = 2 * precision * recall / (precision + recall).

  • Specificity = TN / (TN + FP).

  • ROC-AUC measures ranking across thresholds, while PR-AUC is often more informative when the positive class is rare.

Report the effect size and confidence interval alongside the p-value. A statistically significant difference of 0.1% accuracy may have no practical value.

Common Pitfalls

  • Unpaired testing: Comparing unrelated samples discards the benefit of evaluating both models on identical cases.

  • Leakage: Fitting a scaler or other transformer before cross-validation exposes validation information to training.

  • Test-set tuning: Repeatedly adjusting models after examining test results invalidates the final estimate.

  • Ignoring dependence: Fold scores from overlapping CV splits are not independent; do not treat them as an ordinary independent-sample t-test.

  • Overrelying on normality: Binary correctness differences can make the t-test approximation weak. For classifiers, consider paired bootstrap confidence intervals or McNemar’s test.

  • Small samples: A p-value is unstable with few observations; report raw disagreement counts and effect sizes.

  • Metric mismatch: Offline significance does not establish an online business improvement; validate important changes through an A/B test.

46

How do you evaluate fraud detection with imbalanced classes?

I would evaluate a fraud detection model primarily as a ranking and threshold-selection problem, not with accuracy alone. The evaluation must balance missed fraud against false alarms, report precision-recall performance, and verify that any chosen threshold generalizes to unseen transactions. The main trade-off is detecting more fraud while keeping the false-positive burden operationally manageable.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Training set: Fit model parameters, preprocessing, feature selection, and calibration.

  • Validation set: Select hyperparameters and the decision threshold.

  • Test set: Keep untouched until the final evaluation.

  • Cross-validation: Use StratifiedKFold for ordinary classification, GroupKFold when customers or devices repeat, and TimeSeriesSplit for chronological transaction data.

  • Prevent leakage by ensuring every feature exists at scoring time and excluding duplicated entities, post-outcome information, or aggregates that include validation and test records.

For temporal fraud data, a random split usually overestimates performance because it allows the model to learn from future transactions.

Metrics for Imbalanced Fraud

Let TP, FP, TN, and FN mean true positives, false positives, true negatives, and false negatives.

  • Accuracy = (TP + TN) / (TP + TN + FP + FN). It can be misleading because always predicting “not fraud” may achieve very high accuracy.

  • Precision = TP / (TP + FP): the proportion of flagged cases that are genuinely fraudulent.

  • Recall = TP / (TP + FN): the proportion of fraud detected.

  • F1 = 2 * precision * recall / (precision + recall): balances precision and recall.

  • Specificity = TN / (TN + FP): measures correct rejection of legitimate transactions.

  • ROC curve: plots true-positive rate against false-positive rate across thresholds. ROC-AUC measures ranking quality, but can appear optimistic for extremely rare fraud.

  • Precision-recall curve: plots precision against recall. PR-AUC, commonly computed with average_precision_score, is generally more informative when fraud is rare.

  • Log loss = -1/n * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i)), where y_i is the true label, p_i is the predicted fraud probability, and n is the number of observations.

  • Brier score = 1/n * sum((y_i - p_i)^2): lower is better. Reliability curves check whether predicted probabilities correspond to observed fraud frequencies.

Class weighting may improve ranking while making probabilities poorly calibrated. Evaluate calibration separately when probability quality matters.

Code Example

The bundled dataset demonstrates the evaluation API; real fraud data usually has rarer positives and should be split according to transaction time or customer groups.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    brier_score_loss,
    classification_report,
    confusion_matrix,
    log_loss,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Binary classification data; class 1 is treated as the positive event.
data = load_breast_cancer()
X = data.data
y = data.target

# Preserve the test set for the final evaluation.
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    stratify=y_train_val,
    random_state=42,
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(
        class_weight="balanced",
        max_iter=1000,
        random_state=42,
    )),
])

# Preprocessing is fitted inside each training fold, avoiding leakage.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    pipeline,
    X_train,
    y_train,
    cv=cv,
    scoring={
        "roc_auc": "roc_auc",
        "pr_auc": "average_precision",
        "f1": "f1",
    },
    n_jobs=-1,
)
print("Cross-validation means:", {
    name: round(float(scores.mean()), 3)
    for name, scores in cv_results.items()
})

pipeline.fit(X_train, y_train)
val_probability = pipeline.predict_proba(X_val)[:, 1]

# Select the threshold using validation predictions only.
precision, recall, thresholds = precision_recall_curve(y_val, val_probability)
f1 = 2 * precision * recall / np.maximum(precision + recall, 1e-12)
best_index = int(np.argmax(f1[:-1]))
selected_threshold = float(thresholds[best_index])

# Evaluate once on the untouched test set.
test_probability = pipeline.predict_proba(X_test)[:, 1]
test_prediction = (test_probability >= selected_threshold).astype(int)

print("Selected threshold:", selected_threshold)
print(classification_report(y_test, test_prediction, zero_division=0))
print("Confusion matrix:\n", confusion_matrix(y_test, test_prediction))
print("ROC-AUC:", roc_auc_score(y_test, test_probability))
print("PR-AUC:", average_precision_score(y_test, test_probability))
print("Log loss:", log_loss(y_test, test_probability))
print("Brier score:", brier_score_loss(y_test, test_probability))

Model Selection and Thresholding

  • Compare candidate models against a DummyClassifier(strategy="most_frequent") baseline.

  • Use GridSearchCV or RandomizedSearchCV with ranking metrics such as PR-AUC, and constrain recall according to operational requirements.

  • Select thresholds from out-of-fold or validation predictions. Optimize business cost when false positives and false negatives have different costs; maximizing F1 is only a starting point.

  • Use nested cross-validation when an honest performance estimate is needed for model selection.

  • Report confidence intervals using bootstrap resampling. For paired predictions on the same test examples, McNemar’s test can compare classification errors.

Common Pitfalls

  • Tuning hyperparameters or thresholds on the test set.

  • Using random splits for temporal, customer-level, or device-level data.

  • Reporting accuracy without a class-balanced baseline.

  • Ignoring calibration when scores are used as probability estimates.

  • Selecting a threshold on training predictions, causing overfitting.

  • Assuming offline metrics represent production impact; validate important changes through an A/B test and monitor drift, recall, precision, and operational review volume over time.

47

How to set up an A/B test for a machine learning model

An A/B test is a randomized controlled experiment that compares two versions of a machine learning model by serving them to different user segments and measuring predefined business metrics. It solves the problem of determining whether a model change truly improves real-world performance beyond what offline evaluation can guarantee. The main trade-off is the cost and complexity of running a live experiment, including the risk of exposing an inferior model to users if not properly mitigated.

Read full answer

Applies toA/B testing methodology (Python 3.12 with NumPy 2.x, SciPy 1.13+, statsmodels 0.14+)

Why A/B Test a Machine Learning Model?

Offline evaluation (cross-validation, holdout tests) provides unbiased estimates of model performance on historical data, but it cannot capture concept drift, changes in user behavior, or misalignment between offline metrics and business goals. An A/B test measures the actual impact on a live metric (e.g., conversion rate, revenue per user) by exposing real users to the models under controlled conditions.

Designing the Experiment

  • Primary metric: Choose a single, sensitive business metric (e.g., click-through rate, daily active users) that directly reflects the goal of the change.

  • Randomization unit: Randomly assign independent users (or sessions, or items) to control (A) or treatment (B). Use a consistent hash on a user ID to ensure the same user always sees the same model.

  • Traffic split: For a safe new model, start with a 50/50 split; to limit risk, use a smaller treatment group (e.g., 10% B, 90% A) and increase gradually if results are positive.

  • Guardrail metrics: Monitor secondary metrics (error rate, latency, crash rate) so the experiment can be stopped early if the new model causes regressions.

  • Sample size and duration: Perform a power analysis to determine the minimum sample size needed to detect a practically meaningful effect with 80% power and 95% confidence. Run the experiment for at least one full business cycle (e.g., one week) to account for day-of-week effects.

  • Pre-registration: Document the hypothesis, primary metric, split ratio, and analysis plan before looking at the results to avoid peeking and p-hacking.

Statistical Analysis

After the experiment, compare the primary metric between the two groups. For binary metrics (e.g., conversion), use a two-proportion z-test; for continuous metrics (e.g., revenue), use a Welch’s t-test or bootstrap confidence intervals. Report the p-value, confidence interval, and the absolute/relative lift. Statistical significance (p < 0.05) indicates the observed difference is unlikely due to chance, but also evaluate practical significance: is the lift large enough to justify the deployment cost and potential risk?

Code Example

The following Python script simulates a simple A/B test with a binary conversion metric.

import numpy as np
from scipy import stats
import statsmodels.api as sm

np.random.seed(42)
n = 10000
p_a, p_b = 0.10, 0.12
group = np.random.choice(['A', 'B'], n, p=[0.5, 0.5])
conv = np.where(group == 'A',
                np.random.binomial(1, p_a, n),
                np.random.binomial(1, p_b, n))

# Conversion rates
cr_a = conv[group == 'A'].mean()
cr_b = conv[group == 'B'].mean()
print(f"Control: {cr_a:.4f}, Treatment: {cr_b:.4f}")

# Z-test for proportions
count = np.array([conv[group == 'A'].sum(), conv[group == 'B'].sum()])
nobs = np.array([(group == 'A').sum(), (group == 'B').sum()])
zstat, pval = sm.stats.proportions_ztest(count, nobs)
print(f"Z={zstat:.3f}, p={pval:.4f}")

# Confidence interval for the difference
diff = cr_b - cr_a
se = np.sqrt(cr_a*(1-cr_a)/nobs[0] + cr_b*(1-cr_b)/nobs[1])
z = stats.norm.ppf(0.975)
ci = (diff - z*se, diff + z*se)
print(f"Lift: {diff:.4f}, 95% CI: [{ci[0]:.4f}, {ci[1]:.4f}]")

Common Pitfalls

  • Peeking at results: Continuously testing for significance before the pre-registered end time inflates the false positive rate. Wait until the full sample is collected.

  • Multiple testing: Running many A/B tests simultaneously without correction (e.g., Bonferroni) leads to false positives.

  • Interference: One group’s behavior affecting the other (e.g., social network effects) violates independence. Use cluster randomization or switchback designs if interference is suspected.

  • Short duration: Stopping after a few days may capture novelty effect rather than long-term impact. Run for at least one full week.

  • Metric misalignment: Optimizing a proxy metric (e.g., clicks) that does not ultimately drive business value (e.g., customer lifetime value).

  • Data leakage: Ensure the model being tested does not use information from the experiment period during training; otherwise the A/B test is invalid.

48

How should you evaluate a self-driving computer vision model?

Evaluating a self-driving vision model requires multi-level evaluation covering perception accuracy, robustness, safety, and closed-loop behavior. It reveals whether predictions remain reliable under realistic conditions; the main trade-off is reproducible offline coverage versus expensive but realistic simulation and road testing.

Read full answer

Applies tomachine learning concepts; examples use scikit-learn 1.5+ and NumPy 2.x with Python 3.12

Evaluation Setup

  • Split data into training, validation, and test sets by route, vehicle, geography, or time—not by random nearby frames—to prevent leakage. Fit preprocessing on training folds only using Pipeline or ColumnTransformer, and keep the test set untouched until the model and threshold are frozen.

  • Use StratifiedKFold to preserve class proportions, GroupKFold for correlated groups, and TimeSeriesSplit for chronological data. Tune with GridSearchCV or RandomizedSearchCV; use nested cross-validation for the final estimate.

  • Compare against a DummyClassifier baseline. Report group-aware confidence intervals or paired comparisons; ordinary frame-level resampling does not account for correlated driving scenes.

Metrics by Task

Let TP, TN, FP, and FN mean true positives, true negatives, false positives, and false negatives.

Metric Formula and meaning
Accuracy (TP + TN) / (TP + TN + FP + FN); can mislead with imbalanced classes.
Precision TP / (TP + FP); proportion of positive predictions that are correct.
Recall TP / (TP + FN); proportion of actual positives detected.
F1 2 * precision * recall / (precision + recall); harmonic mean of precision and recall.
Specificity TN / (TN + FP); proportion of actual negatives correctly rejected.

A ROC curve plots recall against false-positive rate across thresholds, while ROC-AUC summarizes ranking performance. A precision-recall curve plots precision against recall and is usually more informative for rare events. Log loss is -(1/N) * sum(y*log(p) + (1-y)*log(1-p)), where N is the sample count, y is the binary true label, and p is the predicted positive probability.

Autonomous-driving task Important metrics
Object detection Precision, recall, average precision, and mean average precision across IoU thresholds; IoU is intersection area divided by union area.
Semantic segmentation Mean IoU and Dice score, including rare safety-critical classes.
Depth or trajectory Absolute and relative depth error; average and final displacement error.
Tracking ID switches, track fragmentation, recall, and association errors.
Calibration Brier score = (1/N) * sum((p-y)^2) using the symbols above, plus reliability curves.

For regression, MAE is (1/n) * sum(|y_i-y_hat_i|), MSE is (1/n) * sum((y_i-y_hat_i)^2), and RMSE is the square root of MSE. Here, n is the sample count, y_i is the true value, and y_hat_i is the prediction. R² is 1 - sum((y_i-y_hat_i)^2) / sum((y_i-y_bar)^2), where y_bar is the mean target. MAPE is 100/n * sum(|y_i-y_hat_i|/|y_i|), but is undefined when targets are zero and unstable near zero. Quantile (pinball) loss is (1/n) * sum(max(q*(y_i-y_hat_i), 0) + max((1-q)*(y_hat_i-y_i), 0)), where q is the target quantile.

Closed-Loop and Operational Evaluation

Stress-test lighting, weather, road type, obstruction, sensor degradation, and geographic shifts. Measure detection latency, end-to-end latency, memory use, CPU performance, and accelerator throughput on deployment hardware.

In simulation or a vehicle, evaluate collision rate, lane departure, traffic-rule violations, emergency interventions, discomfort, and time to collision. Validate offline metrics against road-test outcomes and, when appropriate, controlled A/B tests because online safety and user-experience metrics may differ.

Code Example

This example demonstrates leakage-safe pipeline evaluation, nested cross-validation, and validation-set threshold selection.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    GridSearchCV,
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25,
    stratify=y_train_val, random_state=42
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=2000, random_state=42)),
])

inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=43)
search = GridSearchCV(
    pipeline,
    param_grid={"model__C": [0.01, 0.1, 1.0, 10.0]},
    scoring="average_precision",
    cv=inner_cv,
)

# Each outer fold performs its own inner hyperparameter search.
scores = cross_validate(
    search, X_train, y_train, cv=outer_cv,
    scoring={
        "accuracy": "accuracy",
        "precision": "precision",
        "recall": "recall",
        "roc_auc": "roc_auc",
        "pr_auc": "average_precision",
    },
)

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)
print("Dummy baseline accuracy:", round(baseline.score(X_train, y_train), 3))

search.fit(X_train, y_train)
val_prob = search.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_prob)
eligible = np.flatnonzero(precision[:-1] >= 0.90)
if eligible.size == 0:
    raise RuntimeError("No validation threshold satisfies precision >= 0.90")
threshold = float(thresholds[eligible[-1]])

test_prob = search.predict_proba(X_test)[:, 1]
test_pred = (test_prob >= threshold).astype(int)

print("Cross-validation means:",
      {name: round(values.mean(), 3) for name, values in scores.items()
       if name.startswith("test_")})
print("Selected threshold:", round(threshold, 3))
print(classification_report(y_test, test_pred, zero_division=0))
print("Confusion matrix:\n", confusion_matrix(y_test, test_pred))
print("ROC-AUC:", round(roc_auc_score(y_test, test_prob), 3))
print("PR-AUC:", round(average_precision_score(y_test, test_prob), 3))

Common Pitfalls

  • Data leakage: duplicated frames, future information, or preprocessing fitted on all data inflates results.

  • Tuning on the test set: repeatedly adjusting thresholds or models against test performance invalidates the estimate.

  • Wrong metric: accuracy can hide dangerous misses; rare hazards often require PR-AUC, recall, or worst-case analysis.

  • Invalid splitting: random frame splits leak route, location, and temporal information.

  • Unrepresentative data: clean daytime footage overstates real-world robustness.

  • Offline-only conclusions: acceptable dataset scores do not guarantee safe closed-loop behavior.

Interviewers commonly ask how to select operating thresholds, define safety-critical failures, prevent correlated-frame leakage, and determine whether an improvement is statistically and operationally meaningful.

49

How do you continuously evaluate an online learning system?

Continuous evaluation measures an online model as new traffic arrives, comparing live predictions with delayed labels, baselines, and operational targets. Prequential evaluation predicts each observation before using its label to update the model, avoiding evaluation on examples the model has already learned. The main trade-off is rapid adaptation versus stability: frequent updates can respond quickly to drift but also amplify noisy or biased feedback.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Establish an initial training set, a validation set for hyperparameters and thresholds, and a final offline test set that remains untouched until evaluation finishes.

  • Use KFold for independent observations, StratifiedKFold for classification, GroupKFold for grouped observations, and TimeSeriesSplit for temporal data. Do not randomly split time-ordered or grouped data.

  • Compare candidates with GridSearchCV or RandomizedSearchCV; use nested cross-validation when tuning and estimating performance together. Include a DummyClassifier baseline and report confidence intervals or paired statistical tests.

  • In the live stream, predict first, score when the label arrives, and then update. If labels are delayed, retain the original prediction timestamp and score it when the label becomes available. Do not tune decisions on the same stream used for final reporting.

Metrics

Choose metrics according to the error costs rather than using accuracy alone:

Metric Formula Meaning
Accuracy (TP + TN) / N Correct predictions; misleading with class imbalance
Precision TP / (TP + FP) Correctness of positive predictions
Recall TP / (TP + FN) Fraction of positives detected
Specificity TN / (TN + FP) Fraction of negatives correctly rejected
F1 2 * P * R / (P + R) Harmonic mean of precision and recall
ROC-AUC Area under the TPR-versus-FPR curve Ranking quality across thresholds
PR-AUC Area under the precision-recall curve Usually more informative for rare positives

TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives; N is the number of observations; P and R are precision and recall. The ROC curve plots recall against false-positive rate, while the precision-recall curve plots precision against recall.

For binary probabilities, log loss is -1/N * sum(y_i * ln(p_i) + (1-y_i) * ln(1-p_i)), where y_i is the true binary label and p_i is its predicted probability. The Brier score is 1/N * sum((p_i-y_i)^2); reliability curves compare predicted probabilities with observed positive frequencies.

For regression, let y_i be the actual value, q_i the prediction, and mean(y) the target mean:

  • MAE = 1/N * sum(abs(y_i-q_i)).

  • MSE = 1/N * sum((y_i-q_i)^2).

  • RMSE = sqrt(MSE).

  • R^2 = 1 - sum((y_i-q_i)^2) / sum((y_i-mean(y))^2).

  • MAPE = 100/N * sum(abs(y_i-q_i) / abs(y_i)); it is unstable or undefined near zero.

  • Quantile loss = 1/N * sum(max(tau*e_i, (tau-1)*e_i)), where e_i = y_i-q_i and tau is the target quantile between 0 and 1.

Continuous Monitoring

  • Monitor input and concept drift, missingness, feature ranges, label distributions, calibration, and metrics over rolling windows with confidence intervals.

  • Track latency, throughput, memory, compute cost, uptime, and business outcomes. Use shadow deployment before promotion and an A/B test when changes affect users.

  • Define alert thresholds and rollback conditions. Recalibrate probabilities, retrain from versioned data, and revalidate decision thresholds when justified.

  • Fit imputers, scalers, encoders, and feature selection on initial training data only. Scaling is required for k-NN, SVM, PCA, and distance-based models but generally unnecessary for trees. Monitor missing-value indicators, class imbalance, and overfitting; use CPU or GPUs according to measured workload rather than assumption.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import (
    average_precision_score, classification_report, confusion_matrix,
    log_loss, precision_recall_curve, roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold, cross_validate, train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

X_cv, X_holdout, y_cv, y_holdout = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_holdout, y_holdout, test_size=0.5, random_state=42, stratify=y_holdout
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", SGDClassifier(loss="log_loss", random_state=42)),
])

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_validate(
    pipeline, X_cv, y_cv, scoring=scoring, cv=cv
)
pipeline.fit(X_cv, y_cv)

# Select the threshold using validation data only.
val_probability = pipeline.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_val, val_probability
)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(thresholds),
    where=(precision[:-1] + recall[:-1]) != 0,
)
threshold = float(thresholds[np.argmax(f1)])

scaler = pipeline.named_steps["scale"]
estimator = pipeline.named_steps["model"]
all_labels, all_probabilities = [], []

# Prequential evaluation: predict, then update with each revealed label.
for start in range(0, len(X_test), 50):
    X_batch, y_batch = X_test[start:start + 50], y_test[start:start + 50]
    for i in range(len(y_batch)):
        X_row = scaler.transform(X_batch[i:i + 1])
        probability = estimator.predict_proba(X_row)[0, 1]
        all_probabilities.append(probability)
        all_labels.append(y_batch[i])
        estimator.partial_fit(X_row, y_batch[i:i + 1])

all_probabilities = np.asarray(all_probabilities)
predictions = (all_probabilities >= threshold).astype(int)

print("Selected threshold:", threshold)
print("CV mean ROC-AUC:", np.mean(cv_scores["test_roc_auc"]))
print(classification_report(all_labels, predictions))
print("Confusion matrix:\n", confusion_matrix(all_labels, predictions))
print("ROC-AUC:", roc_auc_score(all_labels, all_probabilities))
print("PR-AUC:", average_precision_score(all_labels, all_probabilities))
print("Log loss:", log_loss(all_labels, all_probabilities, labels=[0, 1]))

Common Pitfalls

  • Leakage: preprocessing or threshold selection must not use test data.

  • Wrong splitting: random splits invalidate temporal and grouped evaluation.

  • Test-set tuning: repeatedly changing decisions against reported results invalidates the estimate.

  • Wrong metric: accuracy can hide minority-class failures; use precision, recall, F1, PR-AUC, and calibration when appropriate.

  • Delayed-label neglect: score the original prediction rather than a later prediction.

  • Silent degradation: monitor drift and operational metrics even before enough labels arrive.

  • Unstable updates: version data and models, control learning rates, and retain rollback procedures.

50

How do precision and recall measure churn model business impact?

Precision and recall translate a churn model’s targeting quality into business impact: precision measures how many selected customers truly churn, while recall measures how many churners the model finds. False positives consume offer costs, whereas false negatives leave revenue at risk. The trade-off is contact efficiency versus coverage of preventable churn.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Evaluation Setup

  • Create separate training, validation, and test sets. For production churn prediction, prefer a chronological split because training outcomes must precede validation and test outcomes.

  • Use KFold for ordinary cross-validation when preserving class proportions is unnecessary; use StratifiedKFold for imbalanced classification, GroupKFold for repeated records from the same customer, and TimeSeriesSplit for ordered observations.

  • Fit preprocessing and models only on training folds. A scikit-learn Pipeline prevents preprocessing leakage.

  • Use validation data for model and threshold selection, and keep the test set untouched until final evaluation.

  • Compare against a DummyClassifier baseline. Use nested cross-validation when an honest estimate must include hyperparameter and threshold selection.

Metrics and Business Value

Let TP, FP, FN, and TN denote true positives, false positives, false negatives, and true negatives.

Metric Formula Interpretation
Accuracy (TP + TN) / (TP + TN + FP + FN) Overall correctness; potentially misleading when churn is rare.
Precision TP / (TP + FP) How many targeted customers actually churn.
Recall TP / (TP + FN) How many actual churners are identified.
F1 2 * P * R / (P + R) Harmonic balance of precision and recall; P and R are precision and recall.
Specificity TN / (TN + FP) Correctly identifying non-churners, reducing unnecessary contacts.
ROC curve (FPR, TPR) across thresholds FPR = FP / (FP + TN) and TPR = TP / (TP + FN).
ROC-AUC P(score_positive > score_negative) Ranking probability across thresholds; tied scores contribute one half. It ignores prevalence and can look high despite poor precision.
PR curve (recall, precision) across thresholds Shows the precision available at each recall level.
PR-AUC/AP Area under the PR curve Usually more informative for rare positives; scikit-learn computes average precision with average_precision_score.
Log loss -(1/n) * sum(y_i * log(p_i) + (1-y_i) * log(1-p_i)) Penalizes incorrect probabilities; n is the sample count, y_i the true label, p_i the predicted probability, and log the natural logarithm.
Brier score (1/n) * sum((p_i - y_i)^2) Mean squared probability error; lower is better.

A simplified policy value is:

net_value = TP * r * V - (TP + FP) * c - FN * V

Here r is the expected retention rate among targeted true positives, V is customer value, and c is intervention cost per contact. The confusion matrix alone cannot establish r; causal uplift should come from a validated intervention or experiment.

Threshold and Model Selection

Use precision_recall_curve on validation predictions to select a threshold, such as the most precise threshold that still achieves 80% recall. Compare candidates with cross-validation, confidence intervals, or paired fold-level results. Brier score and reliability curves assess whether probabilities are calibrated enough for expected-value calculations.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    make_scorer,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# 60% train, 20% validation, 20% untouched test
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25,
    stratify=y_train_val, random_state=42,
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
])

scoring = {
    "precision": make_scorer(precision_score, zero_division=0),
    "recall": make_scorer(recall_score, zero_division=0),
    "pr_auc": "average_precision",
    "roc_auc": "roc_auc",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

estimators = {
    "model": model,
    "class_prior_baseline": Pipeline([
        ("classifier", DummyClassifier(strategy="prior")),
    ]),
}

for name, estimator in estimators.items():
    scores = cross_validate(
        estimator, X_train, y_train, cv=cv, scoring=scoring
    )
    print(name, {
        metric: f"{scores[f'test_{metric}'].mean():.3f}"
        for metric in scoring
    })

model.fit(X_train, y_train)

# Select the threshold using validation data only
validation_probabilities = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_val, validation_probabilities
)

minimum_recall = 0.80
eligible = np.flatnonzero(recall[:-1] >= minimum_recall)
if eligible.size == 0:
    raise RuntimeError("No threshold satisfies the minimum recall.")

best_index = eligible[np.argmax(precision[:-1][eligible])]
threshold = thresholds[best_index]

# Evaluate the locked model and threshold on the test set
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= threshold).astype(int)

print("Selected threshold:", threshold)
print(classification_report(
    y_test, test_predictions,
    target_names=["retained", "churn"],
    zero_division=0,
))
print("Confusion matrix:", confusion_matrix(
    y_test, test_predictions, labels=[0, 1]
).tolist())
print("ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("PR-AUC:", average_precision_score(y_test, test_probabilities))

# Illustrative business assumptions; replace with validated estimates.
customer_value = 100.0
intervention_cost = 10.0
expected_retention_rate = 0.5
tn, fp, fn, tp = confusion_matrix(
    y_test, test_predictions, labels=[0, 1]
).ravel()
net_value = (
    tp * expected_retention_rate * customer_value
    - (tp + fp) * intervention_cost
    - fn * customer_value
)
print("Net value on test set:", net_value)

Common Pitfalls

  • Tuning thresholds or hyperparameters on the test set produces optimistic results.

  • Accuracy can hide poor recall on a rare churn class; inspect precision, recall, and PR-AUC.

  • Randomly splitting customer histories can leak information; split by customer or time.

  • Confusion-matrix metrics do not establish causal treatment effect or customer value.

  • Validate the targeting and incentive policy through an A/B test where feasible, and reconcile offline metrics with online revenue or retention outcomes.

51

Why is model explainability important in model evaluation?

Model explainability refers to the techniques and practices that make the decision-making process of a machine learning model understandable to humans. In model evaluation, it complements quantitative metrics by revealing whether the model has learned meaningful patterns or is relying on spurious correlations, thereby ensuring trust, fairness, and regulatory compliance. The main trade-off is that explainability often requires additional computational steps and may only provide approximations of the model’s true behavior.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12 with scikit-learn 1.5+)

The Role of Explainability in Model Evaluation

  • Trust and transparency: Stakeholders need to understand why a model made a certain prediction, especially in high-stakes domains like healthcare or finance.

  • Debugging and error analysis: Explanations can reveal systematic errors, such as the model relying on a feature that is unavailable at prediction time.

  • Fairness and bias detection: By examining feature importances or individual explanations, we can identify if the model is using sensitive attributes (e.g., race, gender) directly or indirectly.

  • Model selection: When two models have similar performance, we may prefer the more interpretable one. Explainability helps justify the choice.

  • Regulatory compliance: Laws like the GDPR in the EU require a “right to explanation” for automated decisions.

Key Explainability Techniques

  • Intrinsic interpretable models: Linear regression, logistic regression, decision trees (with limited depth) are transparent by design.

  • Post-hoc methods for black-box models:

  • Feature importance: Permutation importance, mean decrease in impurity (for tree ensembles).

  • SHAP (SHapley Additive exPlanations): A game-theoretic approach that assigns each feature an importance value for a given prediction.

  • LIME (Local Interpretable Model-agnostic Explanations): Approximates the model locally with a simple interpretable model.

  • Partial dependence plots (PDP): Show the marginal effect of a feature on the predicted outcome.

  • Accumulated local effects (ALE): An alternative to PDP that reduces bias.

  • Attention weights: In transformer models, attention matrices can serve as a (imperfect) explanation.

Integrating Explainability into the Evaluation Workflow

  • Apply explainability methods on the validation/test set, not the training set, to avoid overfitting the explanations.

  • Use domain experts to validate whether the identified important features make sense.

  • Check for data leakage: if a feature that should not be available (e.g., a future event) has high importance, it signals a pipeline error.

  • Compare explanations across different models: a model that uses the same features as a simpler model is more trustworthy.

  • For time series or grouped data, ensure explanations are computed within each fold to respect temporal or group structure.

Trade-offs and Limitations

  • Accuracy vs. interpretability: Complex models (e.g., deep neural networks, gradient boosting) often outperform interpretable ones, but at the cost of understanding.

  • Fidelity: Post-hoc methods approximate the model; a good explanation does not guarantee the model is correct.

  • Computational cost: SHAP and permutation importance can be slow on large datasets.

  • Subjectivity: Different explanation methods may give different rankings; there is no single ground truth.

  • Misleading explanations: If the model is inherently flawed (e.g., trained on biased data), explanations will reflect that bias.

Common Pitfalls

  • Leakage in explanations: Using the entire dataset to compute importance before splitting can lead to optimistic and misleading explanations.

  • Over-reliance on a single method: Always cross-check with another technique (e.g., SHAP and permutation importance).

  • Ignoring domain knowledge: A feature may be statistically important but clinically irrelevant.

  • Using explanations to justify a bad model: Explainability does not fix a model that has low accuracy or high bias.

  • Neglecting calibration: A well-explained model may still be poorly calibrated; check calibration curves.

  • Interview tips: Be ready to discuss a time you used explainability to catch data leakage or bias. Know the difference between global and local explanations, and between feature importance and feature contributions. Prepare to defend the choice of an interpretable model over a slightly more accurate black box in a regulated industry.

52

How does transfer learning change model evaluation?

Transfer learning changes model evaluation because the pretrained model has already learned from another dataset, so evaluation must consider both downstream performance and independence from the pretraining data. Its main trade-off is better sample efficiency and initialization versus risks such as negative transfer, source-domain dependence, and more complex validation.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12 with scikit-learn 1.5+)

Evaluation Setup

  • Create a downstream train/validation/test split that reflects the intended deployment population.

  • Keep the test set untouched until model and threshold selection are complete.

  • Verify that the pretraining corpus does not contain downstream test examples, duplicated records, or future information.

  • Use StratifiedKFold for imbalanced classification, GroupKFold for related subjects, and TimeSeriesSplit when time order matters.

  • Put fine-tuning, early stopping, scaling, feature selection, and threshold selection inside each validation fold. A preprocessing step fitted on all downstream data causes leakage.

  • Evaluate source-domain validation separately; source validation scores do not guarantee target-domain performance.

What Transfer Learning Changes

  • Frozen representation: Train only the downstream head. Evaluation is relatively simple, but performance may remain limited if the source and target domains differ.

  • Fine-tuning: Update pretrained weights using a smaller downstream learning rate and layer-specific choices. Each cross-validation fold must include fine-tuning to obtain an honest estimate.

  • Negative transfer: Pretrained features can hurt performance when their representations are poorly suited to the target. Always compare against a target-only baseline.

  • Use identical splits when comparing a target-only model, a frozen model, and a fine-tuned model.

  • Tune using GridSearchCV or RandomizedSearchCV; use nested cross-validation when reporting an unbiased overall estimate. Compare models with confidence intervals or paired tests rather than one score alone.

Metrics and Model Selection

Choose metrics for the downstream task, not merely because they were used during pretraining.

Metric Formula Main use
Accuracy (TP + TN) / n Balanced classes; misleading with imbalance
Precision TP / (TP + FP) False positives are costly
Recall TP / (TP + FN) False negatives are costly
Specificity TN / (TN + FP) Correct negative identification
F1 2 * precision * recall / (precision + recall) Balance precision and recall
ROC-AUC Area under TPR versus FPR Overall ranking quality
PR-AUC Area under precision versus recall Usually more informative for rare positives
Log loss -mean(y*log(p) + (1-y)*log(1-p)) Quality of predicted probabilities
Brier score mean((p-y)^2) Probability calibration

Here, TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives; n is the sample count; y is the observed label; and p is the predicted probability.

For regression, use MAE for robust absolute error, MSE and RMSE for squared error, R² = 1 - SSE/SST for explained variance, and quantile loss for asymmetric costs. MAPE is unstable when targets are near zero. Evaluate calibration with reliability curves, and compare against a DummyClassifier or DummyRegressor baseline.

Common Pitfalls

  • Leakage: Pretraining on target test records or fitting preprocessing before splitting invalidates results.

  • Wrong validation: Random folds can break temporal, subject, or deployment-group dependencies.

  • Wrong metric: Accuracy can hide failure on rare positives; offline metrics may not reflect online business value.

  • Test-set tuning: Repeated threshold or hyperparameter changes on the test set turn it into a validation set.

  • Unverified overlap: Near-duplicates or shared entities can appear across source and target datasets.

Interviewers commonly ask how you would detect pretraining-data overlap, handle domain shift, select a threshold, distinguish frozen features from fine-tuning, and prove that transfer improves over a simple baseline.

53

How do ensemble learning models combine predictions and get evaluated?

Ensemble learning combines multiple models to improve accuracy, stability, and generalization. Bagging combines models in parallel, boosting adds learners sequentially, and stacking trains a meta-model on base-model outputs. The main trade-off is better predictive performance versus greater computation, tuning complexity, and reduced interpretability.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12 and scikit-learn 1.5+)

Evaluation Setup

Use a three-way evaluation protocol:

  • Training set: Fits model parameters.

  • Validation set: Selects hyperparameters, ensemble weights, stopping iterations, or decision thresholds.

  • Test set: Remains untouched until final evaluation for an unbiased performance estimate.

  • Cross-validation: Use KFold for general data, StratifiedKFold for imbalanced classification, GroupKFold for dependent groups, and TimeSeriesSplit for ordered data.

  • Model selection: Use GridSearchCV or RandomizedSearchCV; use nested cross-validation when tuning and reporting results on the same data.

  • Baseline: Compare against DummyClassifier before complex ensembles.

Internally, bagging can use out-of-bag estimates, boosting can use validation-based early stopping, and stacking requires out-of-fold base predictions for training its meta-model without leakage.

Combining Predictions

Family Combination method Strength Evaluation concern
Bagging, including random forests Trains models on bootstrap samples; classification uses voting or averaged probabilities, while regression uses averaged predictions Reduces variance Out-of-bag scores are possible; random seeds and ensemble size affect stability
Boosting, including gradient boosting Adds learners sequentially, with each learner correcting earlier errors; the final output is a weighted sum Reduces bias and underfitting Monitor validation performance and stop before overfitting
Stacking Trains base models, generates out-of-fold predictions, and trains a meta-model to combine them Combines complementary model strengths In-fold predictions would leak information into the meta-model

Evaluation Metrics

For classification, TP, TN, FP, and FN mean true positives, true negatives, false positives, and false negatives:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN); it can be misleading with imbalanced classes.

  • Precision = TP / (TP + FP); it measures the proportion of predicted positives that are correct.

  • Recall = TP / (TP + FN); it measures the proportion of actual positives detected.

  • Specificity = TN / (TN + FP); it measures the proportion of actual negatives correctly identified.

  • F1 = 2 × precision × recall / (precision + recall); it balances precision and recall.

  • Log loss = -(1/n) × sum(y_i × log(p_i) + (1-y_i) × log(1-p_i)). Here, n is the sample count, y_i is the binary true label, and p_i is the predicted probability.

  • Brier score = (1/n) × sum((p_i-y_i)^2); lower values indicate better probability calibration.

The ROC curve plots recall against false-positive rate across thresholds, and ROC-AUC summarizes ranking performance. The precision-recall curve plots precision against recall; PR-AUC is especially informative for rare positives. Reliability curves compare predicted probabilities with observed frequencies.

For regression, let n be the sample count, y_i the true value, and y_hat_i the prediction:

  • MAE = (1/n) × sum(|y_i-y_hat_i|); MSE = (1/n) × sum((y_i-y_hat_i)^2); RMSE = sqrt(MSE).

  • R² = 1 - sum((y_i-y_hat_i)^2) / sum((y_i-mean(y))^2), where mean(y) is the observed mean.

  • MAPE = (100/n) × sum(|(y_i-y_hat_i)/y_i|); it is unstable or undefined when true values are near zero.

  • Quantile loss = mean(max(tau × (y_i-y_hat_i), (1-tau) × (y_hat_i-y_i))), where tau is the target quantile.

Code Example

This example compares models using cross-validation, selects a threshold from out-of-fold predictions, and evaluates the untouched test set only once.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import (
    HistGradientBoostingClassifier,
    RandomForestClassifier,
)
from sklearn.metrics import (
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

models = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "random_forest": RandomForestClassifier(
        n_estimators=300, random_state=42, n_jobs=-1
    ),
    "gradient_boosting": HistGradientBoostingClassifier(
        max_iter=200, random_state=42
    ),
}

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "accuracy": "accuracy",
    "roc_auc": "roc_auc",
    "average_precision": "average_precision",
}

results = {}
for name, estimator in models.items():
    pipeline = Pipeline([
        ("scale", StandardScaler()),
        ("model", estimator),
    ])
    scores = cross_validate(
        pipeline, X_train, y_train,
        cv=cv, scoring=scoring, n_jobs=-1
    )
    results[name] = scores

best_name = max(
    results,
    key=lambda name: results[name]["test_roc_auc"].mean()
)
best_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", models[best_name]),
])

oof_probability = cross_val_predict(
    best_pipeline, X_train, y_train,
    cv=cv, method="predict_proba", n_jobs=-1
)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_train, oof_probability
)
f1 = np.divide(
    2 * precision * recall,
    precision + recall,
    out=np.zeros_like(precision),
    where=(precision + recall) != 0,
)
best_index = np.argmax(f1[:-1])
threshold = thresholds[best_index]

best_pipeline.fit(X_train, y_train)
test_probability = best_pipeline.predict_proba(X_test)[:, 1]
test_prediction = (test_probability >= threshold).astype(int)

print("Selected model:", best_name)
print("Selected threshold:", threshold)
print("Test ROC-AUC:", roc_auc_score(y_test, test_probability))
print("Test PR-AUC:", average_precision_score(y_test, test_probability))
print("Confusion matrix:\n", confusion_matrix(y_test, test_prediction))
print(classification_report(y_test, test_prediction, digits=4))

Common Pitfalls

  • Leakage: Fit scalers, imputers, encoders, and feature selectors only on training folds; place them inside the pipeline.

  • Tuning on the test set: Repeated test inspection converts it into a validation set.

  • Wrong split strategy: Random splits can invalidate grouped or time-dependent data.

  • Ignoring imbalance: Report precision, recall, F1, PR-AUC, and confusion matrices alongside accuracy.

  • Poor calibration: Compare Brier scores and reliability curves; consider CalibratedClassifierCV.

  • Unstable comparisons: Use repeated cross-validation, confidence intervals, or paired tests for small differences.

  • Business mismatch: Validate important decisions with A/B tests when possible.

  • Scaling and missing values: Trees do not require scaling, but k-NN, SVM, PCA, and gradient descent generally do; impute missing values inside the pipeline.

54

What is adversarial validation in machine learning?

Adversarial validation is a diagnostic technique for determining whether source and target datasets have measurably different feature distributions. It trains a classifier to distinguish the datasets, revealing potential covariate shift that could make a source-trained model unreliable on target data. Its main trade-off is that it identifies observable differences but does not explain their cause or prove that model performance will degrade.

Read full answer

Applies tomachine-learning concepts (all versions)

Evaluation Setup

A typical machine-learning workflow divides labeled data into:

  • Training set: Used to fit model parameters.

  • Validation set: Used for hyperparameter selection and model comparison.

  • Test set: Kept untouched until final evaluation.

Preprocessing such as scaling, imputation, and feature selection must be fitted only on training data. Cross-validation can replace a single validation split: use StratifiedKFold for classification, GroupKFold for grouped observations, and TimeSeriesSplit when temporal ordering matters.

Adversarial validation requires an additional evaluation setup. Source and target observations are assigned a domain label, and a domain classifier is evaluated on held-out observations or through cross-validation. This domain-classifier test split is separate from the final model test set, which should remain untouched. If target data is reserved for final testing, use separate unlabeled target observations for adversarial validation.

How It Works

The domain classifier learns to answer:

“Was this observation sampled from the source dataset or the target dataset?”

For example, a logistic regression can receive each observation’s features and predict its probability of belonging to the target domain. When target is the positive class, its ROC-AUC can be written as:

AUC = P(score(target observation) > score(source observation))

Here, score is the target-domain probability, P is the probability over randomly selected source and target observations, and ties between scores count as one-half.

  • AUC near 0.5: The datasets are difficult to distinguish, suggesting little observable covariate shift, subject to sampling variation.

  • AUC substantially above 0.5: The classifier consistently identifies the target domain, suggesting meaningful feature-distribution differences.

  • AUC near 1.0: The datasets are highly separable, usually indicating strong shift.

A high AUC means the observable feature distributions differ, not that the target labels differ or that the deployed model will fail. The domain classifier might detect an irrelevant feature while missing a smaller but business-critical shift. Examining feature importance or permutation importance can help identify which features drive the separation.

Adversarial validation can detect changes in P(X), where X represents input features. It does not directly establish a change in P(Y | X), where Y represents the target label and P(Y | X) is the label distribution conditional on the features.

Where It Is Used

  • Detecting population differences: Compare training customers with production customers, or users from different countries, devices, or acquisition channels.

  • Assessing temporal drift: Compare older labeled data with recent unlabeled production traffic.

  • Supporting data weighting: Estimate domain probabilities out of fold and use the appropriate source-to-target propensity ratio, with stabilization or clipping when necessary, to make the weighted training distribution more closely resemble the target distribution.

  • Guiding data collection: Identify features that differ most, helping prioritize collecting or labeling target-domain examples.

  • Deciding model strategy: Strong shift may justify target-specific training, domain adaptation, or more representative labeled data.

  • Validating offline-to-online assumptions: Before relying on offline metrics, check whether production users resemble the offline evaluation population.

The shift terminology matters. Covariate shift means P(X) changes while P(Y | X) remains stable. Label shift means P(Y) changes while P(Y | X) remains stable. Concept shift means P(Y | X) changes. Adversarial validation primarily probes the first of these through observable features.

Common Pitfalls

  • Data leakage: Do not use the final model test set to tune the domain classifier or preprocessing.

  • Overinterpreting AUC: A high AUC identifies distribution differences, not their cause or business importance.

  • Confusing shift types: Do not treat evidence of covariate shift as proof of label shift or concept shift.

  • Using in-sample predictions: Fit domain scores out of fold before using them for weighting; otherwise, the classifier may appear unrealistically effective.

  • Ignoring grouped or temporal structure: Use GroupKFold or TimeSeriesSplit when random splitting would mix related observations or use future information.

  • Assuming similarity guarantees transferability: Similar feature distributions do not guarantee consistent labels, unchanged feature-label relationships, or strong business impact.

A useful interview follow-up is: “What would you do after adversarial validation shows a large distribution shift?” A strong answer identifies the differing features, protects the test set, evaluates whether the shift affects relevant predictions, and then considers reweighting, domain adaptation, or collecting representative target labels.

55

How does model drift work, and how is it measured?

Model drift is a change in the relationship between input features and the target, often called concept drift. It is usually detected by comparing a reference population with recent production data using distribution metrics, or by comparing production performance with a validated baseline. The main trade-off is that feature-based monitoring is fast but indirect, while labeled performance monitoring identifies the actual impact but often suffers from delayed labels.

Read full answer

Applies tomachine learning concepts (Python 3.12 examples with scikit-learn 1.5+ and SciPy 1.13+)

Evaluation Setup

A sound evaluation and monitoring process separates three roles:

  • Training set: Fits model parameters.

  • Validation set: Selects hyperparameters, preprocessing choices, and classification thresholds.

  • Test set: Measures final generalization and remains untouched until all decisions are complete.

For production drift, preserve a reference period and compare it with a recent current period. Use StratifiedKFold for imbalanced classification, GroupKFold for related observations, and TimeSeriesSplit for temporal data. GridSearchCV or RandomizedSearchCV can tune models; nested cross-validation provides more honest estimates because model selection occurs inside each outer fold.

Types of Drift

Type Meaning Typical Detection
Data drift Feature distribution changes PSI, KS, Wasserstein distance, population mean difference
Concept drift The relationship ‘P(target features)‘ changes Performance degradation, change-detection tests, error-slice analysis
Model drift Operational term for model behavior becoming outdated or inaccurate Combine feature, performance, calibration, and business-metric monitoring

Data drift does not necessarily imply concept drift. For example, a feature can change distribution while the model’s conditional accuracy remains stable.

Measuring Drift

For a numeric feature, Population Stability Index (PSI) compares reference and current proportions:

PSI = sum((current_pct - reference_pct) * ln(current_pct / reference_pct))

Here, reference_pct is the proportion in each baseline bin, current_pct is the corresponding current proportion, and ln is the natural logarithm. Larger values indicate greater instability, but thresholds such as 0.10 and 0.25 are heuristics rather than universal rules.

Other useful measures include:

  • Kolmogorov-Smirnov statistic: Maximum difference between reference and current empirical cumulative distributions.

  • Wasserstein distance: Magnitude of the shift between numerical distributions.

  • Total variation distance for categorical features: 0.5 * sum(abs(current_probability - reference_probability)).

  • Slice analysis: Checks whether particular customer groups, regions, devices, or time windows deteriorated.

  • Change detectors: Page-Hinkley or ADWIN can identify abrupt shifts in monitored metrics.

Choosing Performance Metrics

For classification, define TP, TN, FP, and FN as true positives, true negatives, false positives, and false negatives:

  • Accuracy: (TP + TN) / (TP + TN + FP + FN).

  • Precision: TP / (TP + FP).

  • Recall: TP / (TP + FN).

  • F1: 2TP / (2TP + FP + FN).

  • Specificity: TN / (TN + FP).

  • Log loss: -1/N * sum(y * log(p) + (1-y) * log(1-p)), where N is the sample count, y is the binary target, and p is the predicted positive probability.

  • Brier score: 1/N * sum((p-y)^2), measuring probability calibration error.

Use ROC-AUC for ranking discrimination and PR-AUC for imbalanced or rare-positive problems. Accuracy can be misleading when classes are imbalanced. Reliability curves compare predicted probabilities with observed event rates.

For regression, compare MAE, RMSE, and R²; use quantile loss for asymmetric costs. MAPE is often unstable when the target is near zero.

Code Example

import numpy as np
from scipy.stats import ks_2samp, wasserstein_distance
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score, brier_score_loss, classification_report,
    confusion_matrix, log_loss, precision_recall_curve, roc_auc_score,
)
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=42
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=5000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    model, X_train, y_train, cv=cv,
    scoring=["accuracy", "precision", "recall", "f1", "roc_auc",
             "average_precision"],
)
print("CV accuracy:", cv_results["test_accuracy"].mean())

# Tune the decision threshold using validation data only.
model.fit(X_train, y_train)
val_prob = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_prob)
# precision and recall have one more element than thresholds; compute F1 for each threshold
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(thresholds),
    where=(precision[:-1] + recall[:-1]) > 0,
)
threshold = thresholds[np.argmax(f1)]

test_prob = model.predict_proba(X_test)[:, 1]
test_pred = (test_prob >= threshold).astype(int)

print(classification_report(y_test, test_pred, digits=3))
print("Confusion matrix:\n", confusion_matrix(y_test, test_pred))
print("ROC-AUC:", roc_auc_score(y_test, test_prob))
print("PR-AUC:", average_precision_score(y_test, test_prob))
print("Log loss:", log_loss(y_test, test_prob))
print("Brier score:", brier_score_loss(y_test, test_prob))

# Illustrative numeric-feature drift between reference and current data.
reference = X_train[:, 0]
current = X_test[:, 0]
edges = np.unique(np.quantile(reference, np.linspace(0, 1, 11)))
edges[0], edges[-1] = -np.inf, np.inf

ref_hist, _ = np.histogram(reference, bins=edges)
cur_hist, _ = np.histogram(current, bins=edges)
ref_pct = np.clip(ref_hist / ref_hist.sum(), 1e-6, None)
cur_pct = np.clip(cur_hist / cur_hist.sum(), 1e-6, None)
psi = np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct))

print("PSI:", psi)
print("KS statistic:", ks_2samp(reference, current).statistic)
print("Wasserstein distance:", wasserstein_distance(reference, current))

Common Pitfalls

  • Do not tune thresholds, features, or hyperparameters on the test set.

  • Do not interpret feature drift alone as proof of concept drift.

  • Use temporal or grouped splits when random splitting leaks information.

  • Compare like-for-like periods and retain the same metric and threshold.

  • Monitor business outcomes and validate important changes through an A/B test when possible.

  • Include a DummyClassifier baseline and investigate performance degradation by data slice before retraining.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 55 Model Evaluation answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects