Skip to content
Machine Learning & Data Science

Supervised Learning interview questions and answers

70 questions, 70 with code. About 3 h 42 min to read them all.

By TestLab · Updated

Plan for an interview
70 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is supervised learning?

Supervised learning is the branch of machine learning in which a model learns a mapping from inputs (features) to known outputs (labels) using examples where the correct answer is given. After training, the model predicts labels for new, unseen inputs. It splits into classification, where the output is a category (spam or not spam), and regression, where the output is a number (house price). Most production ML, from credit scoring to medical diagnosis, is supervised.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Key Ideas

Concept Meaning
Features X Measurable inputs (age, pixel values, word counts)
Label y The answer to learn (class or number)
Training Adjusting model parameters to reduce a loss on labelled examples
Generalization Performing well on new data, not just the training set
Evaluation Measuring performance on held-out data with suitable metrics

Workflow

  1. Collect and label representative data.

  2. Split into training and test sets (and use cross-validation for tuning).

  3. Preprocess: clean, encode categories, scale numeric features.

  4. Build a simple baseline.

  5. Train candidate models and tune hyperparameters with cross-validation.

  6. Evaluate once on the test set.

  7. Deploy and monitor for drift; retrain as needed.

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)                  # features and labels (malignant/benign)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

baseline = DummyClassifier(strategy="most_frequent")
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

for name, est in [("baseline", baseline), ("logistic regression", model)]:
    print(f"{name:20s} CV accuracy {cross_val_score(est, X_train, y_train, cv=5).mean():.3f}")

model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), target_names=["malignant", "benign"]))

Compared with Other Paradigms

Paradigm Labels Example
Supervised Full labels Predict churn from customer history
Unsupervised None Cluster customers into segments
Semi-supervised Few labels + many unlabelled Label propagation for documents
Self-supervised Labels created from the data itself Masked-word pretraining of language models
Reinforcement Rewards from interaction Game-playing agents

Common Pitfalls

  • Poor label quality: noisy or inconsistent labels cap the achievable accuracy.

  • Data leakage: features that contain information unavailable at prediction time.

  • Unrepresentative data: training data that differs from the data the model will see.

2

What is the difference between supervised and unsupervised learning?

Supervised learning trains on labelled examples, inputs paired with the correct outputs, and learns to predict those outputs for new inputs. Unsupervised learning has no labels; it finds structure in the inputs alone, such as clusters, lower-dimensional representations, or unusual points. Supervised models are judged by how well their predictions match known answers, while unsupervised results are judged by internal measures, usefulness for a downstream task, or human inspection.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Aspect Supervised Unsupervised
Data Features + labels Features only
Goal Predict a target Discover structure
Tasks Classification, regression Clustering, dimensionality reduction, anomaly detection, density estimation
Algorithms Logistic/linear regression, trees, gradient boosting, SVM, neural networks k-means, DBSCAN, hierarchical clustering, PCA, t-SNE/UMAP, isolation forest
Evaluation Accuracy, F1, ROC AUC, MAE on held-out labels Silhouette, reconstruction error, business usefulness
Cost Labelling can be expensive Data is cheap, interpretation is harder
Example Predict whether a customer will churn Group customers into segments

In Between

  • Semi-supervised: a few labels plus many unlabelled examples (label propagation, pseudo-labelling).

  • Self-supervised: labels created from the data itself (predicting masked words), the basis of modern foundation models.

  • Unsupervised as preprocessing: PCA features or cluster IDs used as inputs to a supervised model.

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)

# Supervised: uses the species labels to learn a classifier
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
print("supervised CV accuracy:", round(cross_val_score(clf, X, y, cv=5).mean(), 3))

# Unsupervised: groups the flowers without seeing labels
X_scaled = StandardScaler().fit_transform(X)
clusters = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X_scaled)
print("silhouette (internal quality):", round(silhouette_score(X_scaled, clusters), 3))
print("agreement with true species (ARI):", round(adjusted_rand_score(y, clusters), 3))

The adjusted Rand index is only possible here because the iris labels happen to be known; in real unsupervised work there are usually no labels to compare against.

Choosing

  • Use supervised learning when you know what to predict and can obtain labels.

  • Use unsupervised learning to explore data, create features, detect anomalies or when labels are unavailable.

Common Pitfalls

  • Expecting clusters to match a specific business category without supervision.

  • Evaluating a supervised model on training data, or an unsupervised one only by eye.

3

What types of problems can supervised learning solve?

Supervised learning solves any problem where you can collect examples of inputs paired with the correct outputs and want to predict that output for new inputs. The main families are classification (predict a category: binary, multi-class or multi-label), regression (predict a number), and structured variants such as ranking, sequence labelling, object detection and forecasting. The type of target decides the model’s output layer, loss function and evaluation metrics.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Problem Types

Type Target Examples Typical metrics
Binary classification Two classes Spam detection, loan default, disease yes/no ROC AUC, F1, precision/recall
Multi-class classification One of K classes Digit recognition, product category Accuracy, macro F1
Multi-label classification Several labels at once Tagging articles with topics Micro/macro F1, Hamming loss
Regression Continuous value House price, demand, delivery time MAE, RMSE, R²
Ordinal regression Ordered categories Star ratings, severity grades MAE on ranks, weighted kappa
Ranking Order of items Search results, recommendations NDCG, MAP
Sequence labelling Label per token Named-entity recognition, part-of-speech tagging Entity-level F1
Object detection / segmentation Boxes or pixel masks Self-driving perception, medical imaging mAP, IoU, Dice
Time-series forecasting Future values Sales, energy load MAE, MASE
Survival analysis Time until an event Customer lifetime, equipment failure Concordance index

Code Example: Same Data, Different Problem Types

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import cross_val_score

X, y = load_diabetes(return_X_y=True)                     # target: disease progression score

# Regression: predict the score itself
reg = RandomForestRegressor(n_estimators=200, random_state=42)
print("regression MAE:", round(-cross_val_score(reg, X, y, cv=5, scoring="neg_mean_absolute_error").mean(), 1))

# Binary classification: is progression above the median?
y_bin = (y > np.median(y)).astype(int)
clf = RandomForestClassifier(n_estimators=200, random_state=42)
print("binary ROC AUC:", round(cross_val_score(clf, X, y_bin, cv=5, scoring="roc_auc").mean(), 3))

# Multi-class: low / medium / high progression (tertiles)
y_multi = np.digitize(y, np.quantile(y, [1 / 3, 2 / 3]))
print("3-class macro F1:", round(cross_val_score(clf, X, y_multi, cv=5, scoring="f1_macro").mean(), 3))

Turning a numeric target into classes discards information, so do it only when the decision really is categorical.

Choosing the Framing

  • Start from the decision the prediction supports: a yes/no action suggests classification; a quantity to plan with suggests regression.

  • Check that labels can be collected reliably and that the input features will be available at prediction time.

Common Pitfalls

  • Treating ordered categories as unordered classes, or as exact numbers when gaps are uneven.

  • Using accuracy for imbalanced or multi-label problems.

4

How are training and testing datasets used in supervised learning?

The training set is the labelled data a model learns from, and the test set is separate labelled data held back to estimate how the model will perform on new, unseen examples. Because a model can memorize its training data, performance must be measured on data it never saw. In practice there is usually a third role, validation, for choosing models and hyperparameters, done with a separate validation set or with cross-validation on the training data, so the test set is used only once at the end.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Roles

Set Used for Typical share
Training Fitting model parameters 60-80%
Validation (or CV folds) Choosing hyperparameters, features, model type, early stopping 10-20%, or k-fold CV within training
Test Final, unbiased estimate of performance 10-20%; touched once

Splitting Strategies

Data Strategy
i.i.d. tabular, balanced Random split
Imbalanced classes Stratified split (stratify=y)
Time series Chronological split; train on past, test on future
Grouped data (patients, users) Group split so the same group is not in both sets
Small data k-fold cross-validation, nested CV for unbiased tuning

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)

# 1. Hold out a test set first and do not look at it until the end
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# 2. Validation happens inside the training data via 5-fold cross-validation
search = GridSearchCV(RandomForestClassifier(random_state=42),
                      {"max_depth": [3, 5, None], "min_samples_leaf": [1, 5]}, cv=5)
search.fit(X_train, y_train)
print("best params:", search.best_params_, "| CV accuracy:", round(search.best_score_, 3))

# 3. Final evaluation, once
print("training accuracy:", round(search.score(X_train, y_train), 3))
print("test accuracy:    ", round(search.score(X_test, y_test), 3))

A training accuracy much higher than the test accuracy indicates overfitting.

Common Pitfalls

  • Leakage: fitting scalers, imputers or feature selection on all data before splitting; use a Pipeline.

  • Reusing the test set for repeated decisions, which turns it into a validation set and makes the estimate optimistic.

  • Duplicates or related records (same patient, same user) across sets.

  • Random splits for time-ordered data, which let the model see the future.

5

What role does a loss function play in supervised learning?

A loss function measures how wrong a model’s prediction is for a training example, turning errors into a number the training algorithm can minimize. It defines what “good” means during training: the model’s parameters are adjusted, for example by gradient descent, to reduce the average loss. Choosing the loss is a modelling decision: it should match the type of target and the cost of different errors, and it is often different from the business metric used for evaluation.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Common Losses

Task Loss Idea in words Used by
Regression Squared error Square of the difference; punishes large errors heavily; gives the mean Linear regression
Regression Absolute error Size of the difference; robust to outliers; gives the median Quantile/LAD regression
Regression Huber Squared for small errors, absolute for large Robust regression
Regression Quantile (pinball) Asymmetric penalty for under- vs over-prediction Prediction intervals
Classification Log loss (cross-entropy) Penalizes confident wrong probabilities Logistic regression, neural networks
Classification Hinge Penalizes points inside the margin SVMs
Classification 0-1 loss Counts mistakes; not differentiable Evaluation only

Loss Versus Metric

  • Loss: smooth and optimizable, used during training (log loss).

  • Metric: what the business cares about, used for evaluation (F1, revenue, MAPE).

  • They should point in the same direction; otherwise adjust the loss, weights or decision threshold.

Regularized Objective

Most models minimize loss + penalty, for example average squared error plus alpha times the sum of squared weights (ridge). The penalty discourages overly complex models.

Code Example

import numpy as np
from sklearn.linear_model import HuberRegressor, LinearRegression, QuantileRegressor

rng = np.random.default_rng(42)
X = rng.uniform(0, 10, (200, 1))
y = 3 * X.ravel() + rng.normal(0, 2, 200)
y[:10] += 60                                               # a few large outliers

for name, model in [("squared loss (OLS)", LinearRegression()),
                    ("Huber loss", HuberRegressor()),
                    ("absolute loss (median)", QuantileRegressor(quantile=0.5, alpha=0.0)),
                    ("pinball loss, 90th pct", QuantileRegressor(quantile=0.9, alpha=0.0))]:
    model.fit(X, y)
    print(f"{name:24s} slope {model.coef_[0]:.2f}, intercept {model.intercept_:.2f}")

# Log loss for classification: confident wrong predictions cost the most
def log_loss_single(y_true, p):
    return -(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))

for p in (0.9, 0.6, 0.1):
    print(f"true class 1, predicted p={p}: log loss {log_loss_single(1, p):.3f}")

The true slope is 3; the outliers pull the squared-loss fit upward more than the Huber and absolute-loss fits.

Common Pitfalls

  • Using squared error with heavy outliers when the business cares about typical errors.

  • Optimizing log loss but judging only by accuracy without tuning the threshold.

6

What are overfitting and underfitting in machine learning models?

Overfitting happens when a model learns the training data too closely, including its noise and quirks, so it performs well on training data but poorly on new data. Underfitting happens when a model is too simple, or too constrained, to capture the real pattern, so it performs poorly even on the training data. The goal is the middle ground where the model captures the true signal and generalizes, which is found by comparing training and validation performance while adjusting model complexity.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Diagnosis

Situation Training error Validation error Typical cause
Underfitting High High (close to training) Model too simple, too much regularization, missing features
Good fit Low Low, slightly higher than training Appropriate complexity
Overfitting Very low Much higher Model too complex or too little data, so noise is memorized

Remedies

Underfitting Overfitting
More flexible model (deeper trees, more layers, non-linear kernels) Simpler model, fewer features
Less regularization More regularization (L1/L2, dropout)
Better or more features More training data, data augmentation
Train longer Early stopping
Ensembles (bagging), cross-validated tuning

Code Example: Polynomial Degree

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

rng = np.random.default_rng(42)
X = np.sort(rng.uniform(-3, 3, 60))[:, None]
y = np.sin(X).ravel() + rng.normal(0, 0.3, 60)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)

for degree in (1, 3, 15):
    model = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression()).fit(X_train, y_train)
    tr = mean_squared_error(y_train, model.predict(X_train))
    va = mean_squared_error(y_val, model.predict(X_val))
    print(f"degree {degree:2d}: train MSE {tr:.3f}, validation MSE {va:.3f}")

Degree 1 (a straight line) underfits the sine curve, degree 3 fits it well, and degree 15 typically has the lowest training error but a higher validation error because it bends to follow noise.

Tools

  • Learning curves: plot training and validation scores as data grows; a persistent gap means overfitting, two low curves mean underfitting.

  • Validation curves: plot scores against one hyperparameter (depth, C, alpha).

Common Pitfalls

  • Judging a model by training accuracy alone.

  • Mistaking leakage for a good fit: suspiciously high validation scores deserve a leakage check.

7

What methods can be used to prevent overfitting?

Overfitting is prevented by limiting how much a model can memorize noise and by giving it more and better evidence. The main methods are more or augmented data, simpler models or fewer features, regularization (L1/L2 penalties, dropout, tree constraints), early stopping, ensembles such as bagging, and honest validation with cross-validation so that tuning itself does not overfit. Which to use depends on the model family and on whether data or capacity is the real constraint.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Methods by Category

Category Method Applies to
Data Collect more data; data augmentation; remove label noise All models
Features Feature selection, dimensionality reduction (PCA) Linear models, k-NN, SVM
Penalties L2 (ridge), L1 (lasso), elastic net Linear/logistic regression, neural networks
Model capacity Max depth, min samples per leaf, fewer layers/units Trees, neural networks
Training control Early stopping on a validation set Gradient boosting, neural networks
Noise Dropout, label smoothing Neural networks
Ensembles Bagging, random forests, averaging High-variance models
Validation Cross-validation, nested CV, a single-use test set Model selection

Code Example: Several Methods on One Problem

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_validate
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=800, n_features=30, n_informative=6, flip_y=0.1, random_state=42)

models = {
    "unpruned tree": DecisionTreeClassifier(random_state=42),
    "tree, max_depth=4, min_leaf=10": DecisionTreeClassifier(max_depth=4, min_samples_leaf=10, random_state=42),
    "random forest (bagging)": RandomForestClassifier(n_estimators=300, random_state=42),
    "boosting + early stopping": HistGradientBoostingClassifier(early_stopping=True, validation_fraction=0.15,
                                                               random_state=42),
}
for name, model in models.items():
    res = cross_validate(model, X, y, cv=5, return_train_score=True)
    print(f"{name:32s} train {res['train_score'].mean():.3f}  val {res['test_score'].mean():.3f}")

The unpruned tree should reach a training accuracy of 1.0 with a much lower validation score; the constrained and ensemble models narrow that gap.

Practical Order of Attack

  1. Check for leakage and data quality problems first.

  2. Get more or augmented data if possible.

  3. Add regularization or constrain capacity, tuned with cross-validation.

  4. Use ensembles or early stopping.

  5. Keep the test set untouched until the end.

Common Pitfalls

  • Over-regularizing until the model underfits.

  • Tuning many hyperparameters on a small validation set, which overfits the validation set.

8

What is the bias-variance tradeoff?

The bias-variance tradeoff describes two sources of prediction error that move in opposite directions as model complexity changes. Bias is error from overly simple assumptions: the model systematically misses the true pattern (underfitting). Variance is error from sensitivity to the particular training sample: the model changes a lot with different data (overfitting). For squared error, expected error = bias² + variance + irreducible noise, so the best model balances the two rather than minimizing either alone.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

How Complexity Affects It

Model Bias Variance Example
Very simple High Low Straight line on curved data
Balanced Moderate Moderate Tuned gradient boosting
Very complex Low High Unpruned deep decision tree

Levers

To reduce bias To reduce variance
More flexible model Regularization (L1/L2, dropout)
Better features More data
Less regularization Bagging / random forests
Boosting Simpler model, feature selection

Code Example: Estimating Bias and Variance by Resampling

import numpy as np
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(42)
true_f = np.sin
x_test = np.linspace(0, 6, 100)[:, None]

def sample(n=50):
    x = rng.uniform(0, 6, (n, 1))
    return x, true_f(x).ravel() + rng.normal(0, 0.3, n)

for depth in (1, 3, 10):
    preds = []
    for _ in range(200):                                   # 200 different training sets
        x, y = sample()
        preds.append(DecisionTreeRegressor(max_depth=depth, random_state=42).fit(x, y).predict(x_test))
    preds = np.array(preds)
    bias_sq = np.mean((preds.mean(axis=0) - true_f(x_test).ravel()) ** 2)
    variance = np.mean(preds.var(axis=0))
    print(f"depth {depth:2d}: bias^2 {bias_sq:.4f}, variance {variance:.4f}, sum {bias_sq + variance:.4f}")

Shallow trees show high bias and low variance; deep trees show the reverse; a moderate depth gives the lowest total. The noise variance (0.09 here) is added on top and cannot be reduced by any model.

Modern Nuance

Very large neural networks and ensembles can show “double descent”, where test error falls again beyond the point of interpolating the training data. The classic tradeoff still guides most practical tuning.

Interview Tips

  • Give the decomposition in words and an example of each side.

  • Link remedies to the side they address: bagging reduces variance, boosting reduces bias.

9

What are validation sets and cross-validation?

A validation set is labelled data held out from training and used to compare models and choose hyperparameters, so those choices are not made on the test set. Cross-validation generalizes this: the training data is split into k folds, the model is trained k times, each time validating on a different fold, and the k scores are averaged. Cross-validation uses data more efficiently and gives a more stable estimate, which is important for small datasets; a single validation split is cheaper for large data or expensive models.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Method How Pros Cons
Single validation split Train on e.g. 80%, validate on 20% Fast; simple Noisy estimate; wastes data
k-fold CV k train/validate rounds Stable estimate; uses all data k times the cost
Stratified k-fold Keeps class ratios per fold Needed for imbalanced classes —
Group k-fold Groups (patients, users) never split across folds Prevents leakage Needs group IDs
Time-series split Train on past, validate on future, rolling forward Respects time order Fewer training points early
Leave-one-out k = n Uses almost all data Very costly, high variance
Nested CV Inner CV tunes, outer CV evaluates Unbiased estimate after tuning Expensive

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (GridSearchCV, StratifiedKFold, cross_val_score, cross_validate,
                                     train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC())

# 1. Single validation split
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.25, stratify=y_train, random_state=42)
print("single split accuracy:", round(pipe.fit(X_tr, y_tr).score(X_val, y_val), 3))

# 2. Stratified 5-fold CV
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
res = cross_validate(pipe, X_train, y_train, cv=cv, scoring=["accuracy", "roc_auc"])
print(f"5-fold accuracy {res['test_accuracy'].mean():.3f} ± {res['test_accuracy'].std():.3f}")

# 3. Nested CV: inner loop tunes C, outer loop estimates performance of the whole procedure
inner = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10]}, cv=StratifiedKFold(3, shuffle=True, random_state=42))
print("nested CV accuracy:", round(cross_val_score(inner, X_train, y_train, cv=cv).mean(), 3))

# 4. Final check on the untouched test set
print("test accuracy:", round(inner.fit(X_train, y_train).score(X_test, y_test), 3))

Common Pitfalls

  • Preprocessing (scaling, imputation, feature selection) fitted outside the folds; keep it in a Pipeline.

  • Shuffled k-fold on time series or grouped data.

  • Reporting the best CV score after tuning as the expected performance; use nested CV or the test set.

10

What is regularization, and how does it work?

Regularization is any technique that discourages a model from becoming too complex, so it generalizes better to new data. Most commonly it adds a penalty term to the training objective: the model minimizes the data loss plus a cost on the size of its weights, which pulls weights toward zero and makes the learned function smoother. The strength of the penalty is a hyperparameter (alpha, lambda or the inverse C) chosen with cross-validation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Penalty Types

Type Penalty (in words) Effect Model
L2 (ridge) alpha × sum of squared weights Shrinks all weights smoothly; handles correlated features Ridge, LogisticRegression default
L1 (lasso) alpha × sum of absolute weights Sets some weights exactly to zero: feature selection Lasso, LogisticRegression(penalty="l1")
Elastic net Mix of L1 and L2 Sparse but stable with correlated features ElasticNet

Other Forms of Regularization

  • Tree constraints: max_depth, min_samples_leaf, pruning (ccp_alpha).

  • Neural networks: weight decay, dropout, early stopping, data augmentation.

  • Ensembles: averaging many models reduces variance.

Why It Works

Penalizing large weights limits how sharply the model can react to small changes in the inputs, which reduces variance at the cost of a little bias. From a Bayesian view, L2 corresponds to a Gaussian prior on the weights and L1 to a Laplace prior.

Code Example

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_regression(n_samples=80, n_features=60, n_informative=8, noise=15, random_state=42)

models = {"no penalty": LinearRegression()}
for alpha in (0.1, 1, 10):
    models[f"ridge alpha={alpha}"] = Ridge(alpha=alpha)
    models[f"lasso alpha={alpha}"] = Lasso(alpha=alpha, max_iter=10000)

for name, est in models.items():
    pipe = make_pipeline(StandardScaler(), est)
    rmse = -cross_val_score(pipe, X, y, cv=5, scoring="neg_root_mean_squared_error").mean()
    pipe.fit(X, y)
    coefs = pipe[-1].coef_
    print(f"{name:18s} CV RMSE {rmse:7.2f} | weight norm {np.linalg.norm(coefs):7.1f} | "
          f"zero weights {int(np.sum(np.abs(coefs) < 1e-8))}")

With 60 features and only 80 samples, the unpenalized model overfits; ridge shrinks the weights and lasso zeroes out most uninformative features.

Common Pitfalls

  • Not scaling features: the penalty then punishes features unevenly.

  • Confusing alpha (larger = stronger) with C (larger = weaker) across APIs.

  • Regularizing the intercept, which most libraries correctly avoid.

11

Describe linear regression.

Linear regression models a numeric target as a weighted sum of input features plus an intercept, y ≈ b0 + b1 x1 + ... + bp xp, and chooses the weights that minimize the sum of squared differences between predictions and actual values (ordinary least squares). It is fast, interpretable, and a strong baseline; each coefficient states how much the prediction changes per unit change in a feature with the others held constant. Its accuracy depends on the relationship being roughly linear, or made linear through feature engineering.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

How It Is Fitted

  • Closed form: the normal equation solves for the weights directly (computed stably with QR or SVD decompositions).

  • Gradient descent: iteratively moves weights against the gradient of the squared error; used for very large data (SGDRegressor).

  • Regularized variants: ridge (L2), lasso (L1), elastic net add penalties to reduce overfitting.

Assumptions

Assumption Check
Linearity Residual vs fitted plot shows no pattern
Independent errors No autocorrelation (Durbin-Watson for time data)
Constant variance Residual spread does not grow with fitted values
Normal errors (for inference) Q-Q plot of residuals
Low multicollinearity Variance inflation factors

Code Example

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import cross_val_score, train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

for name, model in [("mean baseline", DummyRegressor()), ("linear regression", LinearRegression())]:
    mae = -cross_val_score(model, X_train, y_train, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name:18s} CV MAE {mae:.1f}")

lr = LinearRegression().fit(X_train, y_train)
pred = lr.predict(X_test)
print(f"test MAE {mean_absolute_error(y_test, pred):.1f}, R² {r2_score(y_test, pred):.3f}")
for name, coef in sorted(zip(X.columns, lr.coef_), key=lambda t: -abs(t[1]))[:4]:
    print(f"{name:4s} {coef:8.1f}")

# The normal-equation solution matches scikit-learn's
Xb = np.c_[np.ones(len(X_train)), X_train]
w = np.linalg.lstsq(Xb, y_train, rcond=None)[0]
print("matches:", np.allclose(w[1:], lr.coef_))

The diabetes features are already centred and scaled, so their coefficients can be compared directly; with raw features, standardize first.

Strengths and Limits

Strengths Limits
Fast, interpretable, few hyperparameters Misses non-linear patterns unless features are engineered
Works well with many features when regularized Sensitive to outliers (squared loss)
Provides confidence intervals (statsmodels) Correlated features make coefficients unstable

Common Pitfalls

  • Extrapolating far outside the training range.

  • Interpreting coefficients causally without accounting for confounders.

12

What is the difference between simple and multiple linear regression?

Simple linear regression models a numeric target with one input feature, fitting a straight line y = b0 + b1 x. Multiple linear regression uses two or more features, fitting a plane or hyperplane y = b0 + b1 x1 + b2 x2 + ... + bp xp. Both estimate coefficients by minimizing the sum of squared errors, but multiple regression lets each coefficient show a feature’s effect while holding the other features constant, and it brings new issues such as multicollinearity and feature selection.

Read full answer

Applies toscikit-learn 1.5+, statsmodels 0.14+ (Python 3.12)

Comparison

Aspect Simple Multiple
Features One Two or more
Shape Line Plane / hyperplane
Coefficient meaning Change in y per unit of x Change in y per unit of x_j, holding the others fixed
Visualization Scatter plot with a line Partial plots; hard to visualize beyond 2 features
Extra concerns Few Multicollinearity, adjusted R², feature selection, interactions
Confounding Coefficient can absorb effects of omitted variables Can adjust for measured confounders

Code Example

import numpy as np
import pandas as pd
import statsmodels.api as sm

rng = np.random.default_rng(42)
n = 300
size = rng.normal(120, 30, n)                            # m²
age = rng.uniform(0, 50, n)                              # years
# Larger houses tend to be newer in this market, so size and age are correlated
age = age - 0.2 * (size - 120)
price = 50 + 2.0 * size - 1.5 * age + rng.normal(0, 20, n)
df = pd.DataFrame({"size": size, "age": age, "price": price})

simple = sm.OLS(df["price"], sm.add_constant(df[["size"]])).fit()
multiple = sm.OLS(df["price"], sm.add_constant(df[["size", "age"]])).fit()

print("simple:   size coef", round(simple.params["size"], 3), "| R²", round(simple.rsquared, 3))
print("multiple: size coef", round(multiple.params["size"], 3), "age coef", round(multiple.params["age"], 3),
      "| adj R²", round(multiple.rsquared_adj, 3))

The true size effect is 2.0. In the simple model the size coefficient also absorbs part of the age effect (bigger houses are newer, so they look even more valuable); the multiple model separates the two effects.

Assumptions (Both)

  • Linear relationship between features and target.

  • Independent errors with constant variance (homoscedasticity).

  • Approximately normal errors for valid confidence intervals and p-values.

  • For multiple regression: no perfect multicollinearity among features.

Common Pitfalls

  • Interpreting a simple-regression slope as causal when other variables matter.

  • Comparing R² across models with different numbers of features; use adjusted R² or cross-validation.

  • Ignoring correlated features, which inflate coefficient variances (check variance inflation factors).

13

What is logistic regression, and when is it used?

Logistic regression is a classification model that predicts the probability of a class by passing a weighted sum of features through the sigmoid function, p = 1 / (1 + exp(-(b0 + b·x))), which maps any number into (0, 1). It is trained by minimizing log loss (cross-entropy). It is used when you need a fast, interpretable, well-calibrated baseline for binary or multi-class classification, such as credit default, churn, disease risk or click prediction, and when you want to explain how each feature shifts the odds.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Key Ideas

  • Log-odds are linear: log(p / (1 - p)) = b0 + b1 x1 + .... A coefficient b_j is the change in log-odds per unit of x_j; exp(b_j) is the odds ratio.

  • Decision: predict class 1 when p >= threshold (0.5 by default; tune it for the cost of errors).

  • Multi-class: multinomial (softmax) logistic regression, the default in scikit-learn for more than two classes.

  • Regularization: L2 by default (C is the inverse strength); L1 for sparse models.

When to Use

Good fit Less suitable
Need interpretability and odds ratios Strongly non-linear relationships (without feature engineering)
Need calibrated probabilities Raw images, audio, text without good features
Many samples, many sparse features (text) Complex feature interactions
Baseline for any classification task

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2,
                                                    stratify=data.target, random_state=42)

model = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))
print(classification_report(y_test, model.predict(X_test), target_names=data.target_names))

coefs = model[-1].coef_[0]
for i in np.argsort(-np.abs(coefs))[:5]:
    print(f"{data.feature_names[i]:22s} odds ratio per 1 SD: {np.exp(coefs[i]):.2f}")

Because the features are standardized, each odds ratio is the multiplicative change in the odds of the benign class for a one-standard-deviation increase in that feature.

Common Pitfalls

  • Unscaled features with regularization, which penalizes features unequally and slows convergence.

  • Perfect separation, which drives coefficients toward infinity without regularization.

  • Reading coefficients of correlated features as independent effects.

  • Using 0.5 as the threshold when classes are imbalanced or error costs differ.

14

How does ridge regression prevent overfitting?

Ridge regression prevents overfitting by adding an L2 penalty, alpha times the sum of squared coefficients, to the least-squares objective. The penalty shrinks all coefficients toward zero, which reduces the model’s sensitivity to noise and to small changes in the training data (lower variance) at the cost of a small bias. It is especially effective when there are many features, features are correlated (multicollinearity), or the number of features approaches the number of samples, where ordinary least squares becomes unstable.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

How It Works

  • Objective: minimize the sum of squared errors + alpha × sum of squared weights.

  • Closed form: w = (XᵀX + alpha I)⁻¹ Xᵀy. Adding alpha to the diagonal makes the matrix well-conditioned even when features are correlated or p > n.

  • Effect: directions in feature space with little information (small singular values) are shrunk the most, so noise-driven coefficients shrink while strong signals survive.

  • alpha = 0 gives ordinary least squares; very large alpha drives all coefficients toward zero (underfitting).

Ridge Versus Lasso

Aspect Ridge (L2) Lasso (L1)
Coefficients Shrunk, rarely exactly zero Some exactly zero
Correlated features Spreads weight among them Picks one, somewhat arbitrarily
Solution Closed form Iterative
Use Many small effects, multicollinearity Feature selection, sparse truth

Code Example

import numpy as np
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n, p = 60, 40
base = rng.normal(size=(n, 5))
X = np.hstack([base + rng.normal(0, 0.05, (n, 5)) for _ in range(8)])     # 40 highly correlated features
y = base @ np.array([3.0, -2.0, 1.0, 0.5, 0.0]) + rng.normal(0, 1.0, n)

ols = make_pipeline(StandardScaler(), LinearRegression())
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 25)))

for name, model in [("OLS", ols), ("Ridge", ridge)]:
    rmse = -cross_val_score(model, X, y, cv=5, scoring="neg_root_mean_squared_error").mean()
    model.fit(X, y)
    print(f"{name:5s} CV RMSE {rmse:.3f} | largest |coef| {np.abs(model[-1].coef_).max():.2f}")
print("chosen alpha:", round(ridge[-1].alpha_, 4))

With 40 near-duplicate features and 60 samples, OLS coefficients become large and unstable; ridge keeps them small and predicts better.

Common Pitfalls

  • Not standardizing features, so the penalty affects features unequally.

  • Choosing alpha on the test set; use RidgeCV or cross-validation.

  • Expecting ridge to remove features; use lasso or elastic net for sparsity.

15

What is Lasso regression, and what is its unique property?

Lasso regression (Least Absolute Shrinkage and Selection Operator) is linear regression with an L1 penalty: it minimizes squared error plus alpha times the sum of the absolute values of the coefficients. Its unique property is that it sets some coefficients exactly to zero, performing automatic feature selection and producing sparse, more interpretable models. Ridge regression, by contrast, shrinks coefficients but almost never makes them exactly zero.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Why Coefficients Become Exactly Zero

Geometrically, the L1 constraint region is a diamond (in 2D) with corners on the axes; the least-squares contours usually touch it at a corner, where one coefficient is zero. Algebraically, the absolute-value penalty has a constant slope, so a coefficient is set to zero whenever the feature’s correlation with the residual is weaker than alpha (the “soft-thresholding” rule in coordinate descent).

Properties

Property Detail
Sparsity Drops irrelevant features
Correlated features Tends to keep one and drop the rest; selection can be unstable
p > n Selects at most n features
Tuning alpha via cross-validation (LassoCV)
Bias Shrinks kept coefficients too; refit OLS on selected features if unbiased estimates matter
Alternative Elastic net mixes L1 and L2 for correlated features

Code Example

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LassoCV, Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y, true_coef = make_regression(n_samples=200, n_features=50, n_informative=6, noise=10,
                                  coef=True, random_state=42)

lasso = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=42)).fit(X, y)
ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0)).fit(X, y)

lasso_coef, ridge_coef = lasso[-1].coef_, ridge[-1].coef_
print("chosen alpha:", round(lasso[-1].alpha_, 3))
print("non-zero coefficients — lasso:", int(np.sum(lasso_coef != 0)), "| ridge:", int(np.sum(ridge_coef != 0)))
print("truly informative features:", sorted(np.flatnonzero(true_coef)))
print("features kept by lasso:     ", sorted(np.flatnonzero(lasso_coef)))

Lasso should keep the 6 informative features and a few or none of the others, while ridge leaves all 50 non-zero.

Common Pitfalls

  • Forgetting to standardize features, so the penalty favours large-scale features.

  • Treating the selected set as definitive when features are correlated; check stability with bootstrap resampling.

  • Using lasso when many small effects all matter; ridge often predicts better there.

16

What is the principle behind the Support Vector Machine (SVM)?

The principle of the SVM is maximum-margin separation: among all boundaries that separate two classes, choose the one that leaves the widest gap (margin) to the nearest training points, because a wide margin tends to generalize better. The nearest points, the support vectors, alone define the boundary. A soft margin with penalty C tolerates some violations for noisy data, and the kernel trick lets the SVM draw non-linear boundaries by computing similarities in an implicit higher-dimensional space.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Elements

Element Meaning
Hyperplane Boundary w·x + b = 0

| Margin | Width 2 / ||w|| between the classes’ closest points | | Objective | Minimize (1/2)||w||^2 + C × total slack (soft margin) | | Support vectors | Points on or inside the margin; the only ones that matter | | C | Large C: fewer violations, narrower margin; small C: wider margin, more tolerance | | Kernel | Linear, polynomial, RBF (gamma controls reach), sigmoid | | Hinge loss | Equivalent view: L2-regularized hinge loss max(0, 1 - y f(x)) |

Code Example

from sklearn.datasets import make_moons
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=500, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
                    {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.1, 0.5, 1, 5]}, cv=5)
grid.fit(X_train, y_train)
svc = grid.best_estimator_[-1]
print("best:", grid.best_params_)
print("support vectors:", len(svc.support_), "of", len(X_train))
print("test accuracy:", round(grid.score(X_test, y_test), 3))

When to Use SVMs

  • Small to medium datasets (up to tens of thousands of rows), often high-dimensional.

  • Text classification with a linear kernel.

  • When a strong, well-regularized classifier is needed without much feature engineering.

For large data, use LinearSVC or SGDClassifier(loss="hinge"), because kernel SVM training scales between O(n²) and O(n³).

Common Pitfalls

  • Forgetting feature scaling.

  • Tuning C and gamma separately instead of together.

  • Treating decision_function outputs as probabilities.

17

How do you handle categorical variables in supervised learning models?

Categorical variables take values from a fixed set of categories (city, product type, plan), and most models need them converted to numbers in a way that does not invent false relationships. The standard choices are one-hot encoding for nominal categories with few levels, ordinal encoding for ordered categories (and for tree models), target encoding for high-cardinality features, and native categorical support in gradient-boosting libraries. Encoders must be fitted on training data only and must handle categories that appear later.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Encoders

Encoder Output Best for Watch out for
One-hot One 0/1 column per category Linear models, SVM, k-NN, neural nets Many columns for high cardinality
Ordinal One integer column Ordered categories; tree models Implies order for linear models
Target encoding Mean target per category (cross-fitted) High cardinality (zip codes, IDs) Leakage if not cross-fitted
Frequency / count Category frequency Quick high-cardinality signal Different categories can collide
Hashing Fixed number of columns Huge, open-ended vocabularies Collisions, no inverse
Embeddings Learned dense vectors Neural networks, many categories Needs enough data
Native support Model handles categories HistGradientBoosting, LightGBM, CatBoost Library-specific

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, TargetEncoder

rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
    "plan": rng.choice(["basic", "plus", "pro"], n),
    "size": rng.choice(["S", "M", "L"], n),
    "zip": rng.choice([f"z{i}" for i in range(200)], n),
    "tenure": rng.integers(1, 72, n),
})
zip_risk = {f"z{i}": rng.normal(0, 1) for i in range(200)}
logit = (-1 + 1.2 * (df["plan"] == "basic") + 0.5 * df["size"].map({"S": 0, "M": 1, "L": 2})
         + df["zip"].map(zip_risk) - 0.03 * df["tenure"])
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

linear = Pipeline([
    ("prep", ColumnTransformer([
        ("onehot", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
        ("ordinal", OrdinalEncoder(categories=[["S", "M", "L"]]), ["size"]),
        ("target", TargetEncoder(random_state=42), ["zip"]),        # cross-fitted internally
        ("num", StandardScaler(), ["tenure"]),
    ])),
    ("model", LogisticRegression(max_iter=1000)),
])
boosted = Pipeline([
    ("prep", ColumnTransformer([("cat", OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
                                 ["plan", "size", "zip"])], remainder="passthrough")),
    ("model", HistGradientBoostingClassifier(categorical_features=[0, 1, 2], random_state=42)),
])
for name, model in [("logistic + encoders", linear), ("boosting, native categories", boosted)]:
    print(f"{name:28s} CV ROC AUC {cross_val_score(model, df, y, cv=5, scoring='roc_auc').mean():.3f}")

Native categorical support in HistGradientBoostingClassifier allows at most 255 categories per feature (the max_bins limit); for more levels, group rare categories or target-encode the column.

Common Pitfalls

  • Label-encoding nominal categories for linear models, which invents an order.

  • Fitting encoders (especially target encoders) on all data before splitting.

  • Not handling unseen categories at prediction time.

18

What are ensemble methods, and how do they improve model performance?

Ensemble methods combine the predictions of several models to get a result that is more accurate and more stable than any single model. They work because different models make different errors: averaging reduces random errors (variance), and sequentially correcting mistakes reduces systematic errors (bias). The main families are bagging (random forests), boosting (AdaBoost, gradient boosting such as XGBoost and LightGBM), and stacking or voting, which combine different model types.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Families

Family How Reduces Examples
Bagging Train models on bootstrap samples in parallel; average or vote Variance BaggingClassifier, RandomForest, ExtraTrees
Boosting Train models sequentially; each focuses on previous errors Bias (and variance with shrinkage) AdaBoost, gradient boosting, XGBoost, LightGBM, CatBoost
Voting Combine different models’ votes or probabilities Variance VotingClassifier
Stacking Meta-model learns how to combine base models’ out-of-fold predictions Both StackingClassifier

Why Averaging Helps

If M models have errors with variance σ² and pairwise correlation ρ, the average has variance roughly ρσ² + (1 - ρ)σ²/M. Uncorrelated errors shrink as M grows; correlated errors don’t. That is why random forests decorrelate trees by sampling features at each split.

Code Example

from sklearn.datasets import make_classification
from sklearn.ensemble import (HistGradientBoostingClassifier, RandomForestClassifier, StackingClassifier,
                              VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=2000, n_features=25, n_informative=10, flip_y=0.05, random_state=42)

lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
rf = RandomForestClassifier(n_estimators=300, random_state=42)
gb = HistGradientBoostingClassifier(random_state=42)

models = {
    "single decision tree": DecisionTreeClassifier(random_state=42),
    "logistic regression": lr,
    "random forest (bagging)": rf,
    "gradient boosting": gb,
    "soft voting": VotingClassifier([("lr", lr), ("rf", rf), ("gb", gb)], voting="soft"),
    "stacking": StackingClassifier([("lr", lr), ("rf", rf), ("gb", gb)], final_estimator=LogisticRegression()),
}
for name, model in models.items():
    print(f"{name:24s} CV ROC AUC {cross_val_score(model, X, y, cv=5, scoring='roc_auc', n_jobs=-1).mean():.3f}")

Trade-Offs

  • Better accuracy and robustness, but more compute, memory and latency.

  • Harder to interpret than a single tree or linear model (use feature importance or SHAP).

  • Gains are largest when base models are good and diverse.

Common Pitfalls

  • Ensembling near-identical models, which adds cost without benefit.

  • Stacking with in-sample predictions, which leaks labels into the meta-model.

19

What is the difference between bagging and boosting?

Bagging (bootstrap aggregating) trains many models independently and in parallel on random bootstrap samples of the data and averages their predictions, which mainly reduces variance; random forests are the best-known example. Boosting trains models sequentially, each one focusing on the errors of the ensemble so far, and combines them in a weighted sum, which mainly reduces bias; AdaBoost and gradient boosting (XGBoost, LightGBM, CatBoost) are examples. Bagging suits high-variance learners such as deep trees, while boosting turns weak learners such as shallow trees into a strong model.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Aspect Bagging Boosting
Training Parallel, independent Sequential, dependent
Data per model Bootstrap sample Full data, reweighted (AdaBoost) or fit to residuals/gradients (gradient boosting)
Base learners Strong, high-variance (deep trees) Weak, high-bias (shallow trees)
Combination Simple average / majority vote Weighted sum
Main effect Lower variance Lower bias
Overfitting risk Low; more trees don’t hurt Can overfit with too many rounds; use learning rate and early stopping
Noise / outliers Robust More sensitive (especially AdaBoost)
Tuning effort Low Higher (learning rate, depth, rounds)
Examples RandomForest, BaggingClassifier, ExtraTrees AdaBoost, GradientBoosting, HistGradientBoosting, XGBoost

Code Example

from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import cross_validate
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)

models = {
    "deep tree (high variance)": DecisionTreeClassifier(random_state=42),
    "bagging deep trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=200, n_jobs=-1, random_state=42),
    "stump (high bias)": DecisionTreeClassifier(max_depth=1),
    "AdaBoost on stumps": AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=300,
                                             random_state=42),
    "gradient boosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
    res = cross_validate(model, X, y, cv=5, return_train_score=True, n_jobs=-1)
    print(f"{name:26s} train {res['train_score'].mean():.3f}  val {res['test_score'].mean():.3f}")

Bagging should keep the deep tree’s high training score while raising its validation score (less variance); boosting should lift the stump’s low scores on both sets (less bias).

Interview Tips

  • Summarize as “bagging: parallel, variance; boosting: sequential, bias”.

  • Mention that random forests add random feature selection at each split to decorrelate trees beyond plain bagging.

20

What is feature scaling, and why is it important?

Feature scaling transforms numeric features to comparable ranges, for example standardizing each to mean 0 and standard deviation 1, or rescaling to [0, 1]. It matters for any model that uses distances, dot products, gradients or penalties on weights: k-NN, SVMs, k-means, PCA, linear and logistic regression with regularization, and neural networks. Without scaling, features with large numeric ranges dominate distances and penalties, and gradient descent converges slowly. Tree-based models split on one feature at a time and do not need scaling.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Scalers

Scaler Transformation Use when
StandardScaler (x − mean) / std Default for most models
MinMaxScaler (x − min) / (max − min) Bounded inputs, neural nets, images
RobustScaler (x − median) / IQR Outliers present
MaxAbsScaler x / max( x ) Sparse data (keeps zeros)
PowerTransformer / log Makes skewed data more Gaussian Heavily skewed features
Normalizer Each row to unit length Text vectors, cosine similarity

Which Models Need It

Needs scaling Doesn’t
k-NN, k-means, SVM, PCA Decision trees
Regularized linear/logistic regression Random forests
Neural networks, gradient-descent models Gradient-boosted trees

Code Example

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)          # feature ranges differ by about three orders of magnitude

for name, model in [("k-NN", KNeighborsClassifier()),
                    ("logistic regression", LogisticRegression(max_iter=5000)),
                    ("random forest", RandomForestClassifier(random_state=42))]:
    raw = cross_val_score(model, X, y, cv=5).mean()
    scaled = cross_val_score(make_pipeline(StandardScaler(), model), X, y, cv=5).mean()
    print(f"{name:20s} unscaled {raw:.3f} | scaled {scaled:.3f}")

k-NN should improve the most, the logistic regression should converge more easily and score higher, and the random forest should be essentially unchanged.

Common Pitfalls

  • Fitting the scaler on the full dataset before the split; put it in a Pipeline.

  • Scaling the target unintentionally or forgetting to invert a target transform.

  • Applying StandardScaler to sparse matrices, which destroys sparsity (use with_mean=False or MaxAbsScaler).

21

How is a decision tree constructed?

A decision tree is built top-down by recursive partitioning: starting with all training data at the root, the algorithm tries candidate splits (a feature and a threshold, or a category grouping), picks the split that makes the child nodes as pure as possible according to a criterion such as Gini impurity, entropy or squared error, and repeats on each child. Splitting stops when a node is pure, too small, or a depth limit is reached; leaves predict the majority class (classification) or the mean target (regression). This greedy procedure, used by CART in scikit-learn, finds good but not globally optimal trees.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Algorithm (CART)

  1. For each feature, sort the values and consider thresholds between consecutive distinct values.

  2. For each threshold, compute the weighted impurity of the two children: (n_left/n) * impurity_left + (n_right/n) * impurity_right.

  3. Choose the split with the largest impurity decrease.

  4. Recurse on both children.

  5. Stop when a stopping rule triggers: max_depth, min_samples_split, min_samples_leaf, min_impurity_decrease, or a pure node.

  6. Optionally prune afterwards (cost-complexity pruning with ccp_alpha).

Criteria

Task Criterion Idea
Classification Gini Probability of misclassifying a random sample labelled by the node’s class distribution
Classification Entropy / log loss Information content of the class distribution
Regression Squared error Variance of targets in the node
Regression Absolute error, Poisson Robust or count targets

Code Example: One Split by Hand vs scikit-learn

import numpy as np
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

X, y = load_iris(return_X_y=True)

def gini(labels):
    _, counts = np.unique(labels, return_counts=True)
    p = counts / counts.sum()
    return 1 - np.sum(p ** 2)

best = (None, None, np.inf)
for f in range(X.shape[1]):
    values = np.unique(X[:, f])
    for t in (values[:-1] + values[1:]) / 2:                 # midpoints between distinct values
        left, right = y[X[:, f] <= t], y[X[:, f] > t]
        score = (len(left) * gini(left) + len(right) * gini(right)) / len(y)
        if score < best[2]:
            best = (f, t, score)
print(f"best root split: feature {best[0]} <= {best[1]:.2f}, weighted Gini {best[2]:.3f}")

tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, y)
print(export_text(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"]))

The hand-computed root split separates setosa perfectly; scikit-learn finds an equally good split (petal length and petal width tie here, so the reported feature may differ).

Complexity

Training costs roughly O(n_features × n log n) per level for sorting-based splits; prediction costs O(depth).

Common Pitfalls

  • Growing trees without limits, which memorizes the training data.

  • Expecting the greedy tree to be optimal; small data changes can produce a very different tree (high variance).

  • Using a single tree when a random forest or gradient boosting would be far more accurate.

22

What are the pros and cons of using decision trees?

Decision trees are easy to understand and need little data preparation: they handle numeric and categorical features, ignore feature scaling, capture non-linear relationships and interactions, and can be drawn as a set of if-then rules. Their major weaknesses are high variance (small changes in the data can change the tree a lot), a strong tendency to overfit without depth limits or pruning, step-shaped predictions that extrapolate poorly, and lower accuracy than ensembles of trees such as random forests and gradient boosting.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Pros and Cons

Pros Cons
Interpretable rules; easy to explain to non-experts Overfit easily if grown deep
No scaling needed; handle mixed feature types Unstable: small data changes alter the structure
Capture non-linearity and interactions automatically Axis-aligned splits approximate diagonal boundaries poorly
Fast to train and predict Piecewise-constant predictions; cannot extrapolate trends
Implicit feature selection Impurity-based importance biased toward high-cardinality features
Handle missing values (scikit-learn 1.3+ supports NaN in trees) Greedy: not globally optimal
Building block for strong ensembles Usually less accurate than ensembles

Code Example: Instability and the Ensemble Fix

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, flip_y=0.05, random_state=42)

# Instability: train on two bootstrap samples and compare the root splits
rng = np.random.default_rng(42)
for i in range(3):
    idx = rng.choice(len(X), len(X), replace=True)
    t = DecisionTreeClassifier(random_state=42).fit(X[idx], y[idx])
    print(f"sample {i}: root split on feature {t.tree_.feature[0]} at {t.tree_.threshold[0]:.2f}, "
          f"depth {t.get_depth()}, leaves {t.get_n_leaves()}")

for name, model in [("unpruned tree", DecisionTreeClassifier(random_state=42)),
                    ("tree, max_depth=5", DecisionTreeClassifier(max_depth=5, random_state=42)),
                    ("random forest", RandomForestClassifier(n_estimators=300, random_state=42))]:
    print(f"{name:18s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

When to Use a Single Tree

  • When interpretability matters more than the last few points of accuracy (e.g. policy rules, simple triage).

  • As a quick baseline or to explore interactions in the data.

  • Otherwise, prefer random forests or gradient boosting.

Common Pitfalls

  • Reporting a deep tree’s training accuracy.

  • Over-trusting impurity-based feature importance; check permutation importance.

23

What are Gini impurity and information gain?

Gini impurity and information gain are measures decision trees use to choose splits. Gini impurity is the probability of misclassifying a randomly chosen sample if it were labelled at random according to the node’s class distribution: 1 − sum of p_k². Entropy measures the uncertainty of the class distribution, −sum p_k log2 p_k, and information gain is the reduction in entropy from parent to children, weighted by child size. A good split produces purer children, meaning lower weighted impurity or higher information gain.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (Python 3.12)

Formulas in Words

Measure Definition Range (2 classes) Pure node
Gini impurity 1 minus the sum of squared class proportions 0 to 0.5 0
Entropy Minus the sum of p × log2(p) over classes 0 to 1 bit 0
Information gain Parent entropy minus weighted average child entropy ≥ 0 —
Gini decrease Parent Gini minus weighted average child Gini ≥ 0 —

Worked Example

A parent node has 10 samples: 5 positive, 5 negative. - Parent Gini = 1 − (0.5² + 0.5²) = 0.5; parent entropy = 1 bit. - Split into left (4 positive, 1 negative) and right (1 positive, 4 negative). - Each child: Gini = 1 − (0.8² + 0.2²) = 0.32; entropy = −(0.8 log2 0.8 + 0.2 log2 0.2) ≈ 0.722. - Weighted children: Gini 0.32, entropy 0.722. - Gini decrease = 0.5 − 0.32 = 0.18; information gain = 1 − 0.722 = 0.278 bits.

Code Example

import numpy as np

def gini(counts):
    p = np.asarray(counts) / np.sum(counts)
    return 1 - np.sum(p ** 2)

def entropy(counts):
    p = np.asarray(counts) / np.sum(counts)
    p = p[p > 0]
    return -np.sum(p * np.log2(p))

def split_gain(parent, children, measure):
    n = sum(sum(c) for c in children)
    weighted = sum(sum(c) / n * measure(c) for c in children)
    return measure(parent) - weighted

parent = [5, 5]
children = [[4, 1], [1, 4]]
print("Gini decrease:", round(split_gain(parent, children, gini), 3))        # 0.18
print("Information gain:", round(split_gain(parent, children, entropy), 3))  # 0.278

perfect = [[5, 0], [0, 5]]
print("perfect split gains:", split_gain(parent, perfect, gini), round(split_gain(parent, perfect, entropy), 3))

Gini Versus Entropy

  • They usually pick the same or very similar splits.

  • Gini avoids the logarithm and is slightly faster; it is scikit-learn’s default (criterion="gini").

  • Entropy is a bit more sensitive to changes in small class probabilities.

Common Pitfalls

  • Forgetting to weight child impurities by their sample counts.

  • Information gain favours features with many unique values (IDs); C4.5 uses the gain ratio to correct this.

24

How does random forest improve on basic decision trees?

A random forest improves on a single decision tree by training many deep trees on different bootstrap samples of the data and, at each split, considering only a random subset of features, then averaging their predictions (or taking a majority vote). Individual deep trees have low bias but high variance; averaging many decorrelated trees keeps the low bias while sharply reducing variance. The result is much higher accuracy, robustness to noise and outliers, and a built-in error estimate from out-of-bag samples, at the cost of interpretability and some speed.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

What Changes

Aspect Single decision tree Random forest
Training data All rows Bootstrap sample per tree
Features per split All Random subset (max_features, e.g. sqrt(p) for classification)
Prediction One tree’s leaf Average / vote of hundreds of trees
Variance High Low
Overfitting Strong without pruning Limited; more trees don’t overfit
Validation Needs CV Out-of-bag (OOB) estimate available
Interpretability Readable rules Needs feature importance / SHAP
Cost Fast Roughly n_trees × cost (parallelizable)

Why Feature Subsampling Matters

If one feature is very strong, every bagged tree would split on it first, making the trees similar and their errors correlated. Restricting each split to a random subset forces diversity, and averaging diverse trees cancels more error.

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

tree = DecisionTreeClassifier(random_state=42)
forest = RandomForestClassifier(n_estimators=500, max_features="sqrt", oob_score=True, n_jobs=-1, random_state=42)

for name, model in [("decision tree", tree), ("random forest", forest)]:
    cv = cross_val_score(model, X_train, y_train, cv=5).mean()
    model.fit(X_train, y_train)
    print(f"{name:14s} CV {cv:.3f} | train {model.score(X_train, y_train):.3f} | test {model.score(X_test, y_test):.3f}")
print("random forest OOB accuracy:", round(forest.oob_score_, 3))

for n in (1, 10, 50, 200, 500):
    rf = RandomForestClassifier(n_estimators=n, n_jobs=-1, random_state=42)
    print(f"{n:4d} trees: CV accuracy {cross_val_score(rf, X_train, y_train, cv=5).mean():.3f}")

Accuracy rises quickly with the first few dozen trees and then plateaus; adding trees costs time but does not cause overfitting.

Common Pitfalls

  • Tuning n_estimators as if it controlled overfitting; tune max_features, min_samples_leaf and max_depth instead.

  • Trusting impurity-based importances with high-cardinality features; use permutation importance.

  • Expecting random forests to extrapolate beyond the training range in regression.

25

How is feature importance determined in decision trees and random forests?

Feature importance in tree models measures how much each feature contributes to the model’s predictions. The built-in impurity-based (mean decrease in impurity, MDI) importance sums, over every split that uses a feature, the reduction in impurity weighted by the number of samples reaching that split, then averages across trees and normalizes to sum to 1. A more reliable alternative is permutation importance: shuffle one feature’s values in held-out data and measure how much the model’s score drops. SHAP values add per-prediction attributions.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Methods

Method How Pros Cons
Impurity-based (MDI), feature_importances_ Sum of weighted impurity decreases per feature Free, fast Computed on training data; biased toward high-cardinality and continuous features
Permutation importance Score drop after shuffling a feature on validation data Model-agnostic, measured on held-out data Slower; correlated features share credit and can look unimportant
Drop-column importance Retrain without the feature Direct Very expensive
SHAP (TreeSHAP) Game-theoretic attribution per prediction Local and global explanations Extra library, more computation

Code Example: MDI vs Permutation Importance

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

data = load_breast_cancer(as_frame=True)
X = data.data.copy()
rng = np.random.default_rng(42)
X["random_id"] = rng.integers(0, 10000, len(X))            # pure noise with many unique values
X_train, X_test, y_train, y_test = train_test_split(X, data.target, test_size=0.3, stratify=data.target,
                                                    random_state=42)

rf = RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1).fit(X_train, y_train)
mdi = pd.Series(rf.feature_importances_, index=X.columns)
perm = permutation_importance(rf, X_test, y_test, n_repeats=20, random_state=42, n_jobs=-1)
perm_mean = pd.Series(perm.importances_mean, index=X.columns)

print("Top 5 by MDI:\n", mdi.sort_values(ascending=False).head(5).round(3))
print("Top 5 by permutation:\n", perm_mean.sort_values(ascending=False).head(5).round(3))
print(f"random_id: MDI rank {mdi.rank(ascending=False)['random_id']:.0f} of {len(mdi)}, "
      f"permutation importance {perm_mean['random_id']:.4f}")

The noise column typically receives a non-trivial MDI score because it offers many split points, while its permutation importance is near zero, which illustrates the MDI bias.

Interpreting Results

  • Importance shows association with predictions, not causation.

  • With correlated features, importance is split or masked; group them or use conditional methods.

  • Report variability (standard deviation across repeats or folds).

Common Pitfalls

  • Ranking features by MDI alone in the presence of ID-like or high-cardinality columns.

  • Computing permutation importance on training data, which reflects memorization.

26

What are the basic components of a neural network?

A neural network is built from layers of simple units (neurons), each computing a weighted sum of its inputs plus a bias and passing it through a non-linear activation function. Layers are stacked from an input layer, through one or more hidden layers, to an output layer sized for the task. Training needs a loss function to measure errors, backpropagation to compute gradients, and an optimizer to update the weights, plus regularization and hyperparameters such as the learning rate and batch size.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

Components

Component Role Examples
Inputs Feature vector, image, token IDs Scaled numeric features
Weights and biases Learnable parameters Matrix W, vector b per layer
Neuron / layer Computes activation(W·x + b) Dense (linear), convolutional, recurrent, attention
Activation function Adds non-linearity ReLU, GELU, sigmoid, tanh, softmax
Output layer Matches the task 1 unit (regression), sigmoid (binary), softmax over K classes
Loss function Measures prediction error MSE, cross-entropy
Backpropagation Computes gradients of the loss with respect to all weights Automatic differentiation
Optimizer Updates weights from gradients SGD with momentum, Adam, AdamW
Regularization Limits overfitting Weight decay, dropout, early stopping
Hyperparameters Chosen by the practitioner Learning rate, batch size, epochs, layer sizes

Code Example

import torch
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn

torch.manual_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = (torch.tensor(scaler.transform(a), dtype=torch.float32) for a in (X_train, X_test))
y_train, y_test = (torch.tensor(a, dtype=torch.float32) for a in (y_train, y_test))

model = nn.Sequential(                 # input (30) -> hidden (16) -> hidden (8) -> output (1 logit)
    nn.Linear(30, 16), nn.ReLU(),
    nn.Linear(16, 8), nn.ReLU(),
    nn.Linear(8, 1),
)
loss_fn = nn.BCEWithLogitsLoss()       # sigmoid + binary cross-entropy, numerically stable
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)

for epoch in range(200):
    optimizer.zero_grad()
    loss = loss_fn(model(X_train).squeeze(1), y_train)   # forward pass + loss
    loss.backward()                                      # backpropagation
    optimizer.step()                                     # weight update

with torch.no_grad():
    acc = ((model(X_test).squeeze(1) > 0).float() == y_test).float().mean()
print(f"final training loss {loss.item():.4f}, test accuracy {acc.item():.3f}")
print("parameters:", sum(p.numel() for p in model.parameters()))

The parameter count is 30×16+16 + 16×8+8 + 8×1+1 = 641.

Common Pitfalls

  • No activation between linear layers, which collapses the network into a single linear model.

  • An output activation that does not match the loss (e.g. sigmoid before BCEWithLogitsLoss).

  • Unscaled inputs, which slow or destabilize training.

27

What role do activation functions play in neural networks?

Activation functions apply a non-linear transformation to each neuron’s weighted sum, which is what allows a neural network to model non-linear relationships. Without them, any stack of linear layers collapses into a single linear function, no matter how deep. The choice of activation also affects how well gradients flow during training: ReLU-family functions avoid the vanishing gradients that saturating functions like sigmoid and tanh cause in deep networks, and output activations (sigmoid, softmax) turn scores into probabilities.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

Common Activations

Function Formula in words Range Use Issues
ReLU max(0, x) [0, ∞) Default for hidden layers “Dead” neurons stuck at 0
Leaky ReLU x if x > 0, else a small slope × x (−∞, ∞) Avoids dead neurons Extra hyperparameter
GELU x × probability a standard normal is below x ≈(−0.17, ∞) Transformers Slightly costlier
SiLU / Swish x × sigmoid(x) ≈(−0.28, ∞) CNNs, modern MLPs Slightly costlier
Sigmoid 1 / (1 + e^(−x)) (0, 1) Binary output, gates in LSTMs Saturates: vanishing gradients
Tanh (e^x − e^(−x)) / (e^x + e^(−x)) (−1, 1) RNN hidden states Saturates
Softmax e^(x_k) / sum of e^(x_j) (0, 1), sums to 1 Multi-class output Use with cross-entropy
Linear (none) x (−∞, ∞) Regression output —

Code Example: Why Non-Linearity Matters and Gradient Flow

import torch
from torch import nn

torch.manual_seed(42)

# 1. Without activations, two linear layers equal one linear layer
lin = nn.Sequential(nn.Linear(4, 8), nn.Linear(8, 2))
W = lin[1].weight @ lin[0].weight
b = lin[1].weight @ lin[0].bias + lin[1].bias
x = torch.randn(5, 4)
print("collapses to one linear map:", torch.allclose(lin(x), x @ W.T + b, atol=1e-6))

# 2. Gradient size reaching the first layer of a deep network
def first_layer_grad(activation, depth=20):
    layers = []
    for _ in range(depth):
        layers += [nn.Linear(64, 64), activation()]
    net = nn.Sequential(*layers, nn.Linear(64, 1))
    net(torch.randn(32, 64)).sum().backward()
    return net[0].weight.grad.abs().mean().item()

for act in (nn.Sigmoid, nn.Tanh, nn.ReLU, nn.GELU):
    print(f"{act.__name__:8s} mean |grad| at layer 1: {first_layer_grad(act):.2e}")

With sigmoid, the gradient reaching the first of 20 layers is typically orders of magnitude smaller than with ReLU or GELU, because each sigmoid layer multiplies the gradient by at most 0.25.

Choosing

  • Hidden layers: ReLU (default), GELU or SiLU for transformers and modern architectures.

  • Output: none for regression, sigmoid for binary or multi-label, softmax for multi-class (usually folded into the loss).

Common Pitfalls

  • Applying softmax before nn.CrossEntropyLoss, which already includes it.

  • Sigmoid or tanh in deep hidden stacks without normalization or residual connections.

28

How do you prevent overfitting in neural networks?

Neural networks overfit easily because they have many parameters and can memorize training data. The main defences are more and augmented data, early stopping on a validation set, weight decay (L2 regularization), dropout, normalization layers, smaller or pretrained architectures (transfer learning), and label smoothing. They are usually combined and tuned against validation performance, watching the gap between training and validation loss.

Read full answer

Applies toPyTorch 2.x, scikit-learn 1.5+ (Python 3.12)

Techniques

Technique How it helps PyTorch
More / augmented data More variety to learn from torchvision transforms, text augmentation
Early stopping Stops before memorization begins Track validation loss, keep best weights
Weight decay Keeps weights small torch.optim.AdamW(weight_decay=...)
Dropout Randomly zeroes activations, preventing co-adaptation nn.Dropout(p=0.2-0.5)
Batch / layer normalization Stabilizes training, mild regularization nn.BatchNorm1d, nn.LayerNorm
Smaller model Less capacity to memorize Fewer layers/units
Transfer learning Pretrained features need fewer labelled examples Freeze backbone, train head
Label smoothing Prevents over-confident outputs nn.CrossEntropyLoss(label_smoothing=0.1)
Ensembling Averages out individual overfitting Several seeds / checkpoints

Code Example: Dropout, Weight Decay and Early Stopping

import copy

import torch
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn

torch.manual_seed(42)
X, y = make_classification(n_samples=600, n_features=40, n_informative=8, flip_y=0.1, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_tr)
X_tr, X_val = (torch.tensor(scaler.transform(a), dtype=torch.float32) for a in (X_tr, X_val))
y_tr, y_val = (torch.tensor(a, dtype=torch.float32) for a in (y_tr, y_val))

def train(regularized, epochs=300, patience=20):
    p, wd = (0.4, 1e-3) if regularized else (0.0, 0.0)
    model = nn.Sequential(nn.Linear(40, 256), nn.ReLU(), nn.Dropout(p),
                          nn.Linear(256, 256), nn.ReLU(), nn.Dropout(p), nn.Linear(256, 1))
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=wd)
    loss_fn = nn.BCEWithLogitsLoss()
    best, best_state, wait = float("inf"), None, 0
    for epoch in range(epochs):
        model.train()
        opt.zero_grad()
        loss_fn(model(X_tr).squeeze(1), y_tr).backward()
        opt.step()
        model.eval()
        with torch.no_grad():
            val_loss = loss_fn(model(X_val).squeeze(1), y_val).item()
        if val_loss < best:
            best, best_state, wait = val_loss, copy.deepcopy(model.state_dict()), 0
        elif regularized:
            wait += 1
            if wait >= patience:
                break
    if regularized:
        model.load_state_dict(best_state)
    model.eval()
    with torch.no_grad():
        acc = lambda X_, y_: ((model(X_).squeeze(1) > 0).float() == y_).float().mean().item()
        return acc(X_tr, y_tr), acc(X_val, y_val), epoch + 1

for regularized in (False, True):
    tr, va, ep = train(regularized)
    print(f"regularized={regularized}: train acc {tr:.3f}, val acc {va:.3f}, epochs {ep}")

For a final, unbiased number, evaluate on a separate test set; here the validation set is also used for early stopping.

Common Pitfalls

  • Forgetting model.eval() at evaluation time, which leaves dropout active.

  • Using Adam’s weight_decay (coupled L2) and expecting AdamW behaviour.

  • Early stopping on the test set.

29

What is backpropagation, and why is it important for training neural networks?

Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a neural network efficiently, by applying the chain rule backwards from the output layer to the input layer and reusing intermediate results. Those gradients tell the optimizer how to change each weight to reduce the loss. Its significance is efficiency: it computes all gradients in roughly the cost of one extra forward pass, which is what makes training networks with millions or billions of parameters feasible.

Read full answer

Applies toNumPy 2.x, PyTorch 2.x (Python 3.12)

Steps

  1. Forward pass: compute each layer’s output and store the intermediate values.

  2. Loss: compare the output with the target.

  3. Backward pass: starting from the derivative of the loss with respect to the output, multiply by each layer’s local derivative to get gradients for its weights and for its input, then pass the input gradient to the previous layer.

  4. Update: the optimizer (SGD, Adam) adjusts weights using the gradients.

For a layer h = activation(W x + b): the weight gradient is the incoming gradient (times the activation’s derivative) multiplied by the layer’s input; the gradient passed backwards is that same quantity multiplied by W transposed.

Code Example: Manual Backprop Matches Autograd

import numpy as np
import torch

rng = np.random.default_rng(42)
X = rng.normal(size=(8, 3))
y = rng.normal(size=(8, 1))
W1, b1 = rng.normal(size=(3, 4)), np.zeros(4)
W2, b2 = rng.normal(size=(4, 1)), np.zeros(1)

# Forward
z1 = X @ W1 + b1
h1 = np.maximum(0, z1)                       # ReLU
pred = h1 @ W2 + b2
loss = np.mean((pred - y) ** 2)

# Backward (chain rule)
d_pred = 2 * (pred - y) / len(X)             # dLoss/dpred
dW2 = h1.T @ d_pred
db2 = d_pred.sum(axis=0)
d_h1 = d_pred @ W2.T
d_z1 = d_h1 * (z1 > 0)                       # ReLU derivative
dW1 = X.T @ d_z1
db1 = d_z1.sum(axis=0)

# Same computation with PyTorch autograd
t = {k: torch.tensor(v, requires_grad=True) for k, v in dict(W1=W1, b1=b1, W2=W2, b2=b2).items()}
pred_t = torch.relu(torch.tensor(X) @ t["W1"] + t["b1"]) @ t["W2"] + t["b2"]
loss_t = torch.mean((pred_t - torch.tensor(y)) ** 2)
loss_t.backward()

print("loss equal:", np.isclose(loss, loss_t.item()))
print("dW1 equal:", np.allclose(dW1, t["W1"].grad.numpy()), "| dW2 equal:", np.allclose(dW2, t["W2"].grad.numpy()))

Related Problems

Problem Cause Fix
Vanishing gradients Repeated multiplication by small derivatives (sigmoid/tanh) ReLU/GELU, residual connections, normalization
Exploding gradients Repeated multiplication by large values (deep RNNs) Gradient clipping, careful initialization
Memory use Stored activations for the backward pass Gradient checkpointing, smaller batches

Interview Tips

  • Distinguish backpropagation (computing gradients) from gradient descent (using them).

  • Mention that modern frameworks implement it as reverse-mode automatic differentiation.

30

What is the difference between shallow and deep neural networks?

A shallow neural network has one (sometimes two) hidden layers, while a deep neural network has many hidden layers stacked between input and output. In theory, a single wide hidden layer can approximate any continuous function (the universal approximation theorem), but it may need an impractically large number of units. Deep networks learn hierarchical representations, simple features in early layers combined into complex ones later, which makes them far more efficient for images, audio and text, at the cost of more data, compute and training care.

Read full answer

Applies toscikit-learn 1.5+, PyTorch 2.x (Python 3.12)

Comparison

Aspect Shallow Deep
Hidden layers 1-2 Many (tens to hundreds)
Representation Flat Hierarchical (edges → shapes → objects)
Parameter efficiency May need very wide layers Fewer parameters for complex functions
Data needs Small to medium Large (or pretrained models)
Training difficulty Easy Vanishing/exploding gradients; needs ReLU, normalization, residual connections
Compute Low High (GPUs)
Typical use Tabular data, simple regression/classification Vision, speech, language, generative models

Code Example: Shallow vs Deep on a Simple Problem

from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=1500, noise=0.2, random_state=42)

def n_params(sizes, n_in=2, n_out=1):
    dims = [n_in, *sizes, n_out]
    return sum(a * b + b for a, b in zip(dims[:-1], dims[1:]))

configs = {"shallow (1 x 64)": (64,), "deep (4 x 16)": (16, 16, 16, 16)}
for name, sizes in configs.items():
    model = make_pipeline(StandardScaler(),
                          MLPClassifier(hidden_layer_sizes=sizes, max_iter=2000, random_state=42))
    acc = cross_val_score(model, X, y, cv=5).mean()
    print(f"{name:17s} parameters {n_params(sizes):4d}  CV accuracy {acc:.3f}")

On a simple 2D problem both do well; the advantage of depth appears with structured, high-dimensional data such as images, where convolutional deep networks greatly outperform shallow ones.

When to Choose Which

  • Tabular data with few thousand rows: shallow networks, or more often gradient-boosted trees.

  • Images, audio, text, video: deep architectures (CNNs, transformers), usually pretrained.

Common Pitfalls

  • Making a network deeper without residual connections or normalization, which can make training worse.

  • Using deep networks on small tabular datasets where simpler models win.

31

What are the confusion matrix, precision, recall and F1 score?

A confusion matrix tabulates a classifier’s predictions against the true labels: for binary problems it counts true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is the share of predicted positives that are correct, TP / (TP + FP). Recall (sensitivity) is the share of actual positives that are found, TP / (TP + FN). F1 is the harmonic mean of precision and recall, 2PR / (P + R), which is high only when both are high. Together they show which errors a model makes, which accuracy alone hides.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Binary Confusion Matrix

Predicted negative Predicted positive
Actual negative TN FP (type I error)
Actual positive FN (type II error) TP

When Each Matters

Metric Prioritize when Example
Precision False positives are costly Spam filter (don’t lose real email), fraud alerts reviewed by people
Recall False negatives are costly Cancer screening, safety defects
F1 Need balance, positives are rare Information extraction, imbalanced classification
F-beta Weight recall β times as much as precision F2 for screening

Worked Example

A fraud model flags 50 transactions; 40 are fraud (TP = 40, FP = 10). There were 80 frauds in total (FN = 40). - Precision = 40 / 50 = 0.80. - Recall = 40 / 80 = 0.50. - F1 = 2 × 0.80 × 0.50 / (0.80 + 0.50) ≈ 0.615.

Code Example

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (ConfusionMatrixDisplay, classification_report, confusion_matrix, f1_score,
                             precision_score, recall_score)
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, weights=[0.9, 0.1], flip_y=0.02, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)

for threshold in (0.5, 0.3):
    pred = (model.predict_proba(X_test)[:, 1] >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
    print(f"threshold {threshold}: TP={tp} FP={fp} FN={fn} TN={tn} | "
          f"precision {precision_score(y_test, pred):.3f} recall {recall_score(y_test, pred):.3f} "
          f"F1 {f1_score(y_test, pred):.3f}")

print(classification_report(y_test, model.predict(X_test), digits=3))

Lowering the threshold usually raises recall and lowers precision; the threshold should be chosen on validation data according to the costs of each error.

Multi-Class

Compute precision, recall and F1 per class, then average: macro (unweighted mean, treats classes equally), weighted (by class support), or micro (pooled counts; equals accuracy for single-label problems).

Common Pitfalls

  • Reporting only accuracy on imbalanced data.

  • Mixing up which class is “positive” (pos_label).

  • Comparing F1 scores computed at different thresholds without saying so.

32

What is accuracy, and why isn't it always the best metric?

Accuracy is the fraction of predictions that are correct: (TP + TN) / total. It is intuitive but misleading when classes are imbalanced or errors have different costs. If 99% of transactions are legitimate, a model that always predicts “legitimate” scores 99% accuracy while catching no fraud. Accuracy also ignores the confidence of predictions and depends on a single threshold, so metrics such as precision, recall, F1, ROC AUC, PR AUC and cost-based measures are often more informative.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

When Accuracy Fails

Situation Problem Better metric
Rare positive class Majority-class guessing looks excellent Recall, precision, F1, PR AUC
Unequal error costs Treats a missed cancer like a false alarm Cost-weighted error, recall at fixed precision
Probability quality matters Ignores calibration Log loss, Brier score
Ranking matters Threshold-dependent ROC AUC, PR AUC
Multi-class with small classes Dominated by large classes Macro F1, balanced accuracy

Accuracy is fine when classes are roughly balanced and all errors cost about the same.

Code Example

from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, average_precision_score, balanced_accuracy_score, f1_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10000, n_features=20, weights=[0.99, 0.01], flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

for name, model in [("always majority", DummyClassifier(strategy="most_frequent")),
                    ("random forest", RandomForestClassifier(n_estimators=300, class_weight="balanced",
                                                             random_state=42))]:
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    proba = model.predict_proba(X_test)[:, 1]
    print(f"{name:16s} accuracy {accuracy_score(y_test, pred):.3f} | balanced acc "
          f"{balanced_accuracy_score(y_test, pred):.3f} | recall {recall_score(y_test, pred, zero_division=0):.3f} | "
          f"F1 {f1_score(y_test, pred, zero_division=0):.3f} | PR AUC {average_precision_score(y_test, proba):.3f}")

The dummy model scores about 0.99 accuracy with zero recall; the other metrics expose it immediately. For PR AUC, the no-skill baseline equals the positive rate (about 0.01 here).

Balanced Accuracy

Balanced accuracy averages recall over classes, so a majority-class predictor scores 0.5 in binary problems regardless of imbalance.

Common Pitfalls

  • Celebrating high accuracy without checking the class distribution.

  • Choosing a model by accuracy when the business cares about catching the rare class.

33

What are the ROC curve and AUC?

The ROC (Receiver Operating Characteristic) curve plots a classifier’s true positive rate (recall) against its false positive rate as the decision threshold moves from high to low, showing every trade-off between catching positives and raising false alarms. AUC (area under the ROC curve) summarizes it in one number between 0 and 1: the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one. An AUC of 0.5 is random guessing and 1.0 is perfect ranking.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Key Quantities

Quantity Formula Meaning
True positive rate (TPR, recall) TP / (TP + FN) Share of positives caught
False positive rate (FPR) FP / (FP + TN) Share of negatives wrongly flagged
Point on the curve (FPR, TPR) at one threshold One operating point
AUC Area under the curve Threshold-independent ranking quality

Properties

  • Threshold-independent: evaluates ranking, not one cut-off.

  • Insensitive to class balance in its definition, which is both a strength (comparable across datasets) and a weakness: with very rare positives, a high ROC AUC can hide poor precision. Use the precision-recall curve (average precision) there.

  • Needs scores or probabilities, not hard labels.

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=3000, n_features=10, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
scores = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, scores)
print("ROC AUC:", round(roc_auc_score(y_test, scores), 3))
print("PR AUC (average precision):", round(average_precision_score(y_test, scores), 3))

# AUC as a ranking probability: compare every positive with every negative
pos, neg = scores[y_test == 1], scores[y_test == 0]
pairwise = (pos[:, None] > neg[None, :]).mean() + 0.5 * (pos[:, None] == neg[None, :]).mean()
print("P(score_pos > score_neg):", round(pairwise, 3))

# Operating point: highest TPR with FPR at most 5%
i = np.where(fpr <= 0.05)[0][-1]
print(f"at threshold {thresholds[i]:.3f}: TPR {tpr[i]:.3f}, FPR {fpr[i]:.3f}")

The pairwise probability equals the ROC AUC, which is the clearest way to explain what AUC means.

Common Pitfalls

  • Computing AUC from hard 0/1 predictions, which reduces the curve to a single point.

  • Relying on ROC AUC alone for rare-event detection.

  • Treating AUC as a measure of calibration; a model can rank well yet output badly calibrated probabilities.

34

How do RMSE and MAE compare as regression metrics?

MAE (mean absolute error) is the average size of the errors, and RMSE (root mean squared error) is the square root of the average squared error. Both are in the target’s units. The key difference is that RMSE squares errors before averaging, so it penalizes large errors much more: one big miss raises RMSE far more than MAE. RMSE ≥ MAE always, and the gap grows when errors vary a lot. Choose MAE when all errors matter in proportion to their size and outliers should not dominate; choose RMSE when large errors are especially costly.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Comparison

Aspect MAE RMSE
Formula in words Mean of absolute errors Square root of the mean of squared errors
Units Same as target Same as target
Sensitivity to outliers Linear Quadratic: large errors dominate
Optimal constant prediction Median Mean
Matching training loss Absolute error (L1) Squared error (L2)
Interpretation “Typical miss” Emphasizes worst misses
Relation MAE ≤ RMSE ≤ √n × MAE —

Worked Example

Errors [1, 1, 1, 1, 6]: MAE = 10 / 5 = 2.0; RMSE = √((1+1+1+1+36)/5) = √8 ≈ 2.83. Errors [2, 2, 2, 2, 2]: MAE = 2.0, RMSE = 2.0. Same MAE, but RMSE reveals the single large error.

Code Example

import numpy as np
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

y_true = np.zeros(5)
for name, errors in [("uniform errors", [2, 2, 2, 2, 2]), ("one big error", [1, 1, 1, 1, 6])]:
    y_pred = np.array(errors, dtype=float)
    print(f"{name:15s} MAE {mean_absolute_error(y_true, y_pred):.2f} | "
          f"RMSE {root_mean_squared_error(y_true, y_pred):.2f}")

# Optimal constant predictions: median minimizes MAE, mean minimizes RMSE
y = np.array([10, 12, 13, 14, 100], dtype=float)
for c in (np.median(y), np.mean(y)):
    pred = np.full_like(y, c)
    print(f"predict {c:5.1f}: MAE {mean_absolute_error(y, pred):6.2f}, RMSE {root_mean_squared_error(y, pred):6.2f}")

root_mean_squared_error is available from scikit-learn 1.4.

Choosing

  • Delivery-time estimates where a few very late predictions are unacceptable: RMSE.

  • Demand forecasts where typical error matters and occasional spikes are noise: MAE.

  • Report both: a large RMSE/MAE ratio signals occasional big misses.

  • For relative errors across very different scales, consider MAPE, sMAPE or MASE.

Common Pitfalls

  • Comparing RMSE across datasets with different target scales.

  • Training with squared loss but reporting MAE as if they were aligned.

35

When should you use Mean Absolute Percentage Error (MAPE)?

MAPE (Mean Absolute Percentage Error) is the average of absolute errors divided by the actual values, expressed as a percentage: “on average, forecasts are off by 8%”. Use it when stakeholders think in relative terms, when comparing accuracy across series of very different scales (sales of cheap and expensive products), and when actual values are always well above zero. Avoid it when actuals can be zero or near zero (MAPE explodes or is undefined), and remember that it penalizes over-forecasts more than under-forecasts of the same size.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Strengths and Weaknesses

Strengths Weaknesses
Scale-free, easy to explain Undefined at zero; huge for small actuals
Comparable across products, regions, units Asymmetric: under-forecasts are capped at 100% error, over-forecasts are unbounded
Common in business forecasting Optimizing it biases forecasts low
Weights every observation equally regardless of volume

Alternatives

Metric Use when
WAPE (sum of absolute errors / sum of actuals) Aggregate relative error; robust to small individual actuals
sMAPE Symmetric version, still unstable near zero
MASE Error relative to a naive forecast; works with zeros
MAE / RMSE Errors in original units are meaningful

Code Example

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error

actual = np.array([100, 200, 150, 80, 120], dtype=float)
forecast = np.array([110, 190, 160, 70, 130], dtype=float)
print(f"MAPE {mean_absolute_percentage_error(actual, forecast):.1%}")    # scikit-learn returns a fraction
print(f"WAPE {np.abs(actual - forecast).sum() / actual.sum():.1%}")

# Near-zero actuals blow MAPE up
actual_small = np.array([100, 200, 150, 1, 120], dtype=float)
forecast_small = np.array([110, 190, 160, 5, 130], dtype=float)
print(f"with a near-zero actual: MAPE {mean_absolute_percentage_error(actual_small, forecast_small):.1%}, "
      f"WAPE {np.abs(actual_small - forecast_small).sum() / actual_small.sum():.1%}")

# Asymmetry: same absolute error of 50 on an actual of 100 vs 50
print(f"over-forecast 150 vs actual 100: {mean_absolute_percentage_error([100], [150]):.0%}")
print(f"under-forecast 50 vs actual 100: {mean_absolute_percentage_error([100], [50]):.0%}")
print(f"forecast 100 vs actual 50:       {mean_absolute_percentage_error([50], [100]):.0%}")

A forecast of 5 against an actual of 1 is a 400% error, which dominates the average even though the absolute error is tiny.

Common Pitfalls

  • Using MAPE on intermittent demand (many zeros).

  • Reporting scikit-learn’s value as a percentage without multiplying by 100.

  • Choosing models by MAPE without noticing they systematically under-forecast.

36

Why is hyperparameter tuning important in supervised learning?

Hyperparameters are settings chosen before training, such as tree depth, regularization strength, learning rate or number of neighbours, that control a model’s capacity and how it learns. Tuning them matters because the same algorithm can underfit badly with one setting and overfit with another, and default values rarely suit a specific dataset. Systematic tuning with cross-validation finds the bias-variance balance that generalizes best, often yielding larger gains than switching algorithms, while a held-out test set keeps the final estimate honest.

Read full answer

Applies toscikit-learn 1.5+, SciPy (Python 3.12)

Parameters Versus Hyperparameters

Parameters Hyperparameters
Learned from data during training Set before training
Regression coefficients, tree split values, network weights alpha, C, max_depth, learning rate, n_estimators

Common Hyperparameters

Model Key hyperparameters
Ridge / Lasso / Logistic alpha or C
k-NN n_neighbors, weights, distance metric
Decision tree max_depth, min_samples_leaf, ccp_alpha
Random forest max_features, min_samples_leaf, n_estimators
Gradient boosting learning_rate, max_depth/max_leaf_nodes, number of rounds, subsampling
SVM C, gamma, kernel
Neural network Learning rate, batch size, layers/units, dropout, weight decay

Methods

Grid search, random search, successive halving and Bayesian optimization (e.g. Optuna), all scored with cross-validation.

Code Example: Default vs Tuned

from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import RandomizedSearchCV, cross_val_score, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

default = HistGradientBoostingClassifier(random_state=42)
print("default CV ROC AUC:", round(cross_val_score(default, X_train, y_train, cv=5, scoring="roc_auc").mean(), 4))

search = RandomizedSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    {"learning_rate": loguniform(0.01, 0.3), "max_leaf_nodes": randint(8, 64),
     "min_samples_leaf": randint(5, 50), "l2_regularization": loguniform(1e-4, 10)},
    n_iter=40, cv=5, scoring="roc_auc", random_state=42, n_jobs=-1)
search.fit(X_train, y_train)
print("tuned CV ROC AUC:  ", round(search.best_score_, 4), search.best_params_)
print("tuned test ROC AUC:", round(search.score(X_test, y_test), 4))

On easy datasets the gain may be small; on harder or noisier data, tuning often matters much more. The tuned CV score is slightly optimistic because it was selected as the maximum; the test score is the honest estimate.

Common Pitfalls

  • Tuning on the test set.

  • Tuning too many hyperparameters on small data, which overfits the validation folds.

  • Ignoring preprocessing choices, which are hyperparameters too (put them in the Pipeline and tune them).

37

What is grid search, and how is it used for hyperparameter optimization?

Grid search is an exhaustive hyperparameter optimization method: you list candidate values for each hyperparameter, the search trains and evaluates the model for every combination using cross-validation, and the combination with the best mean score is selected and refitted on the full training data. In scikit-learn this is GridSearchCV, which also supports separate grids for different model configurations, multiple metrics and parallel execution.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Procedure

  1. Split off a test set.

  2. Put preprocessing and the model in a Pipeline.

  3. Define the grid (log-spaced values for scale parameters).

  4. Run GridSearchCV with an appropriate scoring and cv.

  5. Inspect cv_results_: is the best value at the edge of the grid? Are differences within noise?

  6. Evaluate the refitted best model once on the test set.

Code Example

import pandas as pd
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

pipe = Pipeline([("scale", StandardScaler()), ("clf", KNeighborsClassifier())])
param_grid = [   # separate grids per model: the "clf" step itself is a hyperparameter
    {"clf": [KNeighborsClassifier()], "clf__n_neighbors": [3, 5, 9, 15], "clf__weights": ["uniform", "distance"]},
    {"clf": [LogisticRegression(max_iter=2000)], "clf__C": [0.01, 0.1, 1, 10]},
]
search = GridSearchCV(pipe, param_grid, cv=StratifiedKFold(5, shuffle=True, random_state=42),
                      scoring="f1_macro", n_jobs=-1)
search.fit(X_train, y_train)

results = pd.DataFrame(search.cv_results_)[["params", "mean_test_score", "std_test_score", "rank_test_score"]]
print(results.sort_values("rank_test_score").head(5).to_string(index=False))
print("best:", search.best_params_)
print("test macro F1:", round(search.score(X_test, y_test), 3))

The search has 8 + 4 = 12 candidates × 5 folds = 60 fits.

Strengths and Weaknesses

Strengths Weaknesses
Simple, exhaustive, reproducible Cost multiplies with every hyperparameter
Easy to parallelize Only tries listed values
Clear results table Inefficient when few hyperparameters matter

Common Pitfalls

  • Scaling or selecting features outside the pipeline, which leaks fold data.

  • Picking a grid whose best value lies on the boundary and stopping there.

  • Treating best_score_ as an unbiased estimate; it is the maximum of many noisy scores.

39

How does the learning rate affect training convergence?

The learning rate sets the size of each parameter update in gradient-based training: new weights = old weights − learning rate × gradient. It is usually the single most important hyperparameter for neural networks and gradient descent models. Too small, and training converges very slowly or gets stuck; too large, and the loss oscillates or diverges. Good practice is to search it on a log scale and use schedules (warm-up, decay, cosine) or adaptive optimizers (Adam) that adjust step sizes during training.

Read full answer

Applies toNumPy 2.x, PyTorch 2.x (Python 3.12)

Effects

Learning rate Behaviour
Much too small Loss falls very slowly; may stall in flat regions
Good Fast, steady decrease
Slightly too large Noisy loss, may settle at a worse point
Much too large Loss oscillates or explodes (NaN)

For a quadratic loss with curvature L (largest eigenvalue of the Hessian), plain gradient descent converges only if the learning rate is below 2 / L.

Schedules and Optimizers

Technique Idea
Step / exponential decay Reduce the rate as training progresses
Cosine annealing Smoothly decrease to a minimum, optionally with restarts
Warm-up Start small, increase over the first steps (standard for transformers)
ReduceLROnPlateau Lower the rate when validation loss stalls
One-cycle Rise then fall within one run
Adam / AdamW Per-parameter adaptive step sizes
LR range test Increase the rate during a short run and pick a value below where loss starts to blow up

Code Example: Gradient Descent on a Quadratic

import numpy as np

# Loss = 0.5 * (10 * w1^2 + 1 * w2^2): curvatures 10 and 1, so the stable limit is lr < 2 / 10 = 0.2
H = np.diag([10.0, 1.0])
def loss(w): return 0.5 * w @ H @ w

for lr in (0.01, 0.1, 0.19, 0.21):
    w = np.array([1.0, 1.0])
    for _ in range(100):
        w = w - lr * (H @ w)                       # gradient of the loss is H w
    print(f"lr={lr:<5} loss after 100 steps: {loss(w):.3e}")

At 0.01 the loss is still large because the low-curvature direction moves slowly; 0.1 converges fast; 0.19 converges with oscillation; 0.21 diverges.

PyTorch Schedule Example

import torch

model = torch.nn.Linear(10, 1)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-2, total_steps=1000)
for step in range(1000):
    optimizer.zero_grad()
    loss = model(torch.randn(32, 10)).pow(2).mean()
    loss.backward()
    optimizer.step()
    scheduler.step()                                # update the learning rate every step
print("final lr:", scheduler.get_last_lr()[0])

Common Pitfalls

  • Tuning everything else before the learning rate.

  • Changing the batch size without adjusting the learning rate.

  • Calling scheduler.step() at the wrong frequency (per epoch vs per batch).

40

What is early stopping in training machine learning models?

Early stopping halts training when performance on a validation set stops improving, and keeps the model from the best point. In iterative learners such as neural networks and gradient boosting, training error keeps falling while validation error eventually starts to rise as the model begins to memorize noise; stopping at the validation minimum acts as regularization and also saves compute. A patience setting tolerates short plateaus before stopping.

Read full answer

Applies toscikit-learn 1.5+, XGBoost 2.0+ (Python 3.12)

Key Settings

Setting Meaning
Monitored metric Usually validation loss (or AUC, with direction “max”)
Patience Number of epochs/rounds without improvement before stopping
Min delta Smallest change that counts as improvement
Restore best weights Return the model from the best epoch, not the last
Validation data Separate from the test set

Where It Is Built In

Library How
scikit-learn HistGradientBoosting* early_stopping=True, validation_fraction, n_iter_no_change
scikit-learn MLPClassifier, SGDClassifier early_stopping=True
XGBoost early_stopping_rounds with an eval_set
Keras EarlyStopping(patience=..., restore_best_weights=True)
PyTorch Manual loop or Lightning’s EarlyStopping callback

Code Example

from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier

X, y = make_classification(n_samples=5000, n_features=30, n_informative=10, flip_y=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=42)

# scikit-learn: internal validation split
hgb = HistGradientBoostingClassifier(max_iter=2000, learning_rate=0.1, early_stopping=True,
                                     validation_fraction=0.15, n_iter_no_change=20, random_state=42)
hgb.fit(X_train, y_train)
print("HistGradientBoosting stopped after", hgb.n_iter_, "of 2000 iterations; test acc",
      round(hgb.score(X_test, y_test), 3))

# XGBoost: explicit validation set
xgb = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=4, early_stopping_rounds=50,
                    eval_metric="logloss", random_state=42)
xgb.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
print("XGBoost best iteration:", xgb.best_iteration, "| test acc", round(xgb.score(X_test, y_test), 3))

Common Pitfalls

  • Using the test set for early stopping, which makes the test score optimistic.

  • Too little patience with noisy validation curves, stopping too early.

  • Not restoring the best weights in neural networks.

  • A tiny validation set that makes the stopping point random.

41

How would you apply supervised learning to predict stock prices?

Applying supervised learning to stock prices means framing a target such as next-day return or direction, building features only from information available at prediction time (past returns, volatility, volume, market and macro indicators), and training a model with time-ordered validation. The realistic goal is a small, robust edge after costs, not accurate price levels: prices are close to a random walk, so predicting raw prices mostly reproduces yesterday’s price. A sound answer emphasizes leakage control, baselines and backtesting more than model choice.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, NumPy 2.x (Python 3.12)

Design

Step Choice
Target Next-day (or next-week) log return (regression) or its sign (classification)
Features Lagged returns, rolling volatility, momentum, volume change, index return; all shifted so they use only past data
Split Chronological; walk-forward with TimeSeriesSplit
Models Ridge / logistic regression, gradient boosting; deep models only with lots of data
Baselines Zero return, previous return, always “up”
Evaluation MAE or directional accuracy, plus a backtest with transaction costs

Code Example

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 3000
ret = rng.normal(0.0003, 0.01, n)                        # simulated daily returns (no real signal)
df = pd.DataFrame({"ret": ret})
for lag in (1, 2, 5):
    df[f"ret_lag{lag}"] = df["ret"].shift(lag)
df["vol_20"] = df["ret"].rolling(20).std().shift(1)
df["mom_10"] = df["ret"].rolling(10).sum().shift(1)
df["target"] = (df["ret"] > 0).astype(int)                # direction of today's return, from past features
df = df.dropna()

features = [c for c in df.columns if c.startswith(("ret_lag", "vol", "mom"))]
X, y = df[features], df["target"]
cv = TimeSeriesSplit(n_splits=5)

for name, model in [("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
                    ("boosting", HistGradientBoostingClassifier(max_depth=3, random_state=42))]:
    acc = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
    print(f"{name:9s} walk-forward accuracy {acc.mean():.3f} ± {acc.std():.3f}")
print("always-up baseline:", round(y.mean(), 3))

The returns are simulated noise, so a correct pipeline should not beat the always-up baseline meaningfully. If a pipeline beats noise, it has a leakage bug. Run the same code on real data only after this sanity check passes.

Backtest Essentials

  • Convert predictions into positions, apply realistic costs and slippage, and compute risk-adjusted returns (Sharpe ratio, drawdown).

  • Test on a final out-of-sample period never used for development.

Common Pitfalls

  • Features computed with data from the same day as the target (look-ahead bias).

  • Random K-fold splits that mix future and past.

  • Survivorship bias from using only today’s listed companies.

  • Data snooping: trying hundreds of models on the same history.

42

How is supervised learning applied to credit scoring?

Credit scoring uses supervised learning to estimate the probability that an applicant will default (e.g. 90+ days past due within 12 months) from application data, credit bureau history and, where allowed, transaction data. The model’s probabilities drive approve/decline decisions, credit limits and pricing. Because lending is regulated and decisions affect people, the work emphasizes calibrated probabilities, interpretability (reason codes for declines), fairness and stability over time, so logistic regression scorecards remain common alongside gradient-boosted trees with explanations.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Pipeline

Step Details
Target definition Default within a fixed window after the loan starts
Sample Booked loans with matured outcomes; note reject-inference issues (rejected applicants have no outcome)
Features Income, debt-to-income, utilization, delinquencies, credit history length, inquiries
Models Logistic regression scorecard (often with binned/WoE features), gradient boosting
Metrics ROC AUC / Gini (= 2·AUC − 1), KS statistic, calibration, profit at the cut-off
Validation Out-of-time test set (later application dates)
Governance Explainability, adverse-action reasons, fairness tests, model risk documentation, monitoring (PSI)

Code Example

import numpy as np
import pandas as pd
from sklearn.calibration import calibration_curve
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 20000
df = pd.DataFrame({
    "month": rng.integers(0, 24, n),                          # application month
    "income": rng.lognormal(10.8, 0.5, n),
    "dti": rng.uniform(0, 0.6, n),
    "utilization": rng.beta(2, 3, n),
    "delinquencies": rng.poisson(0.3, n),
    "history_years": rng.uniform(0, 25, n),
})
logit = (-3 + 3 * df["dti"] + 2.5 * df["utilization"] + 0.6 * df["delinquencies"]
         - 0.05 * df["history_years"] - 0.4 * (np.log(df["income"]) - 10.8))
df["default"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

features = ["income", "dti", "utilization", "delinquencies", "history_years"]
train, test = df[df["month"] < 18], df[df["month"] >= 18]      # out-of-time split

models = {"logistic scorecard": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
          "gradient boosting": HistGradientBoostingClassifier(max_depth=3, random_state=42)}
for name, model in models.items():
    model.fit(train[features], train["default"])
    p = model.predict_proba(test[features])[:, 1]
    auc = roc_auc_score(test["default"], p)
    frac, mean_p = calibration_curve(test["default"], p, n_bins=5, strategy="quantile")
    print(f"{name:19s} AUC {auc:.3f} Gini {2 * auc - 1:.3f} | predicted vs observed default rate by quintile: "
          f"{[(round(a, 3), round(b, 3)) for a, b in zip(mean_p, frac)]}")

coefs = models["logistic scorecard"][-1].coef_[0]
print(dict(zip(features, coefs.round(2))))      # signs give reason codes: higher DTI raises risk

Common Pitfalls

  • Random splits instead of out-of-time validation.

  • Features unavailable at application time (e.g. post-approval behaviour).

  • Using protected attributes or close proxies; test outcomes across groups and follow local regulation.

  • Ignoring population drift; monitor the population stability index and recalibrate.

43

How can supervised learning be used in recommender systems?

Supervised learning powers most production recommender systems by framing recommendation as predicting an outcome for a (user, item, context) pair: will the user click, buy, watch to the end, or what rating will they give? Models are trained on logged interactions with features about the user, the item and the context, and their predicted scores rank candidate items. Systems typically have two stages: candidate generation (retrieving hundreds of items quickly, often with embeddings) and ranking (a supervised model such as gradient boosting or a deep network scoring each candidate).

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Framings

Framing Label Model output Metric
Pointwise classification Clicked / purchased (0/1) Probability ROC AUC, log loss
Pointwise regression Rating, watch time Number RMSE / MAE
Pairwise ranking Item A preferred over B Score difference Pairwise accuracy
Listwise ranking Ordered list Ranking scores NDCG@k, MAP
Two-tower retrieval Positive vs sampled negatives Embedding similarity Recall@k

Features

User history aggregates, item popularity and attributes, user-item affinity (category match, past interactions), context (time, device), and embeddings from collaborative filtering.

Code Example: Click Prediction Ranker

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import ndcg_score, roc_auc_score

rng = np.random.default_rng(42)
n_users, n_items, n_rows = 500, 200, 40000
user_pref = rng.integers(0, 5, n_users)                   # favourite category per user
item_cat = rng.integers(0, 5, n_items)
item_pop = rng.beta(2, 5, n_items)

rows = pd.DataFrame({"user": rng.integers(0, n_users, n_rows), "item": rng.integers(0, n_items, n_rows),
                     "day": rng.integers(0, 30, n_rows)})
rows["cat_match"] = (user_pref[rows["user"]] == item_cat[rows["item"]]).astype(int)
rows["popularity"] = item_pop[rows["item"]]
rows["hour"] = rng.integers(0, 24, n_rows)
p_click = 1 / (1 + np.exp(-(-3 + 2.0 * rows["cat_match"] + 3 * rows["popularity"])))
rows["click"] = (rng.random(n_rows) < p_click).astype(int)

features = ["cat_match", "popularity", "hour"]
train, test = rows[rows["day"] < 25], rows[rows["day"] >= 25]     # time-based split
model = HistGradientBoostingClassifier(random_state=42).fit(train[features], train["click"])
test = test.assign(score=model.predict_proba(test[features])[:, 1])
print("ROC AUC:", round(roc_auc_score(test["click"], test["score"]), 3))

# Ranking quality per user: NDCG@10 over each user's test impressions
ndcgs = [ndcg_score([g["click"].to_numpy()], [g["score"].to_numpy()], k=10)
         for _, g in test.groupby("user") if g["click"].sum() > 0 and len(g) > 1]
print("mean NDCG@10:", round(float(np.mean(ndcgs)), 3))

Common Pitfalls

  • Feedback loops: logged data only contains items the old system showed; consider exploration and position-bias correction.

  • Random splits that leak future interactions; split by time.

  • Optimizing clicks alone, which can promote clickbait; include longer-term signals.

44

How is supervised learning applied in healthcare diagnostics?

In healthcare diagnostics, supervised learning trains models on patient data labelled with confirmed diagnoses or outcomes to predict disease presence or risk for new patients. Common inputs are medical images (X-rays, CT, retinal photos, pathology slides), signals (ECG), lab results and electronic health records. Models act as decision support, flagging likely cases, prioritizing worklists or estimating risk, and must meet stricter standards than most ML: high sensitivity for serious conditions, calibrated risk estimates, validation on external populations, fairness across patient groups and regulatory approval for clinical use.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Applications

Area Input Model
Radiology Chest X-ray, CT CNNs / vision transformers (often pretrained)
Ophthalmology Retinal fundus images CNNs for diabetic retinopathy
Pathology Whole-slide images Patch-based CNNs, multiple-instance learning
Cardiology ECG 1D CNNs
Clinical risk EHR (labs, vitals, history) Logistic regression, gradient boosting
Laboratory Blood panels Tree ensembles

Workflow

  1. Define the clinical question and the action the prediction triggers.

  2. Collect data with confirmed labels; split by patient (and ideally by site or time).

  3. Train with appropriate metrics: sensitivity, specificity, ROC AUC, PR AUC, calibration.

  4. Choose the threshold with clinicians based on error costs.

  5. Validate externally, check subgroups, and document limitations.

  6. Deploy with monitoring and a human in the loop.

Code Example: Patient-Level Grouped Validation

import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import recall_score, roc_auc_score
from sklearn.model_selection import GroupKFold, KFold, cross_val_predict

rng = np.random.default_rng(42)
n_patients, visits = 400, 5
patient = np.repeat(np.arange(n_patients), visits)
baseline = rng.normal(0, 1, n_patients)[patient]                 # patient-specific signal
disease = (rng.random(n_patients) < 0.2).astype(int)[patient]
X = np.column_stack([baseline + rng.normal(0, 0.1, len(patient)),  # nearly identical across a patient's visits
                     0.6 * disease + rng.normal(0, 1, len(patient)),
                     rng.normal(0, 1, len(patient))])
y = disease

model = HistGradientBoostingClassifier(max_depth=3, random_state=42)
for name, cv, groups in [("random K-fold (leaky)", KFold(5, shuffle=True, random_state=42), None),
                         ("GroupKFold by patient", GroupKFold(5), patient)]:
    p = cross_val_predict(model, X, y, cv=cv, groups=groups, method="predict_proba")[:, 1]
    print(f"{name:22s} ROC AUC {roc_auc_score(y, p):.3f}, sensitivity at 0.3: {recall_score(y, (p >= 0.3).astype(int)):.3f}")

With random splits the model can recognize each patient’s fingerprint feature from their other visits, so its scores look better than they would for new patients; grouping by patient gives the honest estimate.

Common Pitfalls

  • Image-level splits when one patient has many images.

  • Shortcut learning (scanner markers, hospital-specific artefacts).

  • Optimizing accuracy for rare conditions; prioritize sensitivity at an acceptable specificity.

45

What role does supervised learning play in natural language processing?

Supervised learning is central to NLP: tasks such as sentiment analysis, spam detection, topic classification, named-entity recognition, question answering and machine translation are all trained on labelled text pairs. Classic approaches use TF-IDF features with linear models; modern approaches fine-tune pretrained transformers (themselves trained with self-supervised objectives) on a labelled task dataset, and large language models are further refined with supervised instruction tuning on curated prompt-response pairs. Label quality, task definition and evaluation remain the deciding factors.

Read full answer

Applies toscikit-learn 1.5+, Hugging Face Transformers 4.4x (optional) (Python 3.12)

Supervised NLP Tasks

Task Input → label Typical model
Text classification Document → category TF-IDF + linear model; fine-tuned BERT-style encoder
Sentiment analysis Review → positive/negative/neutral Same
Named-entity recognition Token → entity tag (BIO) Token-classification transformer, CRF
Question answering Question + passage → answer span Encoder with span head
Translation, summarization Text → text Encoder-decoder (T5, BART)
Instruction tuning Prompt → desired response Supervised fine-tuning of LLMs

Approaches Compared

Approach Labelled data needed Accuracy Cost
TF-IDF + logistic regression / linear SVM Hundreds to thousands Strong baseline Very low
Fine-tuned transformer Hundreds to thousands Usually best GPU training
Zero/few-shot LLM prompting None to a few Good for many tasks Inference cost, less control

Code Example: Classic Supervised Baseline

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.pipeline import make_pipeline

cats = ["rec.sport.baseball", "sci.space", "talk.politics.mideast"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))

model = make_pipeline(TfidfVectorizer(sublinear_tf=True, ngram_range=(1, 2), min_df=2),
                      LogisticRegression(max_iter=2000, C=10))
model.fit(train.data, train.target)
print(classification_report(test.target, model.predict(test.data), target_names=train.target_names, digits=3))

A transformer fine-tune (for example AutoModelForSequenceClassification from Hugging Face with a pretrained checkpoint) is the next step when the baseline is not enough.

Common Pitfalls

  • Inconsistent annotation guidelines, which cap model accuracy; measure inter-annotator agreement.

  • Leakage through duplicated or near-duplicated texts across splits.

  • Domain shift (training on news, deploying on social media).

46

Implement linear regression from scratch in Python.

Linear regression from scratch fits weights w and bias b so that predictions X w + b minimize the mean squared error. There are two standard approaches: the closed-form least-squares solution (solve the normal equations, done stably with lstsq), and gradient descent, which repeatedly moves the weights against the gradient of the error. The implementation below provides both, adds optional L2 regularization, and checks the results against scikit-learn.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (for data and comparison) (Python 3.12)

Math in Words

  • Prediction: weighted sum of features plus bias.

  • Loss: average of squared differences between predictions and targets.

  • Gradient of the loss with respect to w: (2/n) Xᵀ(Xw + b − y); with respect to b: (2/n) × sum(Xw + b − y).

  • Closed form: w = (XᵀX)⁻¹Xᵀy on features augmented with a column of ones; never invert the matrix directly in practice, use a least-squares solver.

Code Example

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


class LinearRegressionScratch:
    def __init__(self, method="closed_form", lr=0.05, epochs=5000, l2=0.0):
        self.method, self.lr, self.epochs, self.l2 = method, lr, epochs, l2

    def fit(self, X, y):
        n, d = X.shape
        if self.method == "closed_form":
            Xb = np.c_[np.ones(n), X]
            reg = self.l2 * np.eye(d + 1)
            reg[0, 0] = 0                                     # do not penalize the bias
            theta = np.linalg.lstsq(Xb.T @ Xb + reg, Xb.T @ y, rcond=None)[0]
            self.b, self.w = theta[0], theta[1:]
        else:
            self.w, self.b = np.zeros(d), 0.0
            for _ in range(self.epochs):
                error = X @ self.w + self.b - y
                self.w -= self.lr * (2 / n * X.T @ error + 2 * self.l2 / n * self.w)
                self.b -= self.lr * (2 / n * error.sum())
        return self

    def predict(self, X):
        return X @ self.w + self.b


X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)                      # gradient descent needs scaled features
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

closed = LinearRegressionScratch("closed_form").fit(X_train, y_train)
gd = LinearRegressionScratch("gradient_descent").fit(X_train, y_train)
ref = LinearRegression().fit(X_train, y_train)

print("closed form R²:", round(r2_score(y_test, closed.predict(X_test)), 4))
print("grad descent R²:", round(r2_score(y_test, gd.predict(X_test)), 4))
print("sklearn R²:     ", round(ref.score(X_test, y_test), 4))
print("closed form matches sklearn:", np.allclose(closed.w, ref.coef_))
print("largest weight difference, GD vs exact:", round(float(np.abs(gd.w - ref.coef_).max()), 3))

Gradient descent approaches the exact solution gradually; strongly correlated features in this dataset (such as the cholesterol measures) slow the final convergence, so its weights can still differ somewhat even when its R² is nearly identical.

Common Pitfalls

  • Unscaled features with gradient descent, which needs a tiny learning rate and many epochs.

  • Inverting XᵀX explicitly, which is numerically unstable with correlated features.

  • Penalizing the bias term in the regularized version.

47

Write a logistic regression classifier using scikit-learn.

A logistic regression classifier in scikit-learn is built with LogisticRegression, ideally inside a Pipeline with StandardScaler so features are scaled consistently in training and prediction. The workflow is: split the data, compare against a dummy baseline, tune the regularization strength C with cross-validation, evaluate on the test set with suitable metrics, choose a decision threshold if errors have different costs, and interpret the coefficients as odds ratios.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2,
                                                    stratify=data.target, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

print("baseline CV accuracy:", round(cross_val_score(DummyClassifier(), X_train, y_train, cv=cv).mean(), 3))

pipe = Pipeline([("scale", StandardScaler()), ("logreg", LogisticRegression(max_iter=1000))])
grid = GridSearchCV(pipe, {"logreg__C": np.logspace(-3, 2, 11), "logreg__class_weight": [None, "balanced"]},
                    cv=cv, scoring="roc_auc")
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV ROC AUC:", round(grid.best_score_, 4))

model = grid.best_estimator_
proba = model.predict_proba(X_test)[:, 1]
pred = model.predict(X_test)
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=data.target_names, digits=3))

# Interpret: odds ratio per one standard deviation increase
coefs = model.named_steps["logreg"].coef_[0]
top = np.argsort(-np.abs(coefs))[:5]
for i in top:
    print(f"{data.feature_names[i]:22s} coef {coefs[i]:+.2f}  odds ratio {np.exp(coefs[i]):.2f}")

Useful Options

Option Purpose
C Inverse regularization strength (small C = stronger penalty)
penalty="l1" with solver="liblinear" or "saga" Sparse coefficients
class_weight="balanced" Imbalanced classes
max_iter Raise if you see convergence warnings
Multi-class targets Handled automatically with a multinomial (softmax) model

Common Pitfalls

  • Skipping scaling, which slows convergence and makes the penalty uneven.

  • Using predict (threshold 0.5) when costs or class balance call for another threshold.

  • Interpreting coefficients of correlated features as independent effects.

48

Build and visualize a decision tree classifier on a dataset.

A decision tree classifier splits the data with a sequence of feature thresholds chosen to make each group as pure as possible, and its structure can be shown directly as a diagram or as text rules, which makes it one of the most explainable models. The example trains a depth-limited tree on the iris dataset, chooses the depth with cross-validation, evaluates it, and visualizes it with plot_tree and export_text, plus the decision regions for two features.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+ (Python 3.12)

Code Example

import matplotlib

matplotlib.use("Agg")                                # write images to files
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text, plot_tree

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3,
                                                    stratify=iris.target, random_state=42)

# Choose depth and leaf size with cross-validation
search = GridSearchCV(DecisionTreeClassifier(random_state=42),
                      {"max_depth": [1, 2, 3, 4, 5, None], "min_samples_leaf": [1, 3, 5]}, cv=5)
search.fit(X_train, y_train)
tree = search.best_estimator_
print("best:", search.best_params_)
print(classification_report(y_test, tree.predict(X_test), target_names=iris.target_names))

# 1. Text rules
print(export_text(tree, feature_names=list(iris.feature_names)))

# 2. Tree diagram
fig, ax = plt.subplots(figsize=(14, 7))
plot_tree(tree, feature_names=iris.feature_names, class_names=iris.target_names,
          filled=True, rounded=True, impurity=True, ax=ax)
plt.savefig("iris_tree.png", dpi=120, bbox_inches="tight")

# 3. Decision regions using the two petal features
two = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train[:, 2:], y_train)
fig, ax = plt.subplots(figsize=(6, 5))
DecisionBoundaryDisplay.from_estimator(two, X_train[:, 2:], response_method="predict", alpha=0.3, ax=ax,
                                       xlabel="petal length (cm)", ylabel="petal width (cm)")
ax.scatter(X_train[:, 2], X_train[:, 3], c=y_train, edgecolors="k", s=25)
plt.savefig("iris_tree_regions.png", dpi=120, bbox_inches="tight")

Reading the Diagram

  • Each internal node shows the split rule, the impurity (Gini), the number of samples and the class counts (value).

  • Colour intensity reflects the majority class and its purity.

  • Leaves give the predicted class; the path from root to leaf is an if-then rule.

  • Decision regions are axis-aligned rectangles, a hallmark of trees.

Common Pitfalls

  • Visualizing an unpruned tree with hundreds of nodes; limit depth for explanation.

  • Treating the tree as stable: retraining on a slightly different sample can change the rules.

49

Build and train a simple feedforward neural network with Keras.

A feedforward neural network (multi-layer perceptron) passes inputs through stacked fully connected layers with non-linear activations to an output layer suited to the task. In Keras you define it with keras.Sequential, compile it with an optimizer, loss and metrics, and train it with fit, using a validation split and early stopping to avoid overfitting. The example classifies breast-cancer biopsies, with scaling inside the model via a Normalization layer.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)

Code Example

import keras
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split

keras.utils.set_random_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X = X.astype("float32")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=42)

norm = keras.layers.Normalization()
norm.adapt(X_tr)                                         # scaling statistics from the training split only

model = keras.Sequential([
    keras.Input(shape=(X.shape[1],)),
    norm,
    keras.layers.Dense(32, activation="relu"),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(16, activation="relu"),
    keras.layers.Dense(1, activation="sigmoid"),         # probability of the positive class
])
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="binary_crossentropy",
              metrics=["accuracy", keras.metrics.AUC(name="auc")])
model.summary()

history = model.fit(X_tr, y_tr, validation_data=(X_val, y_val), epochs=200, batch_size=32, verbose=0,
                    callbacks=[keras.callbacks.EarlyStopping(monitor="val_auc", mode="max", patience=15,
                                                             restore_best_weights=True)])
print("stopped after", len(history.history["loss"]), "epochs")

proba = model.predict(X_test, verbose=0).ravel()
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))
print(classification_report(y_test, (proba >= 0.5).astype(int), digits=3))
model.save("ffn_breast_cancer.keras")

Design Choices

Choice Reason
Normalization layer inside the model Same scaling at training and inference
ReLU hidden layers Fast, avoids vanishing gradients
Sigmoid output + binary cross-entropy Binary classification
Dropout + early stopping Limits overfitting on a small dataset
Separate validation split Early stopping without touching the test set

For multi-class problems use Dense(K, activation="softmax") with sparse_categorical_crossentropy; for regression use Dense(1) with mse or mae.

Common Pitfalls

  • Fitting the normalization on all data, including validation and test rows.

  • Monitoring a metric name that does not exist (e.g. val_auc without naming the metric “auc”).

  • On small tabular datasets, a neural network may not beat logistic regression or gradient boosting; compare.

50

Write a Python function to compute the F1 score from a confusion matrix.

The F1 score is the harmonic mean of precision and recall. From a binary confusion matrix it is 2·TP / (2·TP + FP + FN), which equals 2PR / (P + R) with precision P = TP / (TP + FP) and recall R = TP / (TP + FN). For multi-class confusion matrices, compute per-class F1 by treating each class as positive in turn, then average them: macro (simple mean), weighted (by class support) or micro (from pooled counts). The function below handles all of these and zero-division edge cases.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (for verification) (Python 3.12)

Code Example

import numpy as np
from sklearn.metrics import confusion_matrix, f1_score


def f1_from_confusion(cm, average="binary", positive_index=1):
    """F1 from a confusion matrix with rows = true classes, columns = predicted classes.

    average: "binary" (2x2 only), "macro", "weighted", "micro", or None for per-class scores.
    Classes with no true and no predicted samples get F1 = 0.
    """
    cm = np.asarray(cm, dtype=float)
    tp = np.diag(cm)
    fp = cm.sum(axis=0) - tp                       # predicted as the class but wrong
    fn = cm.sum(axis=1) - tp                       # belong to the class but missed
    denom = 2 * tp + fp + fn
    per_class = np.divide(2 * tp, denom, out=np.zeros_like(tp), where=denom > 0)

    if average == "binary":
        if cm.shape != (2, 2):
            raise ValueError("binary average needs a 2x2 matrix")
        return per_class[positive_index]
    if average is None:
        return per_class
    if average == "macro":
        return per_class.mean()
    if average == "weighted":
        support = cm.sum(axis=1)
        return np.average(per_class, weights=support)
    if average == "micro":
        total = 2 * tp.sum() + fp.sum() + fn.sum()
        return 2 * tp.sum() / total if total else 0.0
    raise ValueError(f"unknown average: {average}")


# Binary check: TP=40, FP=10, FN=40, TN=910
cm_bin = np.array([[910, 10],
                   [40, 40]])
print("binary F1:", round(f1_from_confusion(cm_bin), 4))          # 2*40 / (80 + 10 + 40) = 0.6154

# Multi-class check against scikit-learn
rng = np.random.default_rng(42)
y_true = rng.integers(0, 3, 300)
y_pred = np.where(rng.random(300) < 0.7, y_true, rng.integers(0, 3, 300))
cm = confusion_matrix(y_true, y_pred)
for avg in ("macro", "weighted", "micro"):
    ours, theirs = f1_from_confusion(cm, avg), f1_score(y_true, y_pred, average=avg)
    print(f"{avg:8s} ours {ours:.4f} | sklearn {theirs:.4f} | equal {np.isclose(ours, theirs)}")

Notes

  • scikit-learn’s confusion_matrix puts true labels in rows and predicted labels in columns; matrices from other tools may be transposed.

  • Micro-averaged F1 equals accuracy for single-label multi-class problems.

  • F-beta generalizes the formula: (1 + β²)·TP / ((1 + β²)·TP + β²·FN + FP).

Common Pitfalls

  • Swapping FP and FN by reading the matrix with the wrong orientation.

  • Averaging per-class precision and recall first, then computing F1 from the averages; that is not macro F1.

51

How do you deal with imbalanced datasets in classification tasks?

Imbalanced datasets, where one class is much rarer than another (fraud, disease, defects), make standard training and metrics favour the majority class. The fix has three parts: evaluate properly (precision, recall, F1, PR AUC instead of accuracy), adjust learning (class weights, resampling such as SMOTE or undersampling, or algorithms robust to imbalance), and choose the decision threshold from validation data according to the cost of each error. Often class weights plus threshold tuning are enough; resampling helps in some cases but must stay inside cross-validation.

Read full answer

Applies toscikit-learn 1.5+, imbalanced-learn 0.12+ (Python 3.12)

Techniques

Technique How Notes
Right metrics Recall, precision, F1, PR AUC, balanced accuracy Accuracy is misleading
Stratified splits stratify=y, StratifiedKFold Keeps rare class in every fold
Class weights class_weight="balanced" Simple, no data duplication
Oversampling Random oversampling, SMOTE (imbalanced-learn) Only on training folds
Undersampling Random or informed removal of majority samples Faster; may lose information
Threshold tuning Pick cut-off on validation predictions Often the biggest win
Ensembles Balanced random forest, EasyEnsemble Combine resampling and bagging
Anomaly detection Model the normal class only Extremely rare, varied positives

Code Example

import numpy as np
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as imb_pipeline
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, f1_score, precision_recall_curve
from sklearn.model_selection import cross_val_predict, train_test_split, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=20000, n_features=20, n_informative=6, weights=[0.98, 0.02],
                           flip_y=0.005, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)

models = {
    "plain": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "class_weight": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, class_weight="balanced")),
    "SMOTE": imb_pipeline(StandardScaler(), SMOTE(random_state=42), LogisticRegression(max_iter=1000)),
}
for name, model in models.items():
    # Out-of-fold probabilities on training data are used to pick the threshold
    oof = cross_val_predict(model, X_train, y_train, cv=cv, method="predict_proba")[:, 1]
    prec, rec, thr = precision_recall_curve(y_train, oof)
    f1 = 2 * prec[:-1] * rec[:-1] / np.maximum(prec[:-1] + rec[:-1], 1e-12)
    best_thr = thr[np.argmax(f1)]

    model.fit(X_train, y_train)
    proba = model.predict_proba(X_test)[:, 1]
    print(f"{name:13s} PR AUC {average_precision_score(y_test, proba):.3f} | F1 @0.5 "
          f"{f1_score(y_test, (proba >= 0.5).astype(int)):.3f} | F1 @tuned {best_thr:.2f}: "
          f"{f1_score(y_test, (proba >= best_thr).astype(int)):.3f}")

PR AUC measures ranking quality and is unaffected by the threshold; the F1 columns show how much the threshold choice matters.

Common Pitfalls

  • Resampling before splitting or outside the CV pipeline, which leaks synthetic copies of test-like points.

  • Tuning the threshold on the test set.

  • Assuming SMOTE always helps; compare it against class weights.

52

How are SVMs used with non-linear kernels?

Non-linear kernels let an SVM learn curved decision boundaries while keeping the same maximum-margin optimization. A kernel function computes the similarity of two points as if they had been mapped into a higher-dimensional feature space, where a linear separator may exist; the SVM never computes that mapping explicitly (the kernel trick). The RBF (Gaussian) kernel is the default choice for non-linear problems, with the polynomial kernel used when feature interactions of a known degree matter; both require scaled features and joint tuning of C with the kernel parameters.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Kernels

Kernel Similarity Parameters Boundary
Linear Dot product C Straight hyperplane
Polynomial (gamma × dot product + coef0)^degree degree, gamma, coef0, C Polynomial curves
RBF exp(−gamma × squared distance) gamma, C Flexible, local
Sigmoid tanh(gamma × dot product + coef0) gamma, coef0, C Rarely best

The decision function becomes a weighted sum of kernel similarities to the support vectors plus a bias, so prediction cost grows with the number of support vectors.

Tuning

  • C: penalty for margin violations (large = less regularization).

  • gamma (RBF): reach of each support vector; large gamma gives wiggly boundaries that overfit, small gamma gives smooth ones that may underfit.

  • Search both on log scales together.

Code Example

import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_circles(n_samples=600, factor=0.5, noise=0.12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

param_grid = [
    {"svc__kernel": ["linear"], "svc__C": [0.1, 1, 10]},
    {"svc__kernel": ["poly"], "svc__degree": [2, 3], "svc__coef0": [0, 1], "svc__C": [0.1, 1, 10]},
    {"svc__kernel": ["rbf"], "svc__gamma": np.logspace(-2, 1, 4), "svc__C": [0.1, 1, 10, 100]},
]
search = GridSearchCV(make_pipeline(StandardScaler(), SVC()), param_grid, cv=5, n_jobs=-1)
search.fit(X_train, y_train)

for kernel in ("linear", "poly", "rbf"):
    mask = np.array([p["svc__kernel"] == kernel for p in search.cv_results_["params"]])
    print(f"{kernel:7s} best CV accuracy {search.cv_results_['mean_test_score'][mask].max():.3f}")
print("overall best:", search.best_params_, "| test accuracy:", round(search.score(X_test, y_test), 3))

Concentric circles cannot be separated by a line, so the linear kernel should sit near 50% while polynomial (degree 2) and RBF kernels separate them well.

When to Use Non-Linear Kernels

  • Low- to medium-dimensional data with curved class boundaries.

  • Up to tens of thousands of samples (training is roughly O(n²) to O(n³)).

  • For larger data, approximate the kernel (Nystroem, RBFSampler) and use a linear SVM.

Common Pitfalls

  • No feature scaling, which distorts distances.

  • Large gamma and C together, which memorizes the training set.

  • Using RBF on very high-dimensional sparse text, where a linear kernel is usually as accurate and much faster.

53

How are decision trees pruned?

Pruning reduces a decision tree’s size to prevent overfitting. Pre-pruning (early stopping) stops growth during construction using limits such as maximum depth, minimum samples per leaf or minimum impurity decrease. Post-pruning grows a large tree first and then removes branches that add little predictive value; scikit-learn implements minimal cost-complexity pruning, which trades the tree’s training error against its number of leaves with a parameter ccp_alpha chosen by cross-validation.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Pre-Pruning Parameters

Parameter Effect
max_depth Limits the number of split levels
min_samples_split A node needs at least this many samples to split
min_samples_leaf Each leaf must keep at least this many samples
max_leaf_nodes Grows best-first up to a leaf budget
min_impurity_decrease A split must reduce impurity by at least this much

Cost-Complexity Pruning

  • For each subtree T, cost = total leaf impurity (training error) + alpha × number of leaves.

  • As alpha increases, the weakest links (branches whose removal increases error the least per removed leaf) are pruned first, producing a nested sequence of smaller trees.

  • cost_complexity_pruning_path returns the effective alphas; choose the best one with cross-validation.

Other post-pruning methods include reduced-error pruning (remove nodes if validation accuracy doesn’t drop) and C4.5’s pessimistic error pruning.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

full = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
path = full.cost_complexity_pruning_path(X_train, y_train)
alphas = np.unique(path.ccp_alphas[:-1])                    # drop the alpha that prunes to a single node

search = GridSearchCV(DecisionTreeClassifier(random_state=42), {"ccp_alpha": alphas}, cv=5)
search.fit(X_train, y_train)
pruned = search.best_estimator_

pre = GridSearchCV(DecisionTreeClassifier(random_state=42),
                   {"max_depth": [2, 3, 4, 5, 6], "min_samples_leaf": [1, 5, 10]}, cv=5).fit(X_train, y_train)

for name, tree in [("unpruned", full), ("post-pruned (ccp)", pruned), ("pre-pruned", pre.best_estimator_)]:
    print(f"{name:18s} leaves {tree.get_n_leaves():3d}, depth {tree.get_depth():2d}, "
          f"train {tree.score(X_train, y_train):.3f}, test {tree.score(X_test, y_test):.3f}")
print("chosen ccp_alpha:", round(search.best_params_["ccp_alpha"], 5))

The pruned trees should be much smaller with similar or better test accuracy than the unpruned tree, which has perfect training accuracy.

Common Pitfalls

  • Choosing ccp_alpha or depth by test accuracy.

  • Over-pruning, which underfits.

  • Pruning trees inside random forests: forests rely on deep, low-bias trees and averaging instead.

54

What is AdaBoost, and how does its algorithm work?

AdaBoost (Adaptive Boosting) builds a strong classifier from many weak learners, usually decision stumps (one-split trees), trained one after another. After each round it increases the weights of misclassified training examples so the next learner focuses on them, and it gives each learner a vote proportional to its accuracy. The final prediction is the weighted vote of all learners. AdaBoost was the first practical boosting algorithm (Freund and Schapire, 1997) and minimizes an exponential loss.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Algorithm (Binary, Labels ±1)

  1. Start with equal weights w_i = 1/n.

  2. For each round m:

  3. Train weak learner h_m on the weighted data.

  4. Weighted error: err_m = sum of weights of misclassified points.

  5. Learner weight: alpha_m = 0.5 × ln((1 − err_m) / err_m); accurate learners get large votes.

  6. Update: multiply weights of misclassified points by exp(alpha_m) and correct ones by exp(−alpha_m), then normalize.

  7. Predict sign(sum_m alpha_m h_m(x)).

A learner with error 0.5 (random) gets alpha = 0; lower error gives a larger vote.

Code Example: From Scratch vs scikit-learn

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y01 = make_classification(n_samples=2000, n_features=10, n_informative=5, random_state=42)
y = np.where(y01 == 1, 1, -1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

def adaboost_fit(X, y, rounds=100):
    n = len(y)
    w = np.full(n, 1 / n)
    learners, alphas = [], []
    for _ in range(rounds):
        stump = DecisionTreeClassifier(max_depth=1, random_state=42).fit(X, y, sample_weight=w)
        pred = stump.predict(X)
        err = np.clip(w[pred != y].sum(), 1e-10, 1 - 1e-10)
        alpha = 0.5 * np.log((1 - err) / err)
        w *= np.exp(-alpha * y * pred)                  # up-weights mistakes, down-weights correct points
        w /= w.sum()
        learners.append(stump)
        alphas.append(alpha)
    return learners, np.array(alphas)

def adaboost_predict(learners, alphas, X):
    return np.sign(sum(a * h.predict(X) for h, a in zip(learners, alphas)))

learners, alphas = adaboost_fit(X_train, y_train)
print("scratch AdaBoost accuracy:", round((adaboost_predict(learners, alphas, X_test) == y_test).mean(), 3))
print("first learner weights:", alphas[:5].round(3))

ref = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, random_state=42).fit(X_train, y_train)
print("scikit-learn AdaBoost accuracy:", round(ref.score(X_test, y_test), 3))
print("single stump accuracy:", round(DecisionTreeClassifier(max_depth=1).fit(X_train, y_train).score(X_test, y_test), 3))

scikit-learn’s SAMME variant uses alpha = ln((1 − err)/err) + ln(K − 1) with a learning rate, which for two classes is twice the value above; predictions are the same.

Strengths and Weaknesses

Strengths Weaknesses
Simple, few hyperparameters Sensitive to label noise and outliers (their weights grow exponentially)
Turns weak learners into a strong model Sequential, so harder to parallelize
Often resists overfitting in practice Usually outperformed by gradient boosting today

Common Pitfalls

  • Using strong base learners (deep trees), which overfit quickly.

  • Applying AdaBoost to very noisy labels.

55

How does gradient boosting differ from AdaBoost?

Both AdaBoost and gradient boosting build an ensemble of weak learners sequentially, but they correct errors differently. AdaBoost reweights training examples, increasing the weight of misclassified points, and combines learners with accuracy-based votes; it effectively minimizes an exponential loss. Gradient boosting fits each new tree to the negative gradient of any differentiable loss (for squared error, simply the residuals), then adds it with a small learning rate. This generality, plus regularization options, makes gradient boosting (XGBoost, LightGBM, CatBoost, scikit-learn’s HistGradientBoosting) the stronger, more widely used method today.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Comparison

Aspect AdaBoost Gradient boosting
How errors are emphasized Sample weights on misclassified points New learner fits pseudo-residuals (negative gradients)
Loss Exponential (implicitly) Any differentiable loss: squared, absolute, Huber, log loss, Poisson, quantile
Base learner Stumps / shallow trees Regression trees (depth 3-8 or leaf-limited)
Learner weight Computed from weighted error Learning rate (shrinkage), plus optimal leaf values
Regularization Number of rounds, learning rate Learning rate, depth, leaves, subsampling, L1/L2 on leaves, early stopping
Noise sensitivity High (weights of outliers explode) Lower with robust losses
Tasks Mainly classification Classification, regression, ranking
Modern libraries AdaBoostClassifier XGBoost, LightGBM, CatBoost, HistGradientBoosting*

Gradient Boosting for Squared Error in Words

  1. Start with a constant prediction (the mean).

  2. Compute residuals (actual − prediction).

  3. Fit a small tree to the residuals.

  4. Add the tree’s predictions times the learning rate.

  5. Repeat; each tree corrects what is left.

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor

# Manual gradient boosting for regression
rng = np.random.default_rng(42)
x = rng.uniform(0, 6, (300, 1))
y = np.sin(x).ravel() + rng.normal(0, 0.2, 300)
pred = np.full_like(y, y.mean())
for m in range(100):
    residual = y - pred                                    # negative gradient of squared error
    tree = DecisionTreeRegressor(max_depth=2, random_state=42).fit(x, residual)
    pred += 0.1 * tree.predict(x)
    if m in (0, 9, 99):
        print(f"after {m + 1:3d} trees: training MSE {np.mean((y - pred) ** 2):.4f}")

# Compare on noisy classification data
X, yc = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.15, random_state=42)
for name, model in [("AdaBoost (stumps)", AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=300,
                                                             random_state=42)),
                    ("Gradient boosting", GradientBoostingClassifier(n_estimators=300, learning_rate=0.05,
                                                                     max_depth=3, subsample=0.8, random_state=42))]:
    print(f"{name:18s} CV accuracy {cross_val_score(model, X, yc, cv=5).mean():.3f}")

Common Pitfalls

  • A large learning rate with many trees, which overfits; use small rates with early stopping.

  • Describing AdaBoost and gradient boosting as unrelated: AdaBoost is gradient boosting with exponential loss and a specific step size.

56

How would you handle missing data in a supervised learning problem?

Handling missing data starts with understanding why values are missing: completely at random (MCAR), dependent on observed data (MAR), or dependent on the missing value itself (MNAR). Then choose a strategy: drop rows or columns only when the loss is small, impute values (mean, median, most frequent, k-NN or iterative model-based imputation), add missing-indicator features when missingness itself carries information, or use models that handle missing values natively, such as scikit-learn’s histogram gradient boosting and XGBoost. Imputation must be fitted on training data only, inside the pipeline.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Strategies

Strategy When Tool
Drop rows Few rows missing, MCAR dropna (on training data, with care)
Drop columns Mostly missing, low value Manual
Mean / median Numeric, quick baseline SimpleImputer(strategy="median")
Most frequent / constant Categorical SimpleImputer(strategy="most_frequent") or "constant"
Missing indicator Missingness is informative (“no income reported”) add_indicator=True, MissingIndicator
k-NN imputation Similar rows have similar values KNNImputer
Iterative / model-based Features predict each other IterativeImputer (experimental import)
Native support Tree boosting HistGradientBoosting*, XGBoost, LightGBM
Domain rules Known meaning (e.g. missing = 0 purchases) Explicit fill

Code Example

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.experimental import enable_iterative_imputer  # noqa: F401  (enables IterativeImputer)
from sklearn.impute import IterativeImputer, KNNImputer, SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
rng = np.random.default_rng(42)
X_missing = X.copy()
mask = rng.random(X.shape) < 0.2                          # 20% of values missing at random
X_missing[mask] = np.nan

models = {
    "median imputer + ridge": make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), Ridge()),
    "median + indicators + ridge": make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
                                                 StandardScaler(), Ridge()),
    "KNN imputer + ridge": make_pipeline(StandardScaler(), KNNImputer(n_neighbors=5), Ridge()),
    "iterative imputer + ridge": make_pipeline(IterativeImputer(random_state=42), StandardScaler(), Ridge()),
    "native NaN (gradient boosting)": HistGradientBoostingRegressor(random_state=42),
}
print("no missing data, ridge:", round(-cross_val_score(make_pipeline(StandardScaler(), Ridge()), X, y, cv=5,
                                                        scoring="neg_mean_absolute_error").mean(), 1))
for name, model in models.items():
    mae = -cross_val_score(model, X_missing, y, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name:32s} CV MAE {mae:.1f}")

StandardScaler ignores NaNs when fitting and keeps them, so it can run before KNNImputer, which needs scaled distances.

Common Pitfalls

  • Imputing with statistics from the full dataset before splitting.

  • Dropping rows when missingness is related to the target, which biases the model.

  • Imputing the target variable; drop rows with missing labels instead.

  • Forgetting that production data may have missing values in new places; test the pipeline on them.

57

What steps prepare a dataset for a supervised learning model?

Preparing a dataset turns raw records into a clean, leakage-free training table whose features will also be available at prediction time. The steps are: define the target and prediction moment, collect and join data, check data quality, split the data before fitting any transformation, clean and impute, encode categories, scale where needed, engineer features, handle imbalance, and wrap all fitted steps in a Pipeline so they are applied identically in training, validation and production.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Steps

Step What to do
1. Define the problem Target, prediction time, unit of prediction (customer, transaction), success metric
2. Collect and join Join sources using only data available before the prediction time
3. Explore and check quality Types, ranges, duplicates, missingness, label quality, class balance
4. Split Train/test (stratified, grouped or by time) before fitting anything
5. Clean Fix types, remove duplicates, handle impossible values and outliers
6. Impute Median/most-frequent/model-based; add missing indicators
7. Encode One-hot, ordinal or target encoding for categories
8. Scale Standardize for linear, distance-based and neural models
9. Engineer features Ratios, aggregates, date parts, text vectors
10. Handle imbalance Class weights, resampling inside CV
11. Package Pipeline + ColumnTransformer; version the data and code

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
    "signup_date": pd.Timestamp("2025-01-01") + pd.to_timedelta(rng.integers(0, 365, n), unit="D"),
    "plan": rng.choice(["basic", "plus", "pro", None], n, p=[0.5, 0.3, 0.15, 0.05]),
    "monthly_spend": rng.lognormal(3, 0.8, n),
    "support_tickets": rng.poisson(1.5, n).astype(float),
})
df.loc[rng.random(n) < 0.1, "monthly_spend"] = np.nan
df = pd.concat([df, df.sample(50, random_state=42)])             # duplicates to clean
y_all = ((df["support_tickets"] > 2) | (df["plan"] == "basic")).astype(int)

# Clean before splitting only what doesn't learn from data: drop exact duplicates, derive date parts
df = df.drop_duplicates()
y_all = y_all.loc[df.index]
df = df.assign(signup_month=df["signup_date"].dt.month).drop(columns="signup_date")

X_train, X_test, y_train, y_test = train_test_split(df, y_all, test_size=0.2, stratify=y_all, random_state=42)

numeric = ["monthly_spend", "support_tickets", "signup_month"]
categorical = ["plan"]
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
                          FunctionTransformer(np.log1p), StandardScaler()), numeric),
    ("cat", make_pipeline(SimpleImputer(strategy="constant", fill_value="missing"),
                          OneHotEncoder(handle_unknown="ignore")), categorical),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
print("CV ROC AUC:", round(cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean(), 3))
model.fit(X_train, y_train)
print("test ROC AUC:", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 3))

Common Pitfalls

  • Fitting imputers, scalers or encoders before the split.

  • Features created from information after the prediction moment (target leakage).

  • Duplicates of the same entity in train and test.

58

How would you handle text data in a supervised learning problem?

Text must be converted into numeric features before most supervised models can use it. The classic route is to clean and tokenize the text, then represent each document as a sparse vector of word or character n-gram counts weighted by TF-IDF, and train a linear model (logistic regression or linear SVM). The modern route uses pretrained language models, either as frozen embedding extractors feeding a simple classifier or fine-tuned end to end. Text columns often sit alongside numeric and categorical features, which a ColumnTransformer combines.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, sentence-transformers 3.x (optional) (Python 3.12)

Options

Representation How Best for
Bag of words / counts CountVectorizer Simple baselines, Naive Bayes
TF-IDF (word n-grams) TfidfVectorizer(ngram_range=(1, 2)) Strong linear baselines
Character n-grams TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5)) Typos, short texts, multiple languages
Hashing HashingVectorizer Huge vocabularies, streaming
Sentence embeddings Pretrained encoder (e.g. sentence-transformers) Semantic similarity, small labelled sets
Fine-tuned transformer BERT-style model with a classification head Highest accuracy, more compute

Code Example: Text + Structured Features

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
pos = ["fast delivery and great quality", "love it, works perfectly", "excellent support, quick refund"]
neg = ["arrived broken, very disappointed", "terrible quality, stopped working", "slow support, no refund"]
n = 600
label = rng.integers(0, 2, n)
df = pd.DataFrame({
    "review": [rng.choice(pos if l else neg) + rng.choice(["", " again", " overall", " honestly"]) for l in label],
    "channel": rng.choice(["web", "app", "store"], n),
    "order_value": rng.lognormal(3.5, 0.6, n),
})

prep = ColumnTransformer([
    ("text", TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=2), "review"),   # a column name, not a list
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["channel"]),
    ("num", StandardScaler(), ["order_value"]),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000, C=5))])
print("CV accuracy:", round(cross_val_score(model, df, label, cv=5).mean(), 3))

model.fit(df, label)
vocab = model.named_steps["prep"].named_transformers_["text"].get_feature_names_out()
coefs = model.named_steps["clf"].coef_[0][:len(vocab)]
print("most positive terms:", list(vocab[np.argsort(coefs)[-5:]]))

The synthetic reviews are easy by construction; real text needs more data and an honest held-out test set.

Preprocessing Choices

  • Lowercasing and removing markup usually help; aggressive stop-word removal can delete meaning (“not”).

  • Keep the vectorizer inside the pipeline so its vocabulary and IDF weights come only from training folds.

Common Pitfalls

  • Passing a list (["review"]) instead of a string to the text vectorizer in ColumnTransformer, which feeds it a 2D array.

  • Leakage through duplicate texts across splits.

  • Very long documents truncated by transformer models without checking where the signal is.

59

When is dimensionality reduction useful, and how is it done?

Dimensionality reduction reduces the number of input features while keeping as much useful information as possible. It is useful when there are many correlated or noisy features, when the number of features is large relative to samples, when models are slow, when distance-based methods suffer from the curse of dimensionality, and for visualizing data in 2D or 3D. It is done by feature selection (keeping a subset of original features) or feature extraction (creating new combined features, such as PCA components or learned embeddings).

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

When It Helps

Situation Benefit
Many correlated features Removes redundancy, stabilizes linear models
p close to or larger than n Reduces overfitting
k-NN, clustering, SVM in high dimensions Distances become more meaningful
Slow training or inference Fewer features to process
Visualization 2D/3D views of structure
Noisy measurements Low-variance components often contain noise

It may not help tree ensembles much, which already pick informative features, and it can hurt interpretability.

Methods

Type Method Notes
Selection Variance threshold, univariate tests (SelectKBest) Fast filters
Selection L1 regularization, RFE, tree importances Model-based
Extraction (linear) PCA, truncated SVD (sparse text) Unsupervised, keeps variance
Extraction (supervised) Linear discriminant analysis (LDA) At most classes − 1 components
Extraction (non-linear) Kernel PCA, autoencoders Complex structure
Visualization t-SNE, UMAP For plots, not usually as model inputs

Code Example

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)                    # 64 pixel features

pca = PCA().fit(StandardScaler().fit_transform(X))
n95 = int((pca.explained_variance_ratio_.cumsum() < 0.95).sum()) + 1
print(f"components for 95% variance: {n95} of {X.shape[1]}")

configs = {
    "k-NN, all 64 features": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "k-NN, PCA(20)": make_pipeline(StandardScaler(), PCA(n_components=20, random_state=42), KNeighborsClassifier()),
    "logreg, SelectKBest(30)": make_pipeline(StandardScaler(), SelectKBest(f_classif, k=30),
                                             LogisticRegression(max_iter=2000)),
    "logreg, PCA(0.95)": make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression(max_iter=2000)),
}
for name, model in configs.items():
    print(f"{name:26s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

PCA(n_components=0.95) keeps as many components as needed to explain 95% of the variance; tune the number of components with cross-validation when accuracy matters.

Common Pitfalls

  • Fitting PCA or feature selection on all data before cross-validation.

  • Forgetting to scale before PCA, so large-scale features dominate the components.

  • Assuming high-variance directions are the most predictive; supervised selection may do better.

60

Which data augmentation techniques improve supervised learning models?

Data augmentation creates new training examples by applying label-preserving transformations to existing ones, which increases data variety, reduces overfitting and makes models robust to variations they will meet in practice. Techniques depend on the data type: geometric and colour transforms for images, synonym replacement and back-translation for text, noise, time-shifting and masking for audio and time series, and SMOTE-style interpolation or noise injection for tabular data. Augmentation is applied only to training data and must not change the correct label.

Read full answer

Applies totorchvision 0.19+, scikit-learn 1.5+, NumPy 2.x (Python 3.12)

Techniques by Data Type

Data Techniques
Images Flips, rotations, random crops, scaling, colour jitter, blur, cutout/random erasing, mixup, CutMix, RandAugment
Text Synonym replacement, random insertion/deletion/swap, back-translation, paraphrasing with a language model
Audio Time shift, speed and pitch change, added noise, SpecAugment (time/frequency masking)
Time series Jittering, scaling, window slicing, time warping, magnitude warping
Tabular SMOTE for minority classes, Gaussian noise on numeric features, mixup

Code Example: Image and Tabular Augmentation

import numpy as np
import torch
from torchvision.transforms import v2

torch.manual_seed(42)

# Images: transforms applied fresh each time a training sample is loaded
train_transform = v2.Compose([
    v2.RandomResizedCrop(size=224, scale=(0.7, 1.0), antialias=True),
    v2.RandomHorizontalFlip(p=0.5),
    v2.RandomRotation(degrees=10),
    v2.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
    v2.ToDtype(torch.float32, scale=True),
    v2.RandomErasing(p=0.25),
])
test_transform = v2.Compose([v2.Resize(256, antialias=True), v2.CenterCrop(224),
                             v2.ToDtype(torch.float32, scale=True)])        # no randomness at evaluation

img = torch.randint(0, 256, (3, 256, 256), dtype=torch.uint8)
print("augmented:", train_transform(img).shape, "| test:", test_transform(img).shape)

# Mixup for a batch: blend pairs of examples and their one-hot labels
def mixup(x, y_onehot, alpha=0.2, rng=np.random.default_rng(42)):
    lam = rng.beta(alpha, alpha)
    perm = torch.randperm(x.size(0))
    return lam * x + (1 - lam) * x[perm], lam * y_onehot + (1 - lam) * y_onehot[perm]

xb, yb = torch.randn(8, 3, 32, 32), torch.nn.functional.one_hot(torch.randint(0, 10, (8,)), 10).float()
xm, ym = mixup(xb, yb)
print("mixed label rows sum to 1:", torch.allclose(ym.sum(1), torch.ones(8)))

# Tabular: Gaussian noise on standardized numeric features (training rows only)
rng = np.random.default_rng(42)
X_train = rng.normal(size=(100, 5))
X_aug = np.vstack([X_train, X_train + rng.normal(0, 0.05, X_train.shape)])
print("tabular rows before/after:", len(X_train), len(X_aug))

Guidelines

  • Choose transformations that reflect real variation: flipping street signs or digits can change their meaning.

  • Tune augmentation strength like a hyperparameter.

  • Keep validation and test data unaugmented (test-time augmentation is a separate, deliberate technique).

  • For tabular data, augmentation helps less than for images or audio; class weights or more data often matter more.

Common Pitfalls

  • Augmenting before splitting, so variants of a test example appear in training.

  • Transformations that change the label (e.g. rotating a “6” into a “9”).

61

What role does transfer learning play in supervised models?

Transfer learning reuses a model trained on a large source task (e.g. ImageNet images or web-scale text) as the starting point for a new supervised task with less labelled data. Early layers learn general features (edges, textures, word meanings) that transfer well, so you either use the pretrained network as a frozen feature extractor with a new classifier on top, or fine-tune some or all of its layers at a low learning rate. It dramatically reduces the data, compute and time needed and is the default approach for images, text and audio today.

Read full answer

Applies toPyTorch 2.x, torchvision 0.19+ (Python 3.12)

Strategies

Strategy What is trained Labelled data Notes
Feature extraction New head only (backbone frozen) Small (hundreds) Fast, low overfitting risk
Partial fine-tuning Head + top layers Medium Adapts high-level features
Full fine-tuning All layers, low learning rate Larger Best accuracy if data allows
Parameter-efficient (LoRA, adapters) Small added modules Small to medium Common for large language models
Domain-adaptive pretraining Continue pretraining on unlabelled in-domain data Unlabelled data Bridges domain gaps

When It Helps Most

  • Small labelled datasets.

  • Source and target domains are related (natural images → product photos).

  • Less helpful when domains differ greatly (natural images → spectrograms), though pretrained weights are still often better than random.

Code Example: Feature Extraction Then Fine-Tuning

import torch
from torch import nn
from torchvision import models

torch.manual_seed(42)
NUM_CLASSES = 5

model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)        # downloads ImageNet weights
for p in model.parameters():
    p.requires_grad = False                                           # phase 1: freeze the backbone
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)               # new trainable head

head_opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
x, y = torch.randn(16, 3, 224, 224), torch.randint(0, NUM_CLASSES, (16,))   # stand-in batch
loss = nn.functional.cross_entropy(model(x), y)
loss.backward()
head_opt.step()

# Phase 2: unfreeze the last block and fine-tune with a lower learning rate for the backbone
for p in model.layer4.parameters():
    p.requires_grad = True
optimizer = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
])
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable parameters: {trainable:,} of {sum(p.numel() for p in model.parameters()):,}")

In real training, loop over a DataLoader with the pretrained model’s preprocessing (ResNet18_Weights.DEFAULT.transforms()), and validate after each epoch.

Common Pitfalls

  • Using different input preprocessing from the pretraining (resolution, normalization).

  • Fine-tuning everything with a high learning rate, which destroys pretrained features.

  • BatchNorm statistics drifting when fine-tuning with tiny batches; keep those layers in eval mode or freeze them.

62

What is end-to-end learning in deep learning?

End-to-end learning trains a single model to map raw inputs directly to the final output, optimizing all components jointly with one loss, instead of chaining separately designed stages such as hand-crafted feature extraction followed by a classifier. Examples include speech recognition from audio waveforms or spectrograms straight to text, image classification from pixels, and translation from source to target sentences. It removes manual feature engineering and lets every layer adapt to the task, but it needs much more labelled data and compute, and it is harder to debug and interpret.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

Pipeline Versus End-to-End

Aspect Traditional pipeline End-to-end
Stages Separate (features → model → post-processing) One differentiable model
Feature design Manual, domain expertise Learned
Optimization Each stage separately Jointly for the final loss
Data needs Lower High (or pretrained models)
Debugging Inspect each stage Harder; intermediate representations are implicit
Example (speech) MFCC features → acoustic model → pronunciation lexicon → language model Audio → transformer → characters/tokens

When It Works Well

  • Large labelled datasets, or strong pretrained models to fine-tune.

  • Tasks where hand-crafted features lose information (vision, speech, language).

  • When intermediate labels are unavailable or unnecessary.

When a Pipeline Is Better

  • Little data, but good domain knowledge for features.

  • Requirements for interpretability or verifiable intermediate steps (e.g. safety-critical systems).

  • When intermediate outputs (detected objects, recognized entities) are useful in their own right.

Code Example: Learned Features vs Hand-Crafted Features

import numpy as np
import torch
from sklearn.linear_model import LogisticRegression
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)

# Signals: class 1 contains a short burst at a random position
def make(n):
    x = rng.normal(0, 1, (n, 128)).astype("float32")
    y = rng.integers(0, 2, n)
    for i in np.where(y == 1)[0]:
        s = rng.integers(0, 112)
        x[i, s:s + 16] += np.sin(np.linspace(0, 4 * np.pi, 16)) * 2
    return x, y

X_tr, y_tr = make(2000)
X_te, y_te = make(500)

# Pipeline: hand-crafted features (mean, std, max) + classifier
feats = lambda X: np.c_[X.mean(1), X.std(1), X.max(1)]
pipe = LogisticRegression().fit(feats(X_tr), y_tr)
print("hand-crafted features accuracy:", round(pipe.score(feats(X_te), y_te), 3))

# End-to-end: a 1D CNN learns its own features from raw samples
model = nn.Sequential(nn.Conv1d(1, 16, 9, padding=4), nn.ReLU(), nn.AdaptiveMaxPool1d(1),
                      nn.Flatten(), nn.Linear(16, 1))
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
xt, yt = torch.tensor(X_tr)[:, None], torch.tensor(y_tr, dtype=torch.float32)
for epoch in range(200):
    opt.zero_grad()
    loss = nn.functional.binary_cross_entropy_with_logits(model(xt).squeeze(1), yt)
    loss.backward()
    opt.step()
with torch.no_grad():
    acc = ((model(torch.tensor(X_te)[:, None]).squeeze(1) > 0).numpy() == y_te).mean()
print("end-to-end CNN accuracy:", round(float(acc), 3))

The CNN learns a filter that matches the burst shape, which the summary statistics capture only partly.

Common Pitfalls

  • Expecting end-to-end models to work with little data and no pretraining.

  • Losing useful structure (e.g. known physics or rules) that a hybrid approach could keep.

63

What is multitask learning, and what are its benefits?

Multitask learning trains one model to perform several related tasks at once, usually with a shared representation (common layers) and task-specific heads, optimizing a combined loss. Learning tasks together lets each benefit from the others’ signal: shared features become more general, the model is regularized against overfitting to one task, and a single model is cheaper to serve than several. Examples include predicting both click and purchase in recommendations, or detecting objects and estimating depth from the same image.

Read full answer

Applies toPyTorch 2.x, NumPy 2.x (Python 3.12)

Benefits and Risks

Benefits Risks
Better generalization through shared representations Negative transfer when tasks conflict
Data efficiency: a data-rich task helps a data-poor one Loss balancing is tricky (tasks on different scales)
Regularization: harder to overfit any single task Gradients from tasks can point in opposite directions
One model to train, deploy and run More complex evaluation and debugging

Architectures

Type Description
Hard parameter sharing Shared backbone + separate heads (most common)
Soft parameter sharing Separate networks with constraints that keep weights similar
Mixture-of-experts (e.g. MMoE) Shared experts with task-specific gating (recommender systems)

Loss Weighting

Total loss = sum of w_t × loss_t. Weights can be fixed and tuned, normalized by each loss’s scale, or learned (e.g. uncertainty weighting). Gradient-surgery methods such as PCGrad reduce conflicts between tasks.

Code Example: Shared Network with Two Heads

import numpy as np
import torch
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)
n = 4000
X = rng.normal(size=(n, 20)).astype("float32")
shared_signal = X[:, :5].sum(axis=1)
y_click = (shared_signal + rng.normal(0, 1, n) > 0).astype("float32")         # classification task
y_spend = (2 * shared_signal + X[:, 5] + rng.normal(0, 1, n)).astype("float32")  # regression task

class MultiTask(nn.Module):
    def __init__(self):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 32), nn.ReLU())
        self.click_head = nn.Linear(32, 1)
        self.spend_head = nn.Linear(32, 1)

    def forward(self, x):
        h = self.shared(x)
        return self.click_head(h).squeeze(1), self.spend_head(h).squeeze(1)

model = MultiTask()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
Xt, yc, ys = torch.tensor(X[:3000]), torch.tensor(y_click[:3000]), torch.tensor(y_spend[:3000])
for epoch in range(300):
    opt.zero_grad()
    click_logit, spend_pred = model(Xt)
    loss = (nn.functional.binary_cross_entropy_with_logits(click_logit, yc)
            + 0.1 * nn.functional.mse_loss(spend_pred, ys))                     # weight balances scales
    loss.backward()
    opt.step()

with torch.no_grad():
    c, s = model(torch.tensor(X[3000:]))
    print("click accuracy:", round(((c > 0).float() == torch.tensor(y_click[3000:])).float().mean().item(), 3))
    print("spend MAE:", round((s - torch.tensor(y_spend[3000:])).abs().mean().item(), 3))

Compare against single-task models trained the same way to check that sharing actually helps.

Common Pitfalls

  • Summing losses with very different scales so one task dominates.

  • Grouping unrelated tasks, which causes negative transfer.

64

How can reinforcement learning be framed as a supervised learning problem?

Reinforcement learning (RL) can be partly recast as supervised learning whenever you can create (input, target) pairs from experience. The main framings are: behaviour cloning, which learns to predict an expert’s action from the state (classification or regression); fitted value iteration, where a regressor learns Q-values from bootstrapped targets r + γ max Q(s', a') computed on logged transitions (fitted Q-iteration, and DQN’s update is a regression step); reward-weighted or return-conditioned learning (e.g. Decision Transformer), which predicts actions conditioned on desired returns; and learning environment models (predict next state and reward). The supervised view simplifies training but must still handle distribution shift, because the agent’s own actions change the data it sees.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Framings

Framing Input → target Limitation
Behaviour cloning State → expert action Compounding errors in unseen states (DAgger helps)
Fitted Q-iteration / DQN updates (State, action) → r + γ max Q(next) Targets move; needs stable training tricks
Return-conditioned policies (State, desired return) → action Needs diverse offline data
Model learning (State, action) → next state, reward Model errors compound in planning
Reward modelling (RLHF) Pair of outputs → which is preferred Reward hacking if the model is wrong

Code Example: Fitted Q-Iteration on a Small Chain

States 0-9 on a line; actions left/right; reaching state 9 gives reward 1. A regressor learns Q-values from a fixed batch of random transitions.

import numpy as np
from sklearn.ensemble import ExtraTreesRegressor

rng = np.random.default_rng(42)
N, GAMMA = 10, 0.9

# Logged random transitions (offline data)
s = rng.integers(0, N - 1, 5000)
a = rng.integers(0, 2, 5000)                               # 0 = left, 1 = right
s_next = np.clip(s + np.where(a == 1, 1, -1), 0, N - 1)
r = (s_next == N - 1).astype(float)
done = s_next == N - 1

X = np.c_[s, a]
q_model = None
for iteration in range(30):
    if q_model is None:
        target = r                                         # first iteration: immediate reward
    else:
        q_next = np.max([q_model.predict(np.c_[s_next, np.full_like(s_next, act)]) for act in (0, 1)], axis=0)
        target = r + GAMMA * q_next * (~done)              # supervised target from the Bellman equation
    q_model = ExtraTreesRegressor(n_estimators=50, random_state=42).fit(X, target)

policy = [int(np.argmax([q_model.predict([[state, act]])[0] for act in (0, 1)])) for state in range(N - 1)]
print("greedy action per state (1 = right):", policy)
print("Q(0, right) ≈", round(q_model.predict([[0, 1]])[0], 3), "| theory:", round(GAMMA ** 8, 3))

Each iteration is an ordinary regression problem; the learned greedy policy should move right everywhere, and Q(0, right) should approach γ⁸ because the reward arrives after nine steps.

Common Pitfalls

  • Treating behaviour cloning as solved by high validation accuracy; evaluate with rollouts.

  • Offline data that never visits important states, which no regressor can fix.

65

What role do attention mechanisms play in neural networks?

Attention mechanisms let a neural network decide, for each position or output, which parts of the input to focus on, by computing weights over all inputs and taking a weighted combination of them. Introduced for machine translation to overcome the fixed-length bottleneck of RNN encoders, attention became the core of the Transformer, where self-attention relates every token to every other token in parallel. It captures long-range dependencies, handles variable-length inputs, and powers modern language models, vision transformers and multimodal models; attention weights also give some insight into what the model uses.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

How Scaled Dot-Product Attention Works

  1. Each input is projected into a query, a key and a value vector.

  2. Score each query against every key with a dot product, divided by the square root of the key dimension.

  3. Apply softmax so the scores become weights that sum to 1.

  4. Output = weighted sum of the values.

  5. Multi-head attention runs several of these in parallel with different projections and concatenates the results.

Types

Type Queries from Keys/values from Use
Self-attention Same sequence Same sequence Transformer encoders/decoders
Cross-attention Decoder Encoder output Translation, image captioning
Causal (masked) self-attention Sequence Earlier positions only Language model generation
Additive (Bahdanau) Decoder state Encoder states Early seq2seq models

Code Example

import torch
import torch.nn.functional as F
from torch import nn

torch.manual_seed(42)
batch, seq_len, d_model = 2, 6, 16
x = torch.randn(batch, seq_len, d_model)

# Manual scaled dot-product self-attention
Wq, Wk, Wv = (nn.Linear(d_model, d_model, bias=False) for _ in range(3))
q, k, v = Wq(x), Wk(x), Wv(x)
scores = q @ k.transpose(-2, -1) / d_model ** 0.5           # (batch, seq, seq)
causal = torch.triu(torch.ones(seq_len, seq_len, dtype=torch.bool), diagonal=1)
weights = scores.masked_fill(causal, float("-inf")).softmax(dim=-1)
manual = weights @ v
print("first row of weights (only attends to itself):", weights[0, 0].detach().numpy().round(2))

# PyTorch's fused implementation gives the same result
fused = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print("matches fused version:", torch.allclose(manual, fused, atol=1e-5))

# Multi-head attention module
mha = nn.MultiheadAttention(embed_dim=d_model, num_heads=4, batch_first=True)
out, attn = mha(x, x, x, need_weights=True)
print("output:", tuple(out.shape), "| averaged attention weights:", tuple(attn.shape))

Why It Matters

  • Direct connections between any two positions: no information has to pass through many recurrent steps.

  • Parallel computation across positions, unlike RNNs.

  • Cost grows with the square of sequence length, which motivates efficient variants (FlashAttention, sparse and linear attention).

Common Pitfalls

  • Forgetting padding or causal masks.

  • Reading attention weights as full explanations; they show where information flows, not necessarily why a prediction was made.

66

How would you design a supervised model to predict customer churn?

A churn model predicts which customers are likely to stop using a product within a defined window, so the business can intervene. A good design starts with a precise churn definition and prediction time, builds features from behaviour up to that time (usage trends, payments, support contacts, tenure, plan), trains a classifier (logistic regression and gradient boosting are strong choices), validates on a later time period, and turns predicted probabilities into actions by targeting the customers where retention offers have the highest expected value.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Design

Element Decision
Label e.g. “cancels or has no activity in the next 30 days”
Snapshot Features computed as of a cut-off date; label from the window after it
Features Tenure, plan, price changes, usage last 7/30/90 days and trend, failed payments, support tickets, NPS
Model Logistic regression baseline, gradient boosting
Validation Train on earlier snapshots, test on a later one
Metrics ROC AUC, PR AUC, lift/precision in the top decile, calibration
Action Rank customers; choose who to contact using expected value
Monitoring Drift in features and churn rate; retrain regularly

Code Example

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, roc_auc_score

rng = np.random.default_rng(42)
rows = []
for snapshot in range(6):                                   # monthly snapshots
    n = 3000
    tenure = rng.integers(1, 60, n)
    usage_30 = rng.gamma(2, 10, n)
    usage_trend = rng.normal(0, 1, n)                       # change vs previous month (standardized)
    tickets = rng.poisson(0.5, n)
    failed_pay = rng.binomial(1, 0.05, n)
    plan = rng.choice(["monthly", "annual"], n, p=[0.7, 0.3])
    logit = (-2 - 0.03 * tenure - 0.03 * usage_30 - 0.8 * usage_trend + 0.4 * tickets
             + 1.5 * failed_pay + 1.0 * (plan == "monthly"))
    churn = rng.random(n) < 1 / (1 + np.exp(-logit))
    rows.append(pd.DataFrame({"snapshot": snapshot, "tenure": tenure, "usage_30": usage_30,
                              "usage_trend": usage_trend, "tickets": tickets, "failed_pay": failed_pay,
                              "monthly_plan": (plan == "monthly").astype(int), "churn": churn.astype(int)}))
df = pd.concat(rows, ignore_index=True)

features = ["tenure", "usage_30", "usage_trend", "tickets", "failed_pay", "monthly_plan"]
train, test = df[df["snapshot"] < 5], df[df["snapshot"] == 5]            # out-of-time validation
model = HistGradientBoostingClassifier(max_depth=4, random_state=42).fit(train[features], train["churn"])
p = model.predict_proba(test[features])[:, 1]

print("churn rate:", round(test["churn"].mean(), 3))
print("ROC AUC:", round(roc_auc_score(test["churn"], p), 3), "| PR AUC:", round(average_precision_score(test["churn"], p), 3))
top = test.assign(p=p).nlargest(len(test) // 10, "p")
print("churn rate in top 10% of scores:", round(top["churn"].mean(), 3), "(lift",
      round(top["churn"].mean() / test["churn"].mean(), 1), "x)")

# Expected-value targeting: contact if p * save_rate * customer_value > offer_cost
save_rate, value, cost = 0.3, 200, 15
print("customers worth contacting:", int((p * save_rate * value > cost).sum()), "of", len(test))

Common Pitfalls

  • Leakage from features recorded after the cut-off (e.g. “cancellation reason”).

  • Random splits across snapshots of the same customer.

  • Predicting churn but not whether an intervention changes it; uplift modelling targets persuadable customers.

67

What is your strategy for building a supervised sentiment analysis model?

A supervised sentiment model is built by defining the label scheme (positive/negative, or with neutral, or a 1-5 rating), collecting and annotating representative text from the target domain, establishing a fast baseline (TF-IDF with logistic regression), then fine-tuning a pretrained transformer if the baseline is not good enough. Evaluation uses macro F1 on a held-out set from the same domain, with error analysis on negation, sarcasm and mixed opinions, followed by monitoring after deployment because language and topics drift.

Read full answer

Applies toscikit-learn 1.5+, Hugging Face Transformers 4.4x (optional) (Python 3.12)

Strategy

Step Details
1. Labels Clear guidelines with examples; decide how to treat mixed or neutral text
2. Data Domain-matched texts; double-annotate a sample to measure agreement (Cohen’s kappa)
3. Split Stratified; de-duplicate; split by user or product if they repeat
4. Baseline TF-IDF (word 1-2 grams + char n-grams) + logistic regression
5. Stronger model Fine-tune a pretrained encoder (e.g. a RoBERTa or DeBERTa checkpoint)
6. Evaluation Macro F1, per-class recall, confusion matrix, slices (short texts, negations)
7. Error analysis Inspect misclassified examples; fix labels or add data
8. Deployment Batch or API; monitor class distribution and sampled accuracy

Code Example: Baseline with Error Analysis

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import FeatureUnion, make_pipeline

rng = np.random.default_rng(42)
pos = ["great", "excellent", "love", "fantastic", "works well"]
neg = ["terrible", "awful", "hate", "broken", "waste of money"]
templates = ["the product is {}", "{} experience overall", "honestly {}", "really {} purchase"]
texts, labels = [], []
for _ in range(1500):
    label = rng.integers(0, 2)
    word = rng.choice(pos if label else neg)
    text = rng.choice(templates).format(word)
    if rng.random() < 0.15:                                 # negation flips the sentiment
        text, label = "not " + text, 1 - label
    texts.append(text)
    labels.append(label)

X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.25, stratify=labels, random_state=42)
features = FeatureUnion([("words", TfidfVectorizer(ngram_range=(1, 2))),
                         ("chars", TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)))])
model = make_pipeline(features, LogisticRegression(max_iter=2000, C=5)).fit(X_train, y_train)
pred = model.predict(X_test)
print(classification_report(y_test, pred, target_names=["negative", "positive"], digits=3))

errors = [t for t, p, y in zip(X_test, pred, y_test) if p != y]
print("examples of errors:", errors[:5])
print("share of errors containing 'not':", round(np.mean(["not" in e for e in errors]), 2) if errors else 0)

The synthetic data shows the typical weak spot: negated phrases. Bigrams such as “not great” help a linear model; transformers handle negation better.

Common Pitfalls

  • Training on product reviews and deploying on tweets or support chats.

  • Removing stop words like “not” and “no”.

  • Treating neutral or mixed text as noise instead of defining it in the label scheme.

68

What supervised learning approach would you propose for transaction fraud detection?

Transaction fraud detection can be framed as supervised binary classification: predict, at authorization time, the probability that a transaction is fraudulent, using labels from confirmed chargebacks and investigations. The proposal: rich behavioural features (amount relative to the customer’s history, velocity counts, merchant and device risk, geography), a gradient-boosted tree model with class weighting, strict time-based validation, and a decision threshold chosen from the cost of missed fraud versus declined good customers. Because fraud patterns change and labels arrive late, combine it with rules and anomaly detection, and retrain frequently.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Design

Element Choice
Label Fraud confirmed within N days (chargeback, investigation)
Features Amount, amount / customer’s average, transactions in last 1h/24h, new device, merchant category risk, distance from usual location, time of day
Model Gradient boosting (HistGradientBoostingClassifier, XGBoost, LightGBM)
Imbalance Class weights; evaluate with PR AUC
Validation Train on earlier weeks, test on later weeks
Threshold Minimize expected cost: missed fraud amount vs friction cost of false declines
Latency Features from a real-time store; model scoring in milliseconds
Feedback Label delay; analyst reviews; frequent retraining; drift monitoring

Code Example

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score

rng = np.random.default_rng(42)
n = 60000
df = pd.DataFrame({
    "week": np.sort(rng.integers(0, 12, n)),
    "amount": rng.lognormal(3.5, 1.0, n),
    "amount_vs_avg": rng.lognormal(0, 0.5, n),
    "tx_last_hour": rng.poisson(0.3, n),
    "new_device": rng.binomial(1, 0.05, n),
    "foreign": rng.binomial(1, 0.08, n),
})
logit = (-7 + 1.2 * np.log(df["amount_vs_avg"] + 0.1) + 0.8 * df["tx_last_hour"]
         + 2.0 * df["new_device"] + 1.2 * df["foreign"])
df["fraud"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

features = ["amount", "amount_vs_avg", "tx_last_hour", "new_device", "foreign"]
train, test = df[df["week"] < 9], df[df["week"] >= 9]                # time-based split
model = HistGradientBoostingClassifier(class_weight="balanced", max_depth=4, random_state=42)
model.fit(train[features], train["fraud"])
p = model.predict_proba(test[features])[:, 1]
print("fraud rate:", round(test["fraud"].mean(), 4), "| PR AUC:", round(average_precision_score(test["fraud"], p), 3))

# Choose the threshold that minimizes expected cost on the test weeks (use a validation period in practice)
fraud_amount = test["amount"].to_numpy()
best = min(
    ((t, (fraud_amount * ((p < t) & (test["fraud"] == 1))).sum() + 5.0 * ((p >= t) & (test["fraud"] == 0)).sum())
     for t in np.linspace(0.05, 0.95, 19)),
    key=lambda pair: pair[1])
print(f"threshold {best[0]:.2f} with expected cost {best[1]:,.0f}")

The cost uses the transaction amount for missed fraud and a fixed 5.0 friction cost for each wrongly declined good transaction.

Common Pitfalls

  • Random splits that leak future fraud patterns into training.

  • Features computed with information from after the transaction (e.g. later chargeback counts).

  • Optimizing ROC AUC alone when fraud is extremely rare.

69

How would you build a predictive maintenance system with supervised learning?

A predictive maintenance system uses sensor and operational data to predict equipment failures early enough to schedule maintenance before breakdowns. With supervised learning, you label historical data with failure events and frame either classification (“will this machine fail in the next N days?”) or regression (“remaining useful life, RUL”). The design centres on feature engineering over time windows (rolling statistics, trends), labelling relative to failure times, splitting by machine and time, and choosing alert thresholds that balance missed failures against unnecessary maintenance.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Design

Element Choice
Data Sensor time series (vibration, temperature, pressure), operating settings, maintenance logs, failure records
Label 1 if a failure occurs within the next N cycles/days (classification) or cycles remaining (RUL regression)
Features Rolling mean, std, min/max, slopes over several windows; time since last maintenance; usage counters
Split Hold out whole machines, and later time periods
Model Gradient boosting or random forest; sequence models (1D CNN/LSTM) with lots of data
Metrics Recall of failures at an acceptable false-alarm rate; lead time; for RUL, MAE weighted toward late predictions
Action Maintenance work orders when risk exceeds the threshold

Code Example

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, recall_score
from sklearn.model_selection import GroupShuffleSplit

rng = np.random.default_rng(42)
frames = []
for machine in range(120):
    life = rng.integers(150, 300)                             # cycles until failure
    t = np.arange(life)
    wear = (t / life) ** 3                                    # degradation accelerates near the end
    frames.append(pd.DataFrame({
        "machine": machine, "cycle": t,
        "vibration": 1 + 2 * wear + rng.normal(0, 0.15, life),
        "temperature": 60 + 15 * wear + rng.normal(0, 1.0, life),
        "fail_within_20": (life - t <= 20).astype(int),
    }))
df = pd.concat(frames, ignore_index=True)

# Rolling features per machine (past values only)
g = df.groupby("machine")
for col in ("vibration", "temperature"):
    df[f"{col}_mean10"] = g[col].transform(lambda s: s.rolling(10, min_periods=1).mean())
    df[f"{col}_slope10"] = g[col].transform(lambda s: s.diff(10) / 10)
df = df.dropna()

features = [c for c in df.columns if c.endswith(("mean10", "slope10"))] + ["vibration", "temperature"]
split = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42)
train_idx, test_idx = next(split.split(df, groups=df["machine"]))          # unseen machines in test
train, test = df.iloc[train_idx], df.iloc[test_idx]

model = HistGradientBoostingClassifier(class_weight="balanced", random_state=42)
model.fit(train[features], train["fail_within_20"])
p = model.predict_proba(test[features])[:, 1]
print("PR AUC:", round(average_precision_score(test["fail_within_20"], p), 3))
print("recall of pre-failure cycles at threshold 0.5:", round(recall_score(test["fail_within_20"], (p >= 0.5).astype(int)), 3))

# Lead time: first alert before failure for each test machine
test = test.assign(alert=p >= 0.5)
lead = test[test["alert"]].groupby("machine")["cycle"].min()
life_end = test.groupby("machine")["cycle"].max()
print("median warning (cycles before failure):", float((life_end.loc[lead.index] - lead).median()))

Common Pitfalls

  • Random row splits, so the same machine’s neighbouring cycles appear in train and test.

  • Rolling features that include future readings (centred windows).

  • Too few failures: consider anomaly detection or survival models, and collect more labelled failure data.

70

How would you build a model to automate medical image diagnosis?

Building a medical image diagnosis model starts with a precise clinical task (e.g. detect pneumonia on chest X-rays, or grade diabetic retinopathy), a dataset with reliable reference-standard labels, and patient-level splits with an external test set from other hospitals. The usual model is a pretrained CNN or vision transformer fine-tuned on the images, with augmentation suited to the modality. Evaluation focuses on sensitivity, specificity, ROC AUC and calibration, subgroup performance and failure analysis; deployment is as decision support with human review, monitoring and, for clinical use, regulatory clearance.

Read full answer

Applies toPyTorch 2.x, torchvision 0.19+, scikit-learn 1.5+ (Python 3.12)

Plan

Stage Decisions
Task definition Findings, output (binary, multi-label, segmentation), intended use and user
Data DICOM images, metadata; labels from reports, expert consensus or pathology; de-identification
Splits By patient; external validation set from different sites and scanners
Preprocessing Windowing (CT), resizing, intensity normalization; keep aspect and laterality
Model Pretrained backbone (ResNet, EfficientNet, ViT) fine-tuned; segmentation with U-Net if localization is needed
Imbalance Weighted loss, balanced sampling
Metrics Sensitivity at fixed specificity, ROC AUC, PR AUC, calibration, per-subgroup results
Explainability Saliency maps (e.g. Grad-CAM) for review, not as proof
Deployment PACS integration, human-in-the-loop, drift monitoring, regulatory process

Code Example: Fine-Tuning Skeleton

import torch
from torch import nn
from torchvision import models
from torchvision.transforms import v2

torch.manual_seed(42)

train_tf = v2.Compose([
    v2.RandomResizedCrop(224, scale=(0.85, 1.0), antialias=True),
    v2.RandomRotation(7),                                    # small rotations only; no flips for chest X-rays
    v2.ColorJitter(brightness=0.1, contrast=0.1),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

model = models.densenet121(weights=models.DenseNet121_Weights.DEFAULT)     # downloads ImageNet weights
model.classifier = nn.Linear(model.classifier.in_features, 1)              # binary finding
pos_weight = torch.tensor([4.0])                                           # e.g. negatives/positives ratio
loss_fn = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=1e-4)

# One training step on a stand-in batch (grayscale X-rays repeated to 3 channels)
images = torch.randint(0, 256, (8, 3, 256, 256), dtype=torch.uint8)
labels = torch.tensor([0, 1, 0, 0, 1, 0, 0, 0], dtype=torch.float32)
batch = torch.stack([train_tf(img) for img in images])
model.train()
loss = loss_fn(model(batch).squeeze(1), labels)
loss.backward()
optimizer.step()
print("loss:", round(loss.item(), 4))

In a real project, data loaders read DICOM files, patients are split with GroupShuffleSplit, and the threshold is chosen on validation data to reach the required sensitivity.

Common Pitfalls

  • Image-level instead of patient-level splits.

  • Shortcut learning from text markers, devices or hospital-specific artefacts.

  • Labels mined from reports without checking their accuracy.

  • Reporting internal validation only.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 70 Supervised Learning answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects