Supervised learning is the branch of machine learning in which a model learns a mapping from inputs (features) to known outputs (labels) using examples where the correct answer is given. After training, the model predicts labels for new, unseen inputs. It splits into classification, where the output is a category (spam or not spam), and regression, where the output is a number (house price). Most production ML, from credit scoring to medical diagnosis, is supervised.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Key Ideas
Concept
Meaning
Features X
Measurable inputs (age, pixel values, word counts)
Label y
The answer to learn (class or number)
Training
Adjusting model parameters to reduce a loss on labelled examples
Generalization
Performing well on new data, not just the training set
Evaluation
Measuring performance on held-out data with suitable metrics
Workflow
Collect and label representative data.
Split into training and test sets (and use cross-validation for tuning).
Train candidate models and tune hyperparameters with cross-validation.
Evaluate once on the test set.
Deploy and monitor for drift; retrain as needed.
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True) # features and labels (malignant/benign)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
baseline = DummyClassifier(strategy="most_frequent")
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
for name, est in [("baseline", baseline), ("logistic regression", model)]:
print(f"{name:20s} CV accuracy {cross_val_score(est, X_train, y_train, cv=5).mean():.3f}")
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), target_names=["malignant", "benign"]))
Compared with Other Paradigms
Paradigm
Labels
Example
Supervised
Full labels
Predict churn from customer history
Unsupervised
None
Cluster customers into segments
Semi-supervised
Few labels + many unlabelled
Label propagation for documents
Self-supervised
Labels created from the data itself
Masked-word pretraining of language models
Reinforcement
Rewards from interaction
Game-playing agents
Common Pitfalls
Poor label quality: noisy or inconsistent labels cap the achievable accuracy.
Data leakage: features that contain information unavailable at prediction time.
Unrepresentative data: training data that differs from the data the model will see.
2
What is the difference between supervised and unsupervised learning?
KnownRevise
Supervised learning trains on labelled examples, inputs paired with the correct outputs, and learns to predict those outputs for new inputs. Unsupervised learning has no labels; it finds structure in the inputs alone, such as clusters, lower-dimensional representations, or unusual points. Supervised models are judged by how well their predictions match known answers, while unsupervised results are judged by internal measures, usefulness for a downstream task, or human inspection.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Comparison
Aspect
Supervised
Unsupervised
Data
Features + labels
Features only
Goal
Predict a target
Discover structure
Tasks
Classification, regression
Clustering, dimensionality reduction, anomaly detection, density estimation
Silhouette, reconstruction error, business usefulness
Cost
Labelling can be expensive
Data is cheap, interpretation is harder
Example
Predict whether a customer will churn
Group customers into segments
In Between
Semi-supervised: a few labels plus many unlabelled examples (label propagation, pseudo-labelling).
Self-supervised: labels created from the data itself (predicting masked words), the basis of modern foundation models.
Unsupervised as preprocessing: PCA features or cluster IDs used as inputs to a supervised model.
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
# Supervised: uses the species labels to learn a classifier
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
print("supervised CV accuracy:", round(cross_val_score(clf, X, y, cv=5).mean(), 3))
# Unsupervised: groups the flowers without seeing labels
X_scaled = StandardScaler().fit_transform(X)
clusters = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X_scaled)
print("silhouette (internal quality):", round(silhouette_score(X_scaled, clusters), 3))
print("agreement with true species (ARI):", round(adjusted_rand_score(y, clusters), 3))
The adjusted Rand index is only possible here because the iris labels happen to be known; in real unsupervised work there are usually no labels to compare against.
Choosing
Use supervised learning when you know what to predict and can obtain labels.
Use unsupervised learning to explore data, create features, detect anomalies or when labels are unavailable.
Common Pitfalls
Expecting clusters to match a specific business category without supervision.
Evaluating a supervised model on training data, or an unsupervised one only by eye.
3
What types of problems can supervised learning solve?
KnownRevise
Supervised learning solves any problem where you can collect examples of inputs paired with the correct outputs and want to predict that output for new inputs. The main families are classification (predict a category: binary, multi-class or multi-label), regression (predict a number), and structured variants such as ranking, sequence labelling, object detection and forecasting. The type of target decides the model’s output layer, loss function and evaluation metrics.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Problem Types
Type
Target
Examples
Typical metrics
Binary classification
Two classes
Spam detection, loan default, disease yes/no
ROC AUC, F1, precision/recall
Multi-class classification
One of K classes
Digit recognition, product category
Accuracy, macro F1
Multi-label classification
Several labels at once
Tagging articles with topics
Micro/macro F1, Hamming loss
Regression
Continuous value
House price, demand, delivery time
MAE, RMSE, R²
Ordinal regression
Ordered categories
Star ratings, severity grades
MAE on ranks, weighted kappa
Ranking
Order of items
Search results, recommendations
NDCG, MAP
Sequence labelling
Label per token
Named-entity recognition, part-of-speech tagging
Entity-level F1
Object detection / segmentation
Boxes or pixel masks
Self-driving perception, medical imaging
mAP, IoU, Dice
Time-series forecasting
Future values
Sales, energy load
MAE, MASE
Survival analysis
Time until an event
Customer lifetime, equipment failure
Concordance index
Code Example: Same Data, Different Problem Types
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import cross_val_score
X, y = load_diabetes(return_X_y=True) # target: disease progression score
# Regression: predict the score itself
reg = RandomForestRegressor(n_estimators=200, random_state=42)
print("regression MAE:", round(-cross_val_score(reg, X, y, cv=5, scoring="neg_mean_absolute_error").mean(), 1))
# Binary classification: is progression above the median?
y_bin = (y > np.median(y)).astype(int)
clf = RandomForestClassifier(n_estimators=200, random_state=42)
print("binary ROC AUC:", round(cross_val_score(clf, X, y_bin, cv=5, scoring="roc_auc").mean(), 3))
# Multi-class: low / medium / high progression (tertiles)
y_multi = np.digitize(y, np.quantile(y, [1 / 3, 2 / 3]))
print("3-class macro F1:", round(cross_val_score(clf, X, y_multi, cv=5, scoring="f1_macro").mean(), 3))
Turning a numeric target into classes discards information, so do it only when the decision really is categorical.
Choosing the Framing
Start from the decision the prediction supports: a yes/no action suggests classification; a quantity to plan with suggests regression.
Check that labels can be collected reliably and that the input features will be available at prediction time.
Common Pitfalls
Treating ordered categories as unordered classes, or as exact numbers when gaps are uneven.
Using accuracy for imbalanced or multi-label problems.
4
How are training and testing datasets used in supervised learning?
KnownRevise
The training set is the labelled data a model learns from, and the test set is separate labelled data held back to estimate how the model will perform on new, unseen examples. Because a model can memorize its training data, performance must be measured on data it never saw. In practice there is usually a third role, validation, for choosing models and hyperparameters, done with a separate validation set or with cross-validation on the training data, so the test set is used only once at the end.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Roles
Set
Used for
Typical share
Training
Fitting model parameters
60-80%
Validation (or CV folds)
Choosing hyperparameters, features, model type, early stopping
10-20%, or k-fold CV within training
Test
Final, unbiased estimate of performance
10-20%; touched once
Splitting Strategies
Data
Strategy
i.i.d. tabular, balanced
Random split
Imbalanced classes
Stratified split (stratify=y)
Time series
Chronological split; train on past, test on future
Grouped data (patients, users)
Group split so the same group is not in both sets
Small data
k-fold cross-validation, nested CV for unbiased tuning
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
X, y = load_breast_cancer(return_X_y=True)
# 1. Hold out a test set first and do not look at it until the end
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
# 2. Validation happens inside the training data via 5-fold cross-validation
search = GridSearchCV(RandomForestClassifier(random_state=42),
{"max_depth": [3, 5, None], "min_samples_leaf": [1, 5]}, cv=5)
search.fit(X_train, y_train)
print("best params:", search.best_params_, "| CV accuracy:", round(search.best_score_, 3))
# 3. Final evaluation, once
print("training accuracy:", round(search.score(X_train, y_train), 3))
print("test accuracy: ", round(search.score(X_test, y_test), 3))
A training accuracy much higher than the test accuracy indicates overfitting.
Common Pitfalls
Leakage: fitting scalers, imputers or feature selection on all data before splitting; use a Pipeline.
Reusing the test set for repeated decisions, which turns it into a validation set and makes the estimate optimistic.
Duplicates or related records (same patient, same user) across sets.
Random splits for time-ordered data, which let the model see the future.
5
What role does a loss function play in supervised learning?
KnownRevise
A loss function measures how wrong a model’s prediction is for a training example, turning errors into a number the training algorithm can minimize. It defines what “good” means during training: the model’s parameters are adjusted, for example by gradient descent, to reduce the average loss. Choosing the loss is a modelling decision: it should match the type of target and the cost of different errors, and it is often different from the business metric used for evaluation.
Square of the difference; punishes large errors heavily; gives the mean
Linear regression
Regression
Absolute error
Size of the difference; robust to outliers; gives the median
Quantile/LAD regression
Regression
Huber
Squared for small errors, absolute for large
Robust regression
Regression
Quantile (pinball)
Asymmetric penalty for under- vs over-prediction
Prediction intervals
Classification
Log loss (cross-entropy)
Penalizes confident wrong probabilities
Logistic regression, neural networks
Classification
Hinge
Penalizes points inside the margin
SVMs
Classification
0-1 loss
Counts mistakes; not differentiable
Evaluation only
Loss Versus Metric
Loss: smooth and optimizable, used during training (log loss).
Metric: what the business cares about, used for evaluation (F1, revenue, MAPE).
They should point in the same direction; otherwise adjust the loss, weights or decision threshold.
Regularized Objective
Most models minimize loss + penalty, for example average squared error plus alpha times the sum of squared weights (ridge). The penalty discourages overly complex models.
Code Example
import numpy as np
from sklearn.linear_model import HuberRegressor, LinearRegression, QuantileRegressor
rng = np.random.default_rng(42)
X = rng.uniform(0, 10, (200, 1))
y = 3 * X.ravel() + rng.normal(0, 2, 200)
y[:10] += 60 # a few large outliers
for name, model in [("squared loss (OLS)", LinearRegression()),
("Huber loss", HuberRegressor()),
("absolute loss (median)", QuantileRegressor(quantile=0.5, alpha=0.0)),
("pinball loss, 90th pct", QuantileRegressor(quantile=0.9, alpha=0.0))]:
model.fit(X, y)
print(f"{name:24s} slope {model.coef_[0]:.2f}, intercept {model.intercept_:.2f}")
# Log loss for classification: confident wrong predictions cost the most
def log_loss_single(y_true, p):
return -(y_true * np.log(p) + (1 - y_true) * np.log(1 - p))
for p in (0.9, 0.6, 0.1):
print(f"true class 1, predicted p={p}: log loss {log_loss_single(1, p):.3f}")
The true slope is 3; the outliers pull the squared-loss fit upward more than the Huber and absolute-loss fits.
Common Pitfalls
Using squared error with heavy outliers when the business cares about typical errors.
Optimizing log loss but judging only by accuracy without tuning the threshold.
6
What are overfitting and underfitting in machine learning models?
KnownRevise
Overfitting happens when a model learns the training data too closely, including its noise and quirks, so it performs well on training data but poorly on new data. Underfitting happens when a model is too simple, or too constrained, to capture the real pattern, so it performs poorly even on the training data. The goal is the middle ground where the model captures the true signal and generalizes, which is found by comparing training and validation performance while adjusting model complexity.
Model too simple, too much regularization, missing features
Good fit
Low
Low, slightly higher than training
Appropriate complexity
Overfitting
Very low
Much higher
Model too complex or too little data, so noise is memorized
Remedies
Underfitting
Overfitting
More flexible model (deeper trees, more layers, non-linear kernels)
Simpler model, fewer features
Less regularization
More regularization (L1/L2, dropout)
Better or more features
More training data, data augmentation
Train longer
Early stopping
Ensembles (bagging), cross-validated tuning
Code Example: Polynomial Degree
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
rng = np.random.default_rng(42)
X = np.sort(rng.uniform(-3, 3, 60))[:, None]
y = np.sin(X).ravel() + rng.normal(0, 0.3, 60)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=42)
for degree in (1, 3, 15):
model = make_pipeline(PolynomialFeatures(degree), StandardScaler(), LinearRegression()).fit(X_train, y_train)
tr = mean_squared_error(y_train, model.predict(X_train))
va = mean_squared_error(y_val, model.predict(X_val))
print(f"degree {degree:2d}: train MSE {tr:.3f}, validation MSE {va:.3f}")
Degree 1 (a straight line) underfits the sine curve, degree 3 fits it well, and degree 15 typically has the lowest training error but a higher validation error because it bends to follow noise.
Tools
Learning curves: plot training and validation scores as data grows; a persistent gap means overfitting, two low curves mean underfitting.
Validation curves: plot scores against one hyperparameter (depth, C, alpha).
Common Pitfalls
Judging a model by training accuracy alone.
Mistaking leakage for a good fit: suspiciously high validation scores deserve a leakage check.
7
What methods can be used to prevent overfitting?
KnownRevise
Overfitting is prevented by limiting how much a model can memorize noise and by giving it more and better evidence. The main methods are more or augmented data, simpler models or fewer features, regularization (L1/L2 penalties, dropout, tree constraints), early stopping, ensembles such as bagging, and honest validation with cross-validation so that tuning itself does not overfit. Which to use depends on the model family and on whether data or capacity is the real constraint.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Methods by Category
Category
Method
Applies to
Data
Collect more data; data augmentation; remove label noise
All models
Features
Feature selection, dimensionality reduction (PCA)
Linear models, k-NN, SVM
Penalties
L2 (ridge), L1 (lasso), elastic net
Linear/logistic regression, neural networks
Model capacity
Max depth, min samples per leaf, fewer layers/units
Trees, neural networks
Training control
Early stopping on a validation set
Gradient boosting, neural networks
Noise
Dropout, label smoothing
Neural networks
Ensembles
Bagging, random forests, averaging
High-variance models
Validation
Cross-validation, nested CV, a single-use test set
Model selection
Code Example: Several Methods on One Problem
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_validate
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=800, n_features=30, n_informative=6, flip_y=0.1, random_state=42)
models = {
"unpruned tree": DecisionTreeClassifier(random_state=42),
"tree, max_depth=4, min_leaf=10": DecisionTreeClassifier(max_depth=4, min_samples_leaf=10, random_state=42),
"random forest (bagging)": RandomForestClassifier(n_estimators=300, random_state=42),
"boosting + early stopping": HistGradientBoostingClassifier(early_stopping=True, validation_fraction=0.15,
random_state=42),
}
for name, model in models.items():
res = cross_validate(model, X, y, cv=5, return_train_score=True)
print(f"{name:32s} train {res['train_score'].mean():.3f} val {res['test_score'].mean():.3f}")
The unpruned tree should reach a training accuracy of 1.0 with a much lower validation score; the constrained and ensemble models narrow that gap.
Practical Order of Attack
Check for leakage and data quality problems first.
Get more or augmented data if possible.
Add regularization or constrain capacity, tuned with cross-validation.
Use ensembles or early stopping.
Keep the test set untouched until the end.
Common Pitfalls
Over-regularizing until the model underfits.
Tuning many hyperparameters on a small validation set, which overfits the validation set.
8
What is the bias-variance tradeoff?
KnownRevise
The bias-variance tradeoff describes two sources of prediction error that move in opposite directions as model complexity changes. Bias is error from overly simple assumptions: the model systematically misses the true pattern (underfitting). Variance is error from sensitivity to the particular training sample: the model changes a lot with different data (overfitting). For squared error, expected error = bias² + variance + irreducible noise, so the best model balances the two rather than minimizing either alone.
Code Example: Estimating Bias and Variance by Resampling
import numpy as np
from sklearn.tree import DecisionTreeRegressor
rng = np.random.default_rng(42)
true_f = np.sin
x_test = np.linspace(0, 6, 100)[:, None]
def sample(n=50):
x = rng.uniform(0, 6, (n, 1))
return x, true_f(x).ravel() + rng.normal(0, 0.3, n)
for depth in (1, 3, 10):
preds = []
for _ in range(200): # 200 different training sets
x, y = sample()
preds.append(DecisionTreeRegressor(max_depth=depth, random_state=42).fit(x, y).predict(x_test))
preds = np.array(preds)
bias_sq = np.mean((preds.mean(axis=0) - true_f(x_test).ravel()) ** 2)
variance = np.mean(preds.var(axis=0))
print(f"depth {depth:2d}: bias^2 {bias_sq:.4f}, variance {variance:.4f}, sum {bias_sq + variance:.4f}")
Shallow trees show high bias and low variance; deep trees show the reverse; a moderate depth gives the lowest total. The noise variance (0.09 here) is added on top and cannot be reduced by any model.
Modern Nuance
Very large neural networks and ensembles can show “double descent”, where test error falls again beyond the point of interpolating the training data. The classic tradeoff still guides most practical tuning.
Interview Tips
Give the decomposition in words and an example of each side.
Link remedies to the side they address: bagging reduces variance, boosting reduces bias.
9
What are validation sets and cross-validation?
KnownRevise
A validation set is labelled data held out from training and used to compare models and choose hyperparameters, so those choices are not made on the test set. Cross-validation generalizes this: the training data is split into k folds, the model is trained k times, each time validating on a different fold, and the k scores are averaged. Cross-validation uses data more efficiently and gives a more stable estimate, which is important for small datasets; a single validation split is cheaper for large data or expensive models.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Comparison
Method
How
Pros
Cons
Single validation split
Train on e.g. 80%, validate on 20%
Fast; simple
Noisy estimate; wastes data
k-fold CV
k train/validate rounds
Stable estimate; uses all data
k times the cost
Stratified k-fold
Keeps class ratios per fold
Needed for imbalanced classes
—
Group k-fold
Groups (patients, users) never split across folds
Prevents leakage
Needs group IDs
Time-series split
Train on past, validate on future, rolling forward
Respects time order
Fewer training points early
Leave-one-out
k = n
Uses almost all data
Very costly, high variance
Nested CV
Inner CV tunes, outer CV evaluates
Unbiased estimate after tuning
Expensive
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (GridSearchCV, StratifiedKFold, cross_val_score, cross_validate,
train_test_split)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC())
# 1. Single validation split
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.25, stratify=y_train, random_state=42)
print("single split accuracy:", round(pipe.fit(X_tr, y_tr).score(X_val, y_val), 3))
# 2. Stratified 5-fold CV
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
res = cross_validate(pipe, X_train, y_train, cv=cv, scoring=["accuracy", "roc_auc"])
print(f"5-fold accuracy {res['test_accuracy'].mean():.3f} ± {res['test_accuracy'].std():.3f}")
# 3. Nested CV: inner loop tunes C, outer loop estimates performance of the whole procedure
inner = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10]}, cv=StratifiedKFold(3, shuffle=True, random_state=42))
print("nested CV accuracy:", round(cross_val_score(inner, X_train, y_train, cv=cv).mean(), 3))
# 4. Final check on the untouched test set
print("test accuracy:", round(inner.fit(X_train, y_train).score(X_test, y_test), 3))
Common Pitfalls
Preprocessing (scaling, imputation, feature selection) fitted outside the folds; keep it in a Pipeline.
Shuffled k-fold on time series or grouped data.
Reporting the best CV score after tuning as the expected performance; use nested CV or the test set.
10
What is regularization, and how does it work?
KnownRevise
Regularization is any technique that discourages a model from becoming too complex, so it generalizes better to new data. Most commonly it adds a penalty term to the training objective: the model minimizes the data loss plus a cost on the size of its weights, which pulls weights toward zero and makes the learned function smoother. The strength of the penalty is a hyperparameter (alpha, lambda or the inverse C) chosen with cross-validation.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Penalty Types
Type
Penalty (in words)
Effect
Model
L2 (ridge)
alpha × sum of squared weights
Shrinks all weights smoothly; handles correlated features
Ridge, LogisticRegression default
L1 (lasso)
alpha × sum of absolute weights
Sets some weights exactly to zero: feature selection
Lasso, LogisticRegression(penalty="l1")
Elastic net
Mix of L1 and L2
Sparse but stable with correlated features
ElasticNet
Other Forms of Regularization
Tree constraints: max_depth, min_samples_leaf, pruning (ccp_alpha).
Neural networks: weight decay, dropout, early stopping, data augmentation.
Ensembles: averaging many models reduces variance.
Why It Works
Penalizing large weights limits how sharply the model can react to small changes in the inputs, which reduces variance at the cost of a little bias. From a Bayesian view, L2 corresponds to a Gaussian prior on the weights and L1 to a Laplace prior.
Code Example
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_regression(n_samples=80, n_features=60, n_informative=8, noise=15, random_state=42)
models = {"no penalty": LinearRegression()}
for alpha in (0.1, 1, 10):
models[f"ridge alpha={alpha}"] = Ridge(alpha=alpha)
models[f"lasso alpha={alpha}"] = Lasso(alpha=alpha, max_iter=10000)
for name, est in models.items():
pipe = make_pipeline(StandardScaler(), est)
rmse = -cross_val_score(pipe, X, y, cv=5, scoring="neg_root_mean_squared_error").mean()
pipe.fit(X, y)
coefs = pipe[-1].coef_
print(f"{name:18s} CV RMSE {rmse:7.2f} | weight norm {np.linalg.norm(coefs):7.1f} | "
f"zero weights {int(np.sum(np.abs(coefs) < 1e-8))}")
With 60 features and only 80 samples, the unpenalized model overfits; ridge shrinks the weights and lasso zeroes out most uninformative features.
Common Pitfalls
Not scaling features: the penalty then punishes features unevenly.
Confusing alpha (larger = stronger) with C (larger = weaker) across APIs.
Regularizing the intercept, which most libraries correctly avoid.
11
Describe linear regression.
KnownRevise
Linear regression models a numeric target as a weighted sum of input features plus an intercept, y ≈ b0 + b1 x1 + ... + bp xp, and chooses the weights that minimize the sum of squared differences between predictions and actual values (ordinary least squares). It is fast, interpretable, and a strong baseline; each coefficient states how much the prediction changes per unit change in a feature with the others held constant. Its accuracy depends on the relationship being roughly linear, or made linear through feature engineering.
Closed form: the normal equation solves for the weights directly (computed stably with QR or SVD decompositions).
Gradient descent: iteratively moves weights against the gradient of the squared error; used for very large data (SGDRegressor).
Regularized variants: ridge (L2), lasso (L1), elastic net add penalties to reduce overfitting.
Assumptions
Assumption
Check
Linearity
Residual vs fitted plot shows no pattern
Independent errors
No autocorrelation (Durbin-Watson for time data)
Constant variance
Residual spread does not grow with fitted values
Normal errors (for inference)
Q-Q plot of residuals
Low multicollinearity
Variance inflation factors
Code Example
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import cross_val_score, train_test_split
X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for name, model in [("mean baseline", DummyRegressor()), ("linear regression", LinearRegression())]:
mae = -cross_val_score(model, X_train, y_train, cv=5, scoring="neg_mean_absolute_error").mean()
print(f"{name:18s} CV MAE {mae:.1f}")
lr = LinearRegression().fit(X_train, y_train)
pred = lr.predict(X_test)
print(f"test MAE {mean_absolute_error(y_test, pred):.1f}, R² {r2_score(y_test, pred):.3f}")
for name, coef in sorted(zip(X.columns, lr.coef_), key=lambda t: -abs(t[1]))[:4]:
print(f"{name:4s} {coef:8.1f}")
# The normal-equation solution matches scikit-learn's
Xb = np.c_[np.ones(len(X_train)), X_train]
w = np.linalg.lstsq(Xb, y_train, rcond=None)[0]
print("matches:", np.allclose(w[1:], lr.coef_))
The diabetes features are already centred and scaled, so their coefficients can be compared directly; with raw features, standardize first.
Strengths and Limits
Strengths
Limits
Fast, interpretable, few hyperparameters
Misses non-linear patterns unless features are engineered
Works well with many features when regularized
Sensitive to outliers (squared loss)
Provides confidence intervals (statsmodels)
Correlated features make coefficients unstable
Common Pitfalls
Extrapolating far outside the training range.
Interpreting coefficients causally without accounting for confounders.
12
What is the difference between simple and multiple linear regression?
KnownRevise
Simple linear regression models a numeric target with one input feature, fitting a straight line y = b0 + b1 x. Multiple linear regression uses two or more features, fitting a plane or hyperplane y = b0 + b1 x1 + b2 x2 + ... + bp xp. Both estimate coefficients by minimizing the sum of squared errors, but multiple regression lets each coefficient show a feature’s effect while holding the other features constant, and it brings new issues such as multicollinearity and feature selection.
Coefficient can absorb effects of omitted variables
Can adjust for measured confounders
Code Example
import numpy as np
import pandas as pd
import statsmodels.api as sm
rng = np.random.default_rng(42)
n = 300
size = rng.normal(120, 30, n) # m²
age = rng.uniform(0, 50, n) # years
# Larger houses tend to be newer in this market, so size and age are correlated
age = age - 0.2 * (size - 120)
price = 50 + 2.0 * size - 1.5 * age + rng.normal(0, 20, n)
df = pd.DataFrame({"size": size, "age": age, "price": price})
simple = sm.OLS(df["price"], sm.add_constant(df[["size"]])).fit()
multiple = sm.OLS(df["price"], sm.add_constant(df[["size", "age"]])).fit()
print("simple: size coef", round(simple.params["size"], 3), "| R²", round(simple.rsquared, 3))
print("multiple: size coef", round(multiple.params["size"], 3), "age coef", round(multiple.params["age"], 3),
"| adj R²", round(multiple.rsquared_adj, 3))
The true size effect is 2.0. In the simple model the size coefficient also absorbs part of the age effect (bigger houses are newer, so they look even more valuable); the multiple model separates the two effects.
Assumptions (Both)
Linear relationship between features and target.
Independent errors with constant variance (homoscedasticity).
Approximately normal errors for valid confidence intervals and p-values.
For multiple regression: no perfect multicollinearity among features.
Common Pitfalls
Interpreting a simple-regression slope as causal when other variables matter.
Comparing R² across models with different numbers of features; use adjusted R² or cross-validation.
Logistic regression is a classification model that predicts the probability of a class by passing a weighted sum of features through the sigmoid function, p = 1 / (1 + exp(-(b0 + b·x))), which maps any number into (0, 1). It is trained by minimizing log loss (cross-entropy). It is used when you need a fast, interpretable, well-calibrated baseline for binary or multi-class classification, such as credit default, churn, disease risk or click prediction, and when you want to explain how each feature shifts the odds.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Key Ideas
Log-odds are linear: log(p / (1 - p)) = b0 + b1 x1 + .... A coefficient b_j is the change in log-odds per unit of x_j; exp(b_j) is the odds ratio.
Decision: predict class 1 when p >= threshold (0.5 by default; tune it for the cost of errors).
Multi-class: multinomial (softmax) logistic regression, the default in scikit-learn for more than two classes.
Regularization: L2 by default (C is the inverse strength); L1 for sparse models.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2,
stratify=data.target, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
print("ROC AUC:", round(roc_auc_score(y_test, proba), 3))
print(classification_report(y_test, model.predict(X_test), target_names=data.target_names))
coefs = model[-1].coef_[0]
for i in np.argsort(-np.abs(coefs))[:5]:
print(f"{data.feature_names[i]:22s} odds ratio per 1 SD: {np.exp(coefs[i]):.2f}")
Because the features are standardized, each odds ratio is the multiplicative change in the odds of the benign class for a one-standard-deviation increase in that feature.
Common Pitfalls
Unscaled features with regularization, which penalizes features unequally and slows convergence.
Perfect separation, which drives coefficients toward infinity without regularization.
Reading coefficients of correlated features as independent effects.
Using 0.5 as the threshold when classes are imbalanced or error costs differ.
14
How does ridge regression prevent overfitting?
KnownRevise
Ridge regression prevents overfitting by adding an L2 penalty, alpha times the sum of squared coefficients, to the least-squares objective. The penalty shrinks all coefficients toward zero, which reduces the model’s sensitivity to noise and to small changes in the training data (lower variance) at the cost of a small bias. It is especially effective when there are many features, features are correlated (multicollinearity), or the number of features approaches the number of samples, where ordinary least squares becomes unstable.
Objective: minimize the sum of squared errors + alpha × sum of squared weights.
Closed form: w = (XᵀX + alpha I)⁻¹ Xᵀy. Adding alpha to the diagonal makes the matrix well-conditioned even when features are correlated or p > n.
Effect: directions in feature space with little information (small singular values) are shrunk the most, so noise-driven coefficients shrink while strong signals survive.
alpha = 0 gives ordinary least squares; very large alpha drives all coefficients toward zero (underfitting).
Ridge Versus Lasso
Aspect
Ridge (L2)
Lasso (L1)
Coefficients
Shrunk, rarely exactly zero
Some exactly zero
Correlated features
Spreads weight among them
Picks one, somewhat arbitrarily
Solution
Closed form
Iterative
Use
Many small effects, multicollinearity
Feature selection, sparse truth
Code Example
import numpy as np
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n, p = 60, 40
base = rng.normal(size=(n, 5))
X = np.hstack([base + rng.normal(0, 0.05, (n, 5)) for _ in range(8)]) # 40 highly correlated features
y = base @ np.array([3.0, -2.0, 1.0, 0.5, 0.0]) + rng.normal(0, 1.0, n)
ols = make_pipeline(StandardScaler(), LinearRegression())
ridge = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-3, 3, 25)))
for name, model in [("OLS", ols), ("Ridge", ridge)]:
rmse = -cross_val_score(model, X, y, cv=5, scoring="neg_root_mean_squared_error").mean()
model.fit(X, y)
print(f"{name:5s} CV RMSE {rmse:.3f} | largest |coef| {np.abs(model[-1].coef_).max():.2f}")
print("chosen alpha:", round(ridge[-1].alpha_, 4))
With 40 near-duplicate features and 60 samples, OLS coefficients become large and unstable; ridge keeps them small and predicts better.
Common Pitfalls
Not standardizing features, so the penalty affects features unequally.
Choosing alpha on the test set; use RidgeCV or cross-validation.
Expecting ridge to remove features; use lasso or elastic net for sparsity.
15
What is Lasso regression, and what is its unique property?
KnownRevise
Lasso regression (Least Absolute Shrinkage and Selection Operator) is linear regression with an L1 penalty: it minimizes squared error plus alpha times the sum of the absolute values of the coefficients. Its unique property is that it sets some coefficients exactly to zero, performing automatic feature selection and producing sparse, more interpretable models. Ridge regression, by contrast, shrinks coefficients but almost never makes them exactly zero.
Geometrically, the L1 constraint region is a diamond (in 2D) with corners on the axes; the least-squares contours usually touch it at a corner, where one coefficient is zero. Algebraically, the absolute-value penalty has a constant slope, so a coefficient is set to zero whenever the feature’s correlation with the residual is weaker than alpha (the “soft-thresholding” rule in coordinate descent).
Properties
Property
Detail
Sparsity
Drops irrelevant features
Correlated features
Tends to keep one and drop the rest; selection can be unstable
p > n
Selects at most n features
Tuning
alpha via cross-validation (LassoCV)
Bias
Shrinks kept coefficients too; refit OLS on selected features if unbiased estimates matter
Alternative
Elastic net mixes L1 and L2 for correlated features
Lasso should keep the 6 informative features and a few or none of the others, while ridge leaves all 50 non-zero.
Common Pitfalls
Forgetting to standardize features, so the penalty favours large-scale features.
Treating the selected set as definitive when features are correlated; check stability with bootstrap resampling.
Using lasso when many small effects all matter; ridge often predicts better there.
16
What is the principle behind the Support Vector Machine (SVM)?
KnownRevise
The principle of the SVM is maximum-margin separation: among all boundaries that separate two classes, choose the one that leaves the widest gap (margin) to the nearest training points, because a wide margin tends to generalize better. The nearest points, the support vectors, alone define the boundary. A soft margin with penalty C tolerates some violations for noisy data, and the kernel trick lets the SVM draw non-linear boundaries by computing similarities in an implicit higher-dimensional space.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Core Elements
Element
Meaning
Hyperplane
Boundary w·x + b = 0
| Margin | Width 2 / ||w|| between the classes’ closest points | | Objective | Minimize (1/2)||w||^2 + C × total slack (soft margin) | | Support vectors | Points on or inside the margin; the only ones that matter | | C | Large C: fewer violations, narrower margin; small C: wider margin, more tolerance | | Kernel | Linear, polynomial, RBF (gamma controls reach), sigmoid | | Hinge loss | Equivalent view: L2-regularized hinge loss max(0, 1 - y f(x)) |
Code Example
from sklearn.datasets import make_moons
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=500, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
{"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.1, 0.5, 1, 5]}, cv=5)
grid.fit(X_train, y_train)
svc = grid.best_estimator_[-1]
print("best:", grid.best_params_)
print("support vectors:", len(svc.support_), "of", len(X_train))
print("test accuracy:", round(grid.score(X_test, y_test), 3))
When to Use SVMs
Small to medium datasets (up to tens of thousands of rows), often high-dimensional.
Text classification with a linear kernel.
When a strong, well-regularized classifier is needed without much feature engineering.
For large data, use LinearSVC or SGDClassifier(loss="hinge"), because kernel SVM training scales between O(n²) and O(n³).
Common Pitfalls
Forgetting feature scaling.
Tuning C and gamma separately instead of together.
Treating decision_function outputs as probabilities.
17
How do you handle categorical variables in supervised learning models?
KnownRevise
Categorical variables take values from a fixed set of categories (city, product type, plan), and most models need them converted to numbers in a way that does not invent false relationships. The standard choices are one-hot encoding for nominal categories with few levels, ordinal encoding for ordered categories (and for tree models), target encoding for high-cardinality features, and native categorical support in gradient-boosting libraries. Encoders must be fitted on training data only and must handle categories that appear later.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, TargetEncoder
rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
"plan": rng.choice(["basic", "plus", "pro"], n),
"size": rng.choice(["S", "M", "L"], n),
"zip": rng.choice([f"z{i}" for i in range(200)], n),
"tenure": rng.integers(1, 72, n),
})
zip_risk = {f"z{i}": rng.normal(0, 1) for i in range(200)}
logit = (-1 + 1.2 * (df["plan"] == "basic") + 0.5 * df["size"].map({"S": 0, "M": 1, "L": 2})
+ df["zip"].map(zip_risk) - 0.03 * df["tenure"])
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
linear = Pipeline([
("prep", ColumnTransformer([
("onehot", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
("ordinal", OrdinalEncoder(categories=[["S", "M", "L"]]), ["size"]),
("target", TargetEncoder(random_state=42), ["zip"]), # cross-fitted internally
("num", StandardScaler(), ["tenure"]),
])),
("model", LogisticRegression(max_iter=1000)),
])
boosted = Pipeline([
("prep", ColumnTransformer([("cat", OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
["plan", "size", "zip"])], remainder="passthrough")),
("model", HistGradientBoostingClassifier(categorical_features=[0, 1, 2], random_state=42)),
])
for name, model in [("logistic + encoders", linear), ("boosting, native categories", boosted)]:
print(f"{name:28s} CV ROC AUC {cross_val_score(model, df, y, cv=5, scoring='roc_auc').mean():.3f}")
Native categorical support in HistGradientBoostingClassifier allows at most 255 categories per feature (the max_bins limit); for more levels, group rare categories or target-encode the column.
Common Pitfalls
Label-encoding nominal categories for linear models, which invents an order.
Fitting encoders (especially target encoders) on all data before splitting.
Not handling unseen categories at prediction time.
18
What are ensemble methods, and how do they improve model performance?
KnownRevise
Ensemble methods combine the predictions of several models to get a result that is more accurate and more stable than any single model. They work because different models make different errors: averaging reduces random errors (variance), and sequentially correcting mistakes reduces systematic errors (bias). The main families are bagging (random forests), boosting (AdaBoost, gradient boosting such as XGBoost and LightGBM), and stacking or voting, which combine different model types.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Families
Family
How
Reduces
Examples
Bagging
Train models on bootstrap samples in parallel; average or vote
Variance
BaggingClassifier, RandomForest, ExtraTrees
Boosting
Train models sequentially; each focuses on previous errors
Meta-model learns how to combine base models’ out-of-fold predictions
Both
StackingClassifier
Why Averaging Helps
If M models have errors with variance σ² and pairwise correlation ρ, the average has variance roughly ρσ² + (1 - ρ)σ²/M. Uncorrelated errors shrink as M grows; correlated errors don’t. That is why random forests decorrelate trees by sampling features at each split.
Code Example
from sklearn.datasets import make_classification
from sklearn.ensemble import (HistGradientBoostingClassifier, RandomForestClassifier, StackingClassifier,
VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=2000, n_features=25, n_informative=10, flip_y=0.05, random_state=42)
lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
rf = RandomForestClassifier(n_estimators=300, random_state=42)
gb = HistGradientBoostingClassifier(random_state=42)
models = {
"single decision tree": DecisionTreeClassifier(random_state=42),
"logistic regression": lr,
"random forest (bagging)": rf,
"gradient boosting": gb,
"soft voting": VotingClassifier([("lr", lr), ("rf", rf), ("gb", gb)], voting="soft"),
"stacking": StackingClassifier([("lr", lr), ("rf", rf), ("gb", gb)], final_estimator=LogisticRegression()),
}
for name, model in models.items():
print(f"{name:24s} CV ROC AUC {cross_val_score(model, X, y, cv=5, scoring='roc_auc', n_jobs=-1).mean():.3f}")
Trade-Offs
Better accuracy and robustness, but more compute, memory and latency.
Harder to interpret than a single tree or linear model (use feature importance or SHAP).
Gains are largest when base models are good and diverse.
Common Pitfalls
Ensembling near-identical models, which adds cost without benefit.
Stacking with in-sample predictions, which leaks labels into the meta-model.
19
What is the difference between bagging and boosting?
KnownRevise
Bagging (bootstrap aggregating) trains many models independently and in parallel on random bootstrap samples of the data and averages their predictions, which mainly reduces variance; random forests are the best-known example. Boosting trains models sequentially, each one focusing on the errors of the ensemble so far, and combines them in a weighted sum, which mainly reduces bias; AdaBoost and gradient boosting (XGBoost, LightGBM, CatBoost) are examples. Bagging suits high-variance learners such as deep trees, while boosting turns weak learners such as shallow trees into a strong model.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Comparison
Aspect
Bagging
Boosting
Training
Parallel, independent
Sequential, dependent
Data per model
Bootstrap sample
Full data, reweighted (AdaBoost) or fit to residuals/gradients (gradient boosting)
Base learners
Strong, high-variance (deep trees)
Weak, high-bias (shallow trees)
Combination
Simple average / majority vote
Weighted sum
Main effect
Lower variance
Lower bias
Overfitting risk
Low; more trees don’t hurt
Can overfit with too many rounds; use learning rate and early stopping
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import cross_validate
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)
models = {
"deep tree (high variance)": DecisionTreeClassifier(random_state=42),
"bagging deep trees": BaggingClassifier(DecisionTreeClassifier(), n_estimators=200, n_jobs=-1, random_state=42),
"stump (high bias)": DecisionTreeClassifier(max_depth=1),
"AdaBoost on stumps": AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=300,
random_state=42),
"gradient boosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
res = cross_validate(model, X, y, cv=5, return_train_score=True, n_jobs=-1)
print(f"{name:26s} train {res['train_score'].mean():.3f} val {res['test_score'].mean():.3f}")
Bagging should keep the deep tree’s high training score while raising its validation score (less variance); boosting should lift the stump’s low scores on both sets (less bias).
Interview Tips
Summarize as “bagging: parallel, variance; boosting: sequential, bias”.
Mention that random forests add random feature selection at each split to decorrelate trees beyond plain bagging.
20
What is feature scaling, and why is it important?
KnownRevise
Feature scaling transforms numeric features to comparable ranges, for example standardizing each to mean 0 and standard deviation 1, or rescaling to [0, 1]. It matters for any model that uses distances, dot products, gradients or penalties on weights: k-NN, SVMs, k-means, PCA, linear and logistic regression with regularization, and neural networks. Without scaling, features with large numeric ranges dominate distances and penalties, and gradient descent converges slowly. Tree-based models split on one feature at a time and do not need scaling.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Scalers
Scaler
Transformation
Use when
StandardScaler
(x − mean) / std
Default for most models
MinMaxScaler
(x − min) / (max − min)
Bounded inputs, neural nets, images
RobustScaler
(x − median) / IQR
Outliers present
MaxAbsScaler
x / max(
x
)
Sparse data (keeps zeros)
PowerTransformer / log
Makes skewed data more Gaussian
Heavily skewed features
Normalizer
Each row to unit length
Text vectors, cosine similarity
Which Models Need It
Needs scaling
Doesn’t
k-NN, k-means, SVM, PCA
Decision trees
Regularized linear/logistic regression
Random forests
Neural networks, gradient-descent models
Gradient-boosted trees
Code Example
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_wine(return_X_y=True) # feature ranges differ by about three orders of magnitude
for name, model in [("k-NN", KNeighborsClassifier()),
("logistic regression", LogisticRegression(max_iter=5000)),
("random forest", RandomForestClassifier(random_state=42))]:
raw = cross_val_score(model, X, y, cv=5).mean()
scaled = cross_val_score(make_pipeline(StandardScaler(), model), X, y, cv=5).mean()
print(f"{name:20s} unscaled {raw:.3f} | scaled {scaled:.3f}")
k-NN should improve the most, the logistic regression should converge more easily and score higher, and the random forest should be essentially unchanged.
Common Pitfalls
Fitting the scaler on the full dataset before the split; put it in a Pipeline.
Scaling the target unintentionally or forgetting to invert a target transform.
Applying StandardScaler to sparse matrices, which destroys sparsity (use with_mean=False or MaxAbsScaler).
21
How is a decision tree constructed?
KnownRevise
A decision tree is built top-down by recursive partitioning: starting with all training data at the root, the algorithm tries candidate splits (a feature and a threshold, or a category grouping), picks the split that makes the child nodes as pure as possible according to a criterion such as Gini impurity, entropy or squared error, and repeats on each child. Splitting stops when a node is pure, too small, or a depth limit is reached; leaves predict the majority class (classification) or the mean target (regression). This greedy procedure, used by CART in scikit-learn, finds good but not globally optimal trees.
For each feature, sort the values and consider thresholds between consecutive distinct values.
For each threshold, compute the weighted impurity of the two children: (n_left/n) * impurity_left + (n_right/n) * impurity_right.
Choose the split with the largest impurity decrease.
Recurse on both children.
Stop when a stopping rule triggers: max_depth, min_samples_split, min_samples_leaf, min_impurity_decrease, or a pure node.
Optionally prune afterwards (cost-complexity pruning with ccp_alpha).
Criteria
Task
Criterion
Idea
Classification
Gini
Probability of misclassifying a random sample labelled by the node’s class distribution
Classification
Entropy / log loss
Information content of the class distribution
Regression
Squared error
Variance of targets in the node
Regression
Absolute error, Poisson
Robust or count targets
Code Example: One Split by Hand vs scikit-learn
import numpy as np
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text
X, y = load_iris(return_X_y=True)
def gini(labels):
_, counts = np.unique(labels, return_counts=True)
p = counts / counts.sum()
return 1 - np.sum(p ** 2)
best = (None, None, np.inf)
for f in range(X.shape[1]):
values = np.unique(X[:, f])
for t in (values[:-1] + values[1:]) / 2: # midpoints between distinct values
left, right = y[X[:, f] <= t], y[X[:, f] > t]
score = (len(left) * gini(left) + len(right) * gini(right)) / len(y)
if score < best[2]:
best = (f, t, score)
print(f"best root split: feature {best[0]} <= {best[1]:.2f}, weighted Gini {best[2]:.3f}")
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, y)
print(export_text(tree, feature_names=["sepal_len", "sepal_wid", "petal_len", "petal_wid"]))
The hand-computed root split separates setosa perfectly; scikit-learn finds an equally good split (petal length and petal width tie here, so the reported feature may differ).
Complexity
Training costs roughly O(n_features × n log n) per level for sorting-based splits; prediction costs O(depth).
Common Pitfalls
Growing trees without limits, which memorizes the training data.
Expecting the greedy tree to be optimal; small data changes can produce a very different tree (high variance).
Using a single tree when a random forest or gradient boosting would be far more accurate.
22
What are the pros and cons of using decision trees?
KnownRevise
Decision trees are easy to understand and need little data preparation: they handle numeric and categorical features, ignore feature scaling, capture non-linear relationships and interactions, and can be drawn as a set of if-then rules. Their major weaknesses are high variance (small changes in the data can change the tree a lot), a strong tendency to overfit without depth limits or pruning, step-shaped predictions that extrapolate poorly, and lower accuracy than ensembles of trees such as random forests and gradient boosting.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Pros and Cons
Pros
Cons
Interpretable rules; easy to explain to non-experts
Overfit easily if grown deep
No scaling needed; handle mixed feature types
Unstable: small data changes alter the structure
Capture non-linearity and interactions automatically
Impurity-based importance biased toward high-cardinality features
Handle missing values (scikit-learn 1.3+ supports NaN in trees)
Greedy: not globally optimal
Building block for strong ensembles
Usually less accurate than ensembles
Code Example: Instability and the Ensemble Fix
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, flip_y=0.05, random_state=42)
# Instability: train on two bootstrap samples and compare the root splits
rng = np.random.default_rng(42)
for i in range(3):
idx = rng.choice(len(X), len(X), replace=True)
t = DecisionTreeClassifier(random_state=42).fit(X[idx], y[idx])
print(f"sample {i}: root split on feature {t.tree_.feature[0]} at {t.tree_.threshold[0]:.2f}, "
f"depth {t.get_depth()}, leaves {t.get_n_leaves()}")
for name, model in [("unpruned tree", DecisionTreeClassifier(random_state=42)),
("tree, max_depth=5", DecisionTreeClassifier(max_depth=5, random_state=42)),
("random forest", RandomForestClassifier(n_estimators=300, random_state=42))]:
print(f"{name:18s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
When to Use a Single Tree
When interpretability matters more than the last few points of accuracy (e.g. policy rules, simple triage).
As a quick baseline or to explore interactions in the data.
Otherwise, prefer random forests or gradient boosting.
Gini impurity and information gain are measures decision trees use to choose splits. Gini impurity is the probability of misclassifying a randomly chosen sample if it were labelled at random according to the node’s class distribution: 1 − sum of p_k². Entropy measures the uncertainty of the class distribution, −sum p_k log2 p_k, and information gain is the reduction in entropy from parent to children, weighted by child size. A good split produces purer children, meaning lower weighted impurity or higher information gain.
Parent entropy minus weighted average child entropy
≥ 0
—
Gini decrease
Parent Gini minus weighted average child Gini
≥ 0
—
Worked Example
A parent node has 10 samples: 5 positive, 5 negative. - Parent Gini = 1 − (0.5² + 0.5²) = 0.5; parent entropy = 1 bit. - Split into left (4 positive, 1 negative) and right (1 positive, 4 negative). - Each child: Gini = 1 − (0.8² + 0.2²) = 0.32; entropy = −(0.8 log2 0.8 + 0.2 log2 0.2) ≈ 0.722. - Weighted children: Gini 0.32, entropy 0.722. - Gini decrease = 0.5 − 0.32 = 0.18; information gain = 1 − 0.722 = 0.278 bits.
Code Example
import numpy as np
def gini(counts):
p = np.asarray(counts) / np.sum(counts)
return 1 - np.sum(p ** 2)
def entropy(counts):
p = np.asarray(counts) / np.sum(counts)
p = p[p > 0]
return -np.sum(p * np.log2(p))
def split_gain(parent, children, measure):
n = sum(sum(c) for c in children)
weighted = sum(sum(c) / n * measure(c) for c in children)
return measure(parent) - weighted
parent = [5, 5]
children = [[4, 1], [1, 4]]
print("Gini decrease:", round(split_gain(parent, children, gini), 3)) # 0.18
print("Information gain:", round(split_gain(parent, children, entropy), 3)) # 0.278
perfect = [[5, 0], [0, 5]]
print("perfect split gains:", split_gain(parent, perfect, gini), round(split_gain(parent, perfect, entropy), 3))
Gini Versus Entropy
They usually pick the same or very similar splits.
Gini avoids the logarithm and is slightly faster; it is scikit-learn’s default (criterion="gini").
Entropy is a bit more sensitive to changes in small class probabilities.
Common Pitfalls
Forgetting to weight child impurities by their sample counts.
Information gain favours features with many unique values (IDs); C4.5 uses the gain ratio to correct this.
24
How does random forest improve on basic decision trees?
KnownRevise
A random forest improves on a single decision tree by training many deep trees on different bootstrap samples of the data and, at each split, considering only a random subset of features, then averaging their predictions (or taking a majority vote). Individual deep trees have low bias but high variance; averaging many decorrelated trees keeps the low bias while sharply reducing variance. The result is much higher accuracy, robustness to noise and outliers, and a built-in error estimate from out-of-bag samples, at the cost of interpretability and some speed.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
What Changes
Aspect
Single decision tree
Random forest
Training data
All rows
Bootstrap sample per tree
Features per split
All
Random subset (max_features, e.g. sqrt(p) for classification)
Prediction
One tree’s leaf
Average / vote of hundreds of trees
Variance
High
Low
Overfitting
Strong without pruning
Limited; more trees don’t overfit
Validation
Needs CV
Out-of-bag (OOB) estimate available
Interpretability
Readable rules
Needs feature importance / SHAP
Cost
Fast
Roughly n_trees × cost (parallelizable)
Why Feature Subsampling Matters
If one feature is very strong, every bagged tree would split on it first, making the trees similar and their errors correlated. Restricting each split to a random subset forces diversity, and averaging diverse trees cancels more error.
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
tree = DecisionTreeClassifier(random_state=42)
forest = RandomForestClassifier(n_estimators=500, max_features="sqrt", oob_score=True, n_jobs=-1, random_state=42)
for name, model in [("decision tree", tree), ("random forest", forest)]:
cv = cross_val_score(model, X_train, y_train, cv=5).mean()
model.fit(X_train, y_train)
print(f"{name:14s} CV {cv:.3f} | train {model.score(X_train, y_train):.3f} | test {model.score(X_test, y_test):.3f}")
print("random forest OOB accuracy:", round(forest.oob_score_, 3))
for n in (1, 10, 50, 200, 500):
rf = RandomForestClassifier(n_estimators=n, n_jobs=-1, random_state=42)
print(f"{n:4d} trees: CV accuracy {cross_val_score(rf, X_train, y_train, cv=5).mean():.3f}")
Accuracy rises quickly with the first few dozen trees and then plateaus; adding trees costs time but does not cause overfitting.
Common Pitfalls
Tuning n_estimators as if it controlled overfitting; tune max_features, min_samples_leaf and max_depth instead.
Trusting impurity-based importances with high-cardinality features; use permutation importance.
Expecting random forests to extrapolate beyond the training range in regression.
25
How is feature importance determined in decision trees and random forests?
KnownRevise
Feature importance in tree models measures how much each feature contributes to the model’s predictions. The built-in impurity-based (mean decrease in impurity, MDI) importance sums, over every split that uses a feature, the reduction in impurity weighted by the number of samples reaching that split, then averages across trees and normalizes to sum to 1. A more reliable alternative is permutation importance: shuffle one feature’s values in held-out data and measure how much the model’s score drops. SHAP values add per-prediction attributions.
Computed on training data; biased toward high-cardinality and continuous features
Permutation importance
Score drop after shuffling a feature on validation data
Model-agnostic, measured on held-out data
Slower; correlated features share credit and can look unimportant
Drop-column importance
Retrain without the feature
Direct
Very expensive
SHAP (TreeSHAP)
Game-theoretic attribution per prediction
Local and global explanations
Extra library, more computation
Code Example: MDI vs Permutation Importance
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
data = load_breast_cancer(as_frame=True)
X = data.data.copy()
rng = np.random.default_rng(42)
X["random_id"] = rng.integers(0, 10000, len(X)) # pure noise with many unique values
X_train, X_test, y_train, y_test = train_test_split(X, data.target, test_size=0.3, stratify=data.target,
random_state=42)
rf = RandomForestClassifier(n_estimators=300, random_state=42, n_jobs=-1).fit(X_train, y_train)
mdi = pd.Series(rf.feature_importances_, index=X.columns)
perm = permutation_importance(rf, X_test, y_test, n_repeats=20, random_state=42, n_jobs=-1)
perm_mean = pd.Series(perm.importances_mean, index=X.columns)
print("Top 5 by MDI:\n", mdi.sort_values(ascending=False).head(5).round(3))
print("Top 5 by permutation:\n", perm_mean.sort_values(ascending=False).head(5).round(3))
print(f"random_id: MDI rank {mdi.rank(ascending=False)['random_id']:.0f} of {len(mdi)}, "
f"permutation importance {perm_mean['random_id']:.4f}")
The noise column typically receives a non-trivial MDI score because it offers many split points, while its permutation importance is near zero, which illustrates the MDI bias.
Interpreting Results
Importance shows association with predictions, not causation.
With correlated features, importance is split or masked; group them or use conditional methods.
Report variability (standard deviation across repeats or folds).
Common Pitfalls
Ranking features by MDI alone in the presence of ID-like or high-cardinality columns.
Computing permutation importance on training data, which reflects memorization.
26
What are the basic components of a neural network?
KnownRevise
A neural network is built from layers of simple units (neurons), each computing a weighted sum of its inputs plus a bias and passing it through a non-linear activation function. Layers are stacked from an input layer, through one or more hidden layers, to an output layer sized for the task. Training needs a loss function to measure errors, backpropagation to compute gradients, and an optimizer to update the weights, plus regularization and hyperparameters such as the learning rate and batch size.
1 unit (regression), sigmoid (binary), softmax over K classes
Loss function
Measures prediction error
MSE, cross-entropy
Backpropagation
Computes gradients of the loss with respect to all weights
Automatic differentiation
Optimizer
Updates weights from gradients
SGD with momentum, Adam, AdamW
Regularization
Limits overfitting
Weight decay, dropout, early stopping
Hyperparameters
Chosen by the practitioner
Learning rate, batch size, epochs, layer sizes
Code Example
import torch
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn
torch.manual_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = (torch.tensor(scaler.transform(a), dtype=torch.float32) for a in (X_train, X_test))
y_train, y_test = (torch.tensor(a, dtype=torch.float32) for a in (y_train, y_test))
model = nn.Sequential( # input (30) -> hidden (16) -> hidden (8) -> output (1 logit)
nn.Linear(30, 16), nn.ReLU(),
nn.Linear(16, 8), nn.ReLU(),
nn.Linear(8, 1),
)
loss_fn = nn.BCEWithLogitsLoss() # sigmoid + binary cross-entropy, numerically stable
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
for epoch in range(200):
optimizer.zero_grad()
loss = loss_fn(model(X_train).squeeze(1), y_train) # forward pass + loss
loss.backward() # backpropagation
optimizer.step() # weight update
with torch.no_grad():
acc = ((model(X_test).squeeze(1) > 0).float() == y_test).float().mean()
print(f"final training loss {loss.item():.4f}, test accuracy {acc.item():.3f}")
print("parameters:", sum(p.numel() for p in model.parameters()))
The parameter count is 30×16+16 + 16×8+8 + 8×1+1 = 641.
Common Pitfalls
No activation between linear layers, which collapses the network into a single linear model.
An output activation that does not match the loss (e.g. sigmoid before BCEWithLogitsLoss).
Unscaled inputs, which slow or destabilize training.
27
What role do activation functions play in neural networks?
KnownRevise
Activation functions apply a non-linear transformation to each neuron’s weighted sum, which is what allows a neural network to model non-linear relationships. Without them, any stack of linear layers collapses into a single linear function, no matter how deep. The choice of activation also affects how well gradients flow during training: ReLU-family functions avoid the vanishing gradients that saturating functions like sigmoid and tanh cause in deep networks, and output activations (sigmoid, softmax) turn scores into probabilities.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
Common Activations
Function
Formula in words
Range
Use
Issues
ReLU
max(0, x)
[0, ∞)
Default for hidden layers
“Dead” neurons stuck at 0
Leaky ReLU
x if x > 0, else a small slope × x
(−∞, ∞)
Avoids dead neurons
Extra hyperparameter
GELU
x × probability a standard normal is below x
≈(−0.17, ∞)
Transformers
Slightly costlier
SiLU / Swish
x × sigmoid(x)
≈(−0.28, ∞)
CNNs, modern MLPs
Slightly costlier
Sigmoid
1 / (1 + e^(−x))
(0, 1)
Binary output, gates in LSTMs
Saturates: vanishing gradients
Tanh
(e^x − e^(−x)) / (e^x + e^(−x))
(−1, 1)
RNN hidden states
Saturates
Softmax
e^(x_k) / sum of e^(x_j)
(0, 1), sums to 1
Multi-class output
Use with cross-entropy
Linear (none)
x
(−∞, ∞)
Regression output
—
Code Example: Why Non-Linearity Matters and Gradient Flow
import torch
from torch import nn
torch.manual_seed(42)
# 1. Without activations, two linear layers equal one linear layer
lin = nn.Sequential(nn.Linear(4, 8), nn.Linear(8, 2))
W = lin[1].weight @ lin[0].weight
b = lin[1].weight @ lin[0].bias + lin[1].bias
x = torch.randn(5, 4)
print("collapses to one linear map:", torch.allclose(lin(x), x @ W.T + b, atol=1e-6))
# 2. Gradient size reaching the first layer of a deep network
def first_layer_grad(activation, depth=20):
layers = []
for _ in range(depth):
layers += [nn.Linear(64, 64), activation()]
net = nn.Sequential(*layers, nn.Linear(64, 1))
net(torch.randn(32, 64)).sum().backward()
return net[0].weight.grad.abs().mean().item()
for act in (nn.Sigmoid, nn.Tanh, nn.ReLU, nn.GELU):
print(f"{act.__name__:8s} mean |grad| at layer 1: {first_layer_grad(act):.2e}")
With sigmoid, the gradient reaching the first of 20 layers is typically orders of magnitude smaller than with ReLU or GELU, because each sigmoid layer multiplies the gradient by at most 0.25.
Choosing
Hidden layers: ReLU (default), GELU or SiLU for transformers and modern architectures.
Output: none for regression, sigmoid for binary or multi-label, softmax for multi-class (usually folded into the loss).
Common Pitfalls
Applying softmax before nn.CrossEntropyLoss, which already includes it.
Sigmoid or tanh in deep hidden stacks without normalization or residual connections.
28
How do you prevent overfitting in neural networks?
KnownRevise
Neural networks overfit easily because they have many parameters and can memorize training data. The main defences are more and augmented data, early stopping on a validation set, weight decay (L2 regularization), dropout, normalization layers, smaller or pretrained architectures (transfer learning), and label smoothing. They are usually combined and tuned against validation performance, watching the gap between training and validation loss.
Code Example: Dropout, Weight Decay and Early Stopping
import copy
import torch
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn
torch.manual_seed(42)
X, y = make_classification(n_samples=600, n_features=40, n_informative=8, flip_y=0.1, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_tr)
X_tr, X_val = (torch.tensor(scaler.transform(a), dtype=torch.float32) for a in (X_tr, X_val))
y_tr, y_val = (torch.tensor(a, dtype=torch.float32) for a in (y_tr, y_val))
def train(regularized, epochs=300, patience=20):
p, wd = (0.4, 1e-3) if regularized else (0.0, 0.0)
model = nn.Sequential(nn.Linear(40, 256), nn.ReLU(), nn.Dropout(p),
nn.Linear(256, 256), nn.ReLU(), nn.Dropout(p), nn.Linear(256, 1))
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=wd)
loss_fn = nn.BCEWithLogitsLoss()
best, best_state, wait = float("inf"), None, 0
for epoch in range(epochs):
model.train()
opt.zero_grad()
loss_fn(model(X_tr).squeeze(1), y_tr).backward()
opt.step()
model.eval()
with torch.no_grad():
val_loss = loss_fn(model(X_val).squeeze(1), y_val).item()
if val_loss < best:
best, best_state, wait = val_loss, copy.deepcopy(model.state_dict()), 0
elif regularized:
wait += 1
if wait >= patience:
break
if regularized:
model.load_state_dict(best_state)
model.eval()
with torch.no_grad():
acc = lambda X_, y_: ((model(X_).squeeze(1) > 0).float() == y_).float().mean().item()
return acc(X_tr, y_tr), acc(X_val, y_val), epoch + 1
for regularized in (False, True):
tr, va, ep = train(regularized)
print(f"regularized={regularized}: train acc {tr:.3f}, val acc {va:.3f}, epochs {ep}")
For a final, unbiased number, evaluate on a separate test set; here the validation set is also used for early stopping.
Common Pitfalls
Forgetting model.eval() at evaluation time, which leaves dropout active.
Using Adam’s weight_decay (coupled L2) and expecting AdamW behaviour.
Early stopping on the test set.
29
What is backpropagation, and why is it important for training neural networks?
KnownRevise
Backpropagation is the algorithm that computes the gradient of the loss with respect to every weight in a neural network efficiently, by applying the chain rule backwards from the output layer to the input layer and reusing intermediate results. Those gradients tell the optimizer how to change each weight to reduce the loss. Its significance is efficiency: it computes all gradients in roughly the cost of one extra forward pass, which is what makes training networks with millions or billions of parameters feasible.
Read full answerHide full answer
Applies toNumPy 2.x, PyTorch 2.x (Python 3.12)
Steps
Forward pass: compute each layer’s output and store the intermediate values.
Loss: compare the output with the target.
Backward pass: starting from the derivative of the loss with respect to the output, multiply by each layer’s local derivative to get gradients for its weights and for its input, then pass the input gradient to the previous layer.
Update: the optimizer (SGD, Adam) adjusts weights using the gradients.
For a layer h = activation(W x + b): the weight gradient is the incoming gradient (times the activation’s derivative) multiplied by the layer’s input; the gradient passed backwards is that same quantity multiplied by W transposed.
Repeated multiplication by small derivatives (sigmoid/tanh)
ReLU/GELU, residual connections, normalization
Exploding gradients
Repeated multiplication by large values (deep RNNs)
Gradient clipping, careful initialization
Memory use
Stored activations for the backward pass
Gradient checkpointing, smaller batches
Interview Tips
Distinguish backpropagation (computing gradients) from gradient descent (using them).
Mention that modern frameworks implement it as reverse-mode automatic differentiation.
30
What is the difference between shallow and deep neural networks?
KnownRevise
A shallow neural network has one (sometimes two) hidden layers, while a deep neural network has many hidden layers stacked between input and output. In theory, a single wide hidden layer can approximate any continuous function (the universal approximation theorem), but it may need an impractically large number of units. Deep networks learn hierarchical representations, simple features in early layers combined into complex ones later, which makes them far more efficient for images, audio and text, at the cost of more data, compute and training care.
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_moons(n_samples=1500, noise=0.2, random_state=42)
def n_params(sizes, n_in=2, n_out=1):
dims = [n_in, *sizes, n_out]
return sum(a * b + b for a, b in zip(dims[:-1], dims[1:]))
configs = {"shallow (1 x 64)": (64,), "deep (4 x 16)": (16, 16, 16, 16)}
for name, sizes in configs.items():
model = make_pipeline(StandardScaler(),
MLPClassifier(hidden_layer_sizes=sizes, max_iter=2000, random_state=42))
acc = cross_val_score(model, X, y, cv=5).mean()
print(f"{name:17s} parameters {n_params(sizes):4d} CV accuracy {acc:.3f}")
On a simple 2D problem both do well; the advantage of depth appears with structured, high-dimensional data such as images, where convolutional deep networks greatly outperform shallow ones.
When to Choose Which
Tabular data with few thousand rows: shallow networks, or more often gradient-boosted trees.
Images, audio, text, video: deep architectures (CNNs, transformers), usually pretrained.
Common Pitfalls
Making a network deeper without residual connections or normalization, which can make training worse.
Using deep networks on small tabular datasets where simpler models win.
31
What are the confusion matrix, precision, recall and F1 score?
KnownRevise
A confusion matrix tabulates a classifier’s predictions against the true labels: for binary problems it counts true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is the share of predicted positives that are correct, TP / (TP + FP). Recall (sensitivity) is the share of actual positives that are found, TP / (TP + FN). F1 is the harmonic mean of precision and recall, 2PR / (P + R), which is high only when both are high. Together they show which errors a model makes, which accuracy alone hides.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Binary Confusion Matrix
Predicted negative
Predicted positive
Actual negative
TN
FP (type I error)
Actual positive
FN (type II error)
TP
When Each Matters
Metric
Prioritize when
Example
Precision
False positives are costly
Spam filter (don’t lose real email), fraud alerts reviewed by people
Recall
False negatives are costly
Cancer screening, safety defects
F1
Need balance, positives are rare
Information extraction, imbalanced classification
F-beta
Weight recall β times as much as precision
F2 for screening
Worked Example
A fraud model flags 50 transactions; 40 are fraud (TP = 40, FP = 10). There were 80 frauds in total (FN = 40). - Precision = 40 / 50 = 0.80. - Recall = 40 / 80 = 0.50. - F1 = 2 × 0.80 × 0.50 / (0.80 + 0.50) ≈ 0.615.
Code Example
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (ConfusionMatrixDisplay, classification_report, confusion_matrix, f1_score,
precision_score, recall_score)
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, weights=[0.9, 0.1], flip_y=0.02, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
for threshold in (0.5, 0.3):
pred = (model.predict_proba(X_test)[:, 1] >= threshold).astype(int)
tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
print(f"threshold {threshold}: TP={tp} FP={fp} FN={fn} TN={tn} | "
f"precision {precision_score(y_test, pred):.3f} recall {recall_score(y_test, pred):.3f} "
f"F1 {f1_score(y_test, pred):.3f}")
print(classification_report(y_test, model.predict(X_test), digits=3))
Lowering the threshold usually raises recall and lowers precision; the threshold should be chosen on validation data according to the costs of each error.
Multi-Class
Compute precision, recall and F1 per class, then average: macro (unweighted mean, treats classes equally), weighted (by class support), or micro (pooled counts; equals accuracy for single-label problems).
Common Pitfalls
Reporting only accuracy on imbalanced data.
Mixing up which class is “positive” (pos_label).
Comparing F1 scores computed at different thresholds without saying so.
32
What is accuracy, and why isn't it always the best metric?
KnownRevise
Accuracy is the fraction of predictions that are correct: (TP + TN) / total. It is intuitive but misleading when classes are imbalanced or errors have different costs. If 99% of transactions are legitimate, a model that always predicts “legitimate” scores 99% accuracy while catching no fraud. Accuracy also ignores the confidence of predictions and depends on a single threshold, so metrics such as precision, recall, F1, ROC AUC, PR AUC and cost-based measures are often more informative.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
When Accuracy Fails
Situation
Problem
Better metric
Rare positive class
Majority-class guessing looks excellent
Recall, precision, F1, PR AUC
Unequal error costs
Treats a missed cancer like a false alarm
Cost-weighted error, recall at fixed precision
Probability quality matters
Ignores calibration
Log loss, Brier score
Ranking matters
Threshold-dependent
ROC AUC, PR AUC
Multi-class with small classes
Dominated by large classes
Macro F1, balanced accuracy
Accuracy is fine when classes are roughly balanced and all errors cost about the same.
Code Example
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, average_precision_score, balanced_accuracy_score, f1_score, recall_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=10000, n_features=20, weights=[0.99, 0.01], flip_y=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
for name, model in [("always majority", DummyClassifier(strategy="most_frequent")),
("random forest", RandomForestClassifier(n_estimators=300, class_weight="balanced",
random_state=42))]:
model.fit(X_train, y_train)
pred = model.predict(X_test)
proba = model.predict_proba(X_test)[:, 1]
print(f"{name:16s} accuracy {accuracy_score(y_test, pred):.3f} | balanced acc "
f"{balanced_accuracy_score(y_test, pred):.3f} | recall {recall_score(y_test, pred, zero_division=0):.3f} | "
f"F1 {f1_score(y_test, pred, zero_division=0):.3f} | PR AUC {average_precision_score(y_test, proba):.3f}")
The dummy model scores about 0.99 accuracy with zero recall; the other metrics expose it immediately. For PR AUC, the no-skill baseline equals the positive rate (about 0.01 here).
Balanced Accuracy
Balanced accuracy averages recall over classes, so a majority-class predictor scores 0.5 in binary problems regardless of imbalance.
Common Pitfalls
Celebrating high accuracy without checking the class distribution.
Choosing a model by accuracy when the business cares about catching the rare class.
33
What are the ROC curve and AUC?
KnownRevise
The ROC (Receiver Operating Characteristic) curve plots a classifier’s true positive rate (recall) against its false positive rate as the decision threshold moves from high to low, showing every trade-off between catching positives and raising false alarms. AUC (area under the ROC curve) summarizes it in one number between 0 and 1: the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one. An AUC of 0.5 is random guessing and 1.0 is perfect ranking.
Threshold-independent: evaluates ranking, not one cut-off.
Insensitive to class balance in its definition, which is both a strength (comparable across datasets) and a weakness: with very rare positives, a high ROC AUC can hide poor precision. Use the precision-recall curve (average precision) there.
Needs scores or probabilities, not hard labels.
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score, roc_curve
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=3000, n_features=10, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
scores = LogisticRegression(max_iter=1000).fit(X_train, y_train).predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, scores)
print("ROC AUC:", round(roc_auc_score(y_test, scores), 3))
print("PR AUC (average precision):", round(average_precision_score(y_test, scores), 3))
# AUC as a ranking probability: compare every positive with every negative
pos, neg = scores[y_test == 1], scores[y_test == 0]
pairwise = (pos[:, None] > neg[None, :]).mean() + 0.5 * (pos[:, None] == neg[None, :]).mean()
print("P(score_pos > score_neg):", round(pairwise, 3))
# Operating point: highest TPR with FPR at most 5%
i = np.where(fpr <= 0.05)[0][-1]
print(f"at threshold {thresholds[i]:.3f}: TPR {tpr[i]:.3f}, FPR {fpr[i]:.3f}")
The pairwise probability equals the ROC AUC, which is the clearest way to explain what AUC means.
Common Pitfalls
Computing AUC from hard 0/1 predictions, which reduces the curve to a single point.
Relying on ROC AUC alone for rare-event detection.
Treating AUC as a measure of calibration; a model can rank well yet output badly calibrated probabilities.
34
How do RMSE and MAE compare as regression metrics?
KnownRevise
MAE (mean absolute error) is the average size of the errors, and RMSE (root mean squared error) is the square root of the average squared error. Both are in the target’s units. The key difference is that RMSE squares errors before averaging, so it penalizes large errors much more: one big miss raises RMSE far more than MAE. RMSE ≥ MAE always, and the gap grows when errors vary a lot. Choose MAE when all errors matter in proportion to their size and outliers should not dominate; choose RMSE when large errors are especially costly.
Errors [1, 1, 1, 1, 6]: MAE = 10 / 5 = 2.0; RMSE = √((1+1+1+1+36)/5) = √8 ≈ 2.83. Errors [2, 2, 2, 2, 2]: MAE = 2.0, RMSE = 2.0. Same MAE, but RMSE reveals the single large error.
Code Example
import numpy as np
from sklearn.metrics import mean_absolute_error, root_mean_squared_error
y_true = np.zeros(5)
for name, errors in [("uniform errors", [2, 2, 2, 2, 2]), ("one big error", [1, 1, 1, 1, 6])]:
y_pred = np.array(errors, dtype=float)
print(f"{name:15s} MAE {mean_absolute_error(y_true, y_pred):.2f} | "
f"RMSE {root_mean_squared_error(y_true, y_pred):.2f}")
# Optimal constant predictions: median minimizes MAE, mean minimizes RMSE
y = np.array([10, 12, 13, 14, 100], dtype=float)
for c in (np.median(y), np.mean(y)):
pred = np.full_like(y, c)
print(f"predict {c:5.1f}: MAE {mean_absolute_error(y, pred):6.2f}, RMSE {root_mean_squared_error(y, pred):6.2f}")
root_mean_squared_error is available from scikit-learn 1.4.
Choosing
Delivery-time estimates where a few very late predictions are unacceptable: RMSE.
Demand forecasts where typical error matters and occasional spikes are noise: MAE.
Report both: a large RMSE/MAE ratio signals occasional big misses.
For relative errors across very different scales, consider MAPE, sMAPE or MASE.
Common Pitfalls
Comparing RMSE across datasets with different target scales.
Training with squared loss but reporting MAE as if they were aligned.
35
When should you use Mean Absolute Percentage Error (MAPE)?
KnownRevise
MAPE (Mean Absolute Percentage Error) is the average of absolute errors divided by the actual values, expressed as a percentage: “on average, forecasts are off by 8%”. Use it when stakeholders think in relative terms, when comparing accuracy across series of very different scales (sales of cheap and expensive products), and when actual values are always well above zero. Avoid it when actuals can be zero or near zero (MAPE explodes or is undefined), and remember that it penalizes over-forecasts more than under-forecasts of the same size.
Asymmetric: under-forecasts are capped at 100% error, over-forecasts are unbounded
Common in business forecasting
Optimizing it biases forecasts low
Weights every observation equally regardless of volume
Alternatives
Metric
Use when
WAPE (sum of absolute errors / sum of actuals)
Aggregate relative error; robust to small individual actuals
sMAPE
Symmetric version, still unstable near zero
MASE
Error relative to a naive forecast; works with zeros
MAE / RMSE
Errors in original units are meaningful
Code Example
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error
actual = np.array([100, 200, 150, 80, 120], dtype=float)
forecast = np.array([110, 190, 160, 70, 130], dtype=float)
print(f"MAPE {mean_absolute_percentage_error(actual, forecast):.1%}") # scikit-learn returns a fraction
print(f"WAPE {np.abs(actual - forecast).sum() / actual.sum():.1%}")
# Near-zero actuals blow MAPE up
actual_small = np.array([100, 200, 150, 1, 120], dtype=float)
forecast_small = np.array([110, 190, 160, 5, 130], dtype=float)
print(f"with a near-zero actual: MAPE {mean_absolute_percentage_error(actual_small, forecast_small):.1%}, "
f"WAPE {np.abs(actual_small - forecast_small).sum() / actual_small.sum():.1%}")
# Asymmetry: same absolute error of 50 on an actual of 100 vs 50
print(f"over-forecast 150 vs actual 100: {mean_absolute_percentage_error([100], [150]):.0%}")
print(f"under-forecast 50 vs actual 100: {mean_absolute_percentage_error([100], [50]):.0%}")
print(f"forecast 100 vs actual 50: {mean_absolute_percentage_error([50], [100]):.0%}")
A forecast of 5 against an actual of 1 is a 400% error, which dominates the average even though the absolute error is tiny.
Common Pitfalls
Using MAPE on intermittent demand (many zeros).
Reporting scikit-learn’s value as a percentage without multiplying by 100.
Choosing models by MAPE without noticing they systematically under-forecast.
36
Why is hyperparameter tuning important in supervised learning?
KnownRevise
Hyperparameters are settings chosen before training, such as tree depth, regularization strength, learning rate or number of neighbours, that control a model’s capacity and how it learns. Tuning them matters because the same algorithm can underfit badly with one setting and overfit with another, and default values rarely suit a specific dataset. Systematic tuning with cross-validation finds the bias-variance balance that generalizes best, often yielding larger gains than switching algorithms, while a held-out test set keeps the final estimate honest.
Read full answerHide full answer
Applies toscikit-learn 1.5+, SciPy (Python 3.12)
Parameters Versus Hyperparameters
Parameters
Hyperparameters
Learned from data during training
Set before training
Regression coefficients, tree split values, network weights
alpha, C, max_depth, learning rate, n_estimators
Common Hyperparameters
Model
Key hyperparameters
Ridge / Lasso / Logistic
alpha or C
k-NN
n_neighbors, weights, distance metric
Decision tree
max_depth, min_samples_leaf, ccp_alpha
Random forest
max_features, min_samples_leaf, n_estimators
Gradient boosting
learning_rate, max_depth/max_leaf_nodes, number of rounds, subsampling
On easy datasets the gain may be small; on harder or noisier data, tuning often matters much more. The tuned CV score is slightly optimistic because it was selected as the maximum; the test score is the honest estimate.
Common Pitfalls
Tuning on the test set.
Tuning too many hyperparameters on small data, which overfits the validation folds.
Ignoring preprocessing choices, which are hyperparameters too (put them in the Pipeline and tune them).
37
What is grid search, and how is it used for hyperparameter optimization?
KnownRevise
Grid search is an exhaustive hyperparameter optimization method: you list candidate values for each hyperparameter, the search trains and evaluates the model for every combination using cross-validation, and the combination with the best mean score is selected and refitted on the full training data. In scikit-learn this is GridSearchCV, which also supports separate grids for different model configurations, multiple metrics and parallel execution.
Define the grid (log-spaced values for scale parameters).
Run GridSearchCV with an appropriate scoring and cv.
Inspect cv_results_: is the best value at the edge of the grid? Are differences within noise?
Evaluate the refitted best model once on the test set.
Code Example
import pandas as pd
from sklearn.datasets import load_wine
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
pipe = Pipeline([("scale", StandardScaler()), ("clf", KNeighborsClassifier())])
param_grid = [ # separate grids per model: the "clf" step itself is a hyperparameter
{"clf": [KNeighborsClassifier()], "clf__n_neighbors": [3, 5, 9, 15], "clf__weights": ["uniform", "distance"]},
{"clf": [LogisticRegression(max_iter=2000)], "clf__C": [0.01, 0.1, 1, 10]},
]
search = GridSearchCV(pipe, param_grid, cv=StratifiedKFold(5, shuffle=True, random_state=42),
scoring="f1_macro", n_jobs=-1)
search.fit(X_train, y_train)
results = pd.DataFrame(search.cv_results_)[["params", "mean_test_score", "std_test_score", "rank_test_score"]]
print(results.sort_values("rank_test_score").head(5).to_string(index=False))
print("best:", search.best_params_)
print("test macro F1:", round(search.score(X_test, y_test), 3))
The search has 8 + 4 = 12 candidates × 5 folds = 60 fits.
Strengths and Weaknesses
Strengths
Weaknesses
Simple, exhaustive, reproducible
Cost multiplies with every hyperparameter
Easy to parallelize
Only tries listed values
Clear results table
Inefficient when few hyperparameters matter
Common Pitfalls
Scaling or selecting features outside the pipeline, which leaks fold data.
Picking a grid whose best value lies on the boundary and stopping there.
Treating best_score_ as an unbiased estimate; it is the maximum of many noisy scores.
38
How does random search compare to grid search?
KnownRevise
Grid search evaluates every combination of hyperparameter values from fixed lists, while random search samples a fixed number of combinations from specified ranges or distributions. With the same budget, random search usually finds equally good or better settings, because typically only a few hyperparameters strongly affect performance: a grid wastes most trials repeating the same values of the important ones, while random sampling tries many different values of every hyperparameter (Bergstra and Bengio, 2012). Grid search remains useful for small, discrete spaces and for exhaustive, reproducible comparisons.
Read full answerHide full answer
Applies toscikit-learn 1.5+, SciPy (Python 3.12)
Comparison
Aspect
Grid search
Random search
Candidates
All combinations
n_iter random samples
Cost growth
Multiplies with each added hyperparameter
Fixed by budget
Coverage of each hyperparameter
Only the listed values
Many distinct values
Continuous ranges
Must discretize
Sample from distributions (log-uniform)
Reproducibility
Deterministic
Deterministic with a seed
Best for
1-3 hyperparameters, few values
3+ hyperparameters, wide ranges
scikit-learn
GridSearchCV
RandomizedSearchCV
Code Example: Same Budget
import numpy as np
from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), SVC())
grid = GridSearchCV(pipe, {"svc__C": np.logspace(-2, 3, 5), "svc__gamma": np.logspace(-4, 0, 5)},
cv=5, scoring="roc_auc", n_jobs=-1).fit(X, y) # 25 candidates
rand = RandomizedSearchCV(pipe, {"svc__C": loguniform(1e-2, 1e3), "svc__gamma": loguniform(1e-4, 1e0)},
n_iter=25, cv=5, scoring="roc_auc", random_state=42, n_jobs=-1).fit(X, y)
print("grid :", round(grid.best_score_, 4), {k: round(v, 5) for k, v in grid.best_params_.items()})
print("random:", round(rand.best_score_, 4), {k: round(v, 5) for k, v in rand.best_params_.items()})
print("distinct C values tried — grid:", len(set(grid.cv_results_["param_svc__C"])),
"| random:", len(set(rand.cv_results_["param_svc__C"])))
With 25 trials, the grid tries only 5 distinct values of each hyperparameter, while random search tries 25 of each.
Beyond Both
Successive halving (HalvingRandomSearchCV) gives more resources to promising candidates.
Bayesian optimization (Optuna, scikit-optimize) uses past results to choose the next trial.
Too few random iterations for a large space; budgets of 30-100 are common.
39
How does the learning rate affect training convergence?
KnownRevise
The learning rate sets the size of each parameter update in gradient-based training: new weights = old weights − learning rate × gradient. It is usually the single most important hyperparameter for neural networks and gradient descent models. Too small, and training converges very slowly or gets stuck; too large, and the loss oscillates or diverges. Good practice is to search it on a log scale and use schedules (warm-up, decay, cosine) or adaptive optimizers (Adam) that adjust step sizes during training.
Read full answerHide full answer
Applies toNumPy 2.x, PyTorch 2.x (Python 3.12)
Effects
Learning rate
Behaviour
Much too small
Loss falls very slowly; may stall in flat regions
Good
Fast, steady decrease
Slightly too large
Noisy loss, may settle at a worse point
Much too large
Loss oscillates or explodes (NaN)
For a quadratic loss with curvature L (largest eigenvalue of the Hessian), plain gradient descent converges only if the learning rate is below 2 / L.
Schedules and Optimizers
Technique
Idea
Step / exponential decay
Reduce the rate as training progresses
Cosine annealing
Smoothly decrease to a minimum, optionally with restarts
Warm-up
Start small, increase over the first steps (standard for transformers)
ReduceLROnPlateau
Lower the rate when validation loss stalls
One-cycle
Rise then fall within one run
Adam / AdamW
Per-parameter adaptive step sizes
LR range test
Increase the rate during a short run and pick a value below where loss starts to blow up
Code Example: Gradient Descent on a Quadratic
import numpy as np
# Loss = 0.5 * (10 * w1^2 + 1 * w2^2): curvatures 10 and 1, so the stable limit is lr < 2 / 10 = 0.2
H = np.diag([10.0, 1.0])
def loss(w): return 0.5 * w @ H @ w
for lr in (0.01, 0.1, 0.19, 0.21):
w = np.array([1.0, 1.0])
for _ in range(100):
w = w - lr * (H @ w) # gradient of the loss is H w
print(f"lr={lr:<5} loss after 100 steps: {loss(w):.3e}")
At 0.01 the loss is still large because the low-curvature direction moves slowly; 0.1 converges fast; 0.19 converges with oscillation; 0.21 diverges.
PyTorch Schedule Example
import torch
model = torch.nn.Linear(10, 1)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-2, total_steps=1000)
for step in range(1000):
optimizer.zero_grad()
loss = model(torch.randn(32, 10)).pow(2).mean()
loss.backward()
optimizer.step()
scheduler.step() # update the learning rate every step
print("final lr:", scheduler.get_last_lr()[0])
Common Pitfalls
Tuning everything else before the learning rate.
Changing the batch size without adjusting the learning rate.
Calling scheduler.step() at the wrong frequency (per epoch vs per batch).
40
What is early stopping in training machine learning models?
KnownRevise
Early stopping halts training when performance on a validation set stops improving, and keeps the model from the best point. In iterative learners such as neural networks and gradient boosting, training error keeps falling while validation error eventually starts to rise as the model begins to memorize noise; stopping at the validation minimum acts as regularization and also saves compute. A patience setting tolerates short plateaus before stopping.
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
X, y = make_classification(n_samples=5000, n_features=30, n_informative=10, flip_y=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=42)
# scikit-learn: internal validation split
hgb = HistGradientBoostingClassifier(max_iter=2000, learning_rate=0.1, early_stopping=True,
validation_fraction=0.15, n_iter_no_change=20, random_state=42)
hgb.fit(X_train, y_train)
print("HistGradientBoosting stopped after", hgb.n_iter_, "of 2000 iterations; test acc",
round(hgb.score(X_test, y_test), 3))
# XGBoost: explicit validation set
xgb = XGBClassifier(n_estimators=2000, learning_rate=0.05, max_depth=4, early_stopping_rounds=50,
eval_metric="logloss", random_state=42)
xgb.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
print("XGBoost best iteration:", xgb.best_iteration, "| test acc", round(xgb.score(X_test, y_test), 3))
Common Pitfalls
Using the test set for early stopping, which makes the test score optimistic.
Too little patience with noisy validation curves, stopping too early.
Not restoring the best weights in neural networks.
A tiny validation set that makes the stopping point random.
41
How would you apply supervised learning to predict stock prices?
KnownRevise
Applying supervised learning to stock prices means framing a target such as next-day return or direction, building features only from information available at prediction time (past returns, volatility, volume, market and macro indicators), and training a model with time-ordered validation. The realistic goal is a small, robust edge after costs, not accurate price levels: prices are close to a random walk, so predicting raw prices mostly reproduces yesterday’s price. A sound answer emphasizes leakage control, baselines and backtesting more than model choice.
Next-day (or next-week) log return (regression) or its sign (classification)
Features
Lagged returns, rolling volatility, momentum, volume change, index return; all shifted so they use only past data
Split
Chronological; walk-forward with TimeSeriesSplit
Models
Ridge / logistic regression, gradient boosting; deep models only with lots of data
Baselines
Zero return, previous return, always “up”
Evaluation
MAE or directional accuracy, plus a backtest with transaction costs
Code Example
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n = 3000
ret = rng.normal(0.0003, 0.01, n) # simulated daily returns (no real signal)
df = pd.DataFrame({"ret": ret})
for lag in (1, 2, 5):
df[f"ret_lag{lag}"] = df["ret"].shift(lag)
df["vol_20"] = df["ret"].rolling(20).std().shift(1)
df["mom_10"] = df["ret"].rolling(10).sum().shift(1)
df["target"] = (df["ret"] > 0).astype(int) # direction of today's return, from past features
df = df.dropna()
features = [c for c in df.columns if c.startswith(("ret_lag", "vol", "mom"))]
X, y = df[features], df["target"]
cv = TimeSeriesSplit(n_splits=5)
for name, model in [("logistic", make_pipeline(StandardScaler(), LogisticRegression())),
("boosting", HistGradientBoostingClassifier(max_depth=3, random_state=42))]:
acc = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
print(f"{name:9s} walk-forward accuracy {acc.mean():.3f} ± {acc.std():.3f}")
print("always-up baseline:", round(y.mean(), 3))
The returns are simulated noise, so a correct pipeline should not beat the always-up baseline meaningfully. If a pipeline beats noise, it has a leakage bug. Run the same code on real data only after this sanity check passes.
Backtest Essentials
Convert predictions into positions, apply realistic costs and slippage, and compute risk-adjusted returns (Sharpe ratio, drawdown).
Test on a final out-of-sample period never used for development.
Common Pitfalls
Features computed with data from the same day as the target (look-ahead bias).
Random K-fold splits that mix future and past.
Survivorship bias from using only today’s listed companies.
Data snooping: trying hundreds of models on the same history.
42
How is supervised learning applied to credit scoring?
KnownRevise
Credit scoring uses supervised learning to estimate the probability that an applicant will default (e.g. 90+ days past due within 12 months) from application data, credit bureau history and, where allowed, transaction data. The model’s probabilities drive approve/decline decisions, credit limits and pricing. Because lending is regulated and decisions affect people, the work emphasizes calibrated probabilities, interpretability (reason codes for declines), fairness and stability over time, so logistic regression scorecards remain common alongside gradient-boosted trees with explanations.
Default within a fixed window after the loan starts
Sample
Booked loans with matured outcomes; note reject-inference issues (rejected applicants have no outcome)
Features
Income, debt-to-income, utilization, delinquencies, credit history length, inquiries
Models
Logistic regression scorecard (often with binned/WoE features), gradient boosting
Metrics
ROC AUC / Gini (= 2·AUC − 1), KS statistic, calibration, profit at the cut-off
Validation
Out-of-time test set (later application dates)
Governance
Explainability, adverse-action reasons, fairness tests, model risk documentation, monitoring (PSI)
Code Example
import numpy as np
import pandas as pd
from sklearn.calibration import calibration_curve
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n = 20000
df = pd.DataFrame({
"month": rng.integers(0, 24, n), # application month
"income": rng.lognormal(10.8, 0.5, n),
"dti": rng.uniform(0, 0.6, n),
"utilization": rng.beta(2, 3, n),
"delinquencies": rng.poisson(0.3, n),
"history_years": rng.uniform(0, 25, n),
})
logit = (-3 + 3 * df["dti"] + 2.5 * df["utilization"] + 0.6 * df["delinquencies"]
- 0.05 * df["history_years"] - 0.4 * (np.log(df["income"]) - 10.8))
df["default"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
features = ["income", "dti", "utilization", "delinquencies", "history_years"]
train, test = df[df["month"] < 18], df[df["month"] >= 18] # out-of-time split
models = {"logistic scorecard": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"gradient boosting": HistGradientBoostingClassifier(max_depth=3, random_state=42)}
for name, model in models.items():
model.fit(train[features], train["default"])
p = model.predict_proba(test[features])[:, 1]
auc = roc_auc_score(test["default"], p)
frac, mean_p = calibration_curve(test["default"], p, n_bins=5, strategy="quantile")
print(f"{name:19s} AUC {auc:.3f} Gini {2 * auc - 1:.3f} | predicted vs observed default rate by quintile: "
f"{[(round(a, 3), round(b, 3)) for a, b in zip(mean_p, frac)]}")
coefs = models["logistic scorecard"][-1].coef_[0]
print(dict(zip(features, coefs.round(2)))) # signs give reason codes: higher DTI raises risk
Common Pitfalls
Random splits instead of out-of-time validation.
Features unavailable at application time (e.g. post-approval behaviour).
Using protected attributes or close proxies; test outcomes across groups and follow local regulation.
Ignoring population drift; monitor the population stability index and recalibrate.
43
How can supervised learning be used in recommender systems?
KnownRevise
Supervised learning powers most production recommender systems by framing recommendation as predicting an outcome for a (user, item, context) pair: will the user click, buy, watch to the end, or what rating will they give? Models are trained on logged interactions with features about the user, the item and the context, and their predicted scores rank candidate items. Systems typically have two stages: candidate generation (retrieving hundreds of items quickly, often with embeddings) and ranking (a supervised model such as gradient boosting or a deep network scoring each candidate).
User history aggregates, item popularity and attributes, user-item affinity (category match, past interactions), context (time, device), and embeddings from collaborative filtering.
Code Example: Click Prediction Ranker
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import ndcg_score, roc_auc_score
rng = np.random.default_rng(42)
n_users, n_items, n_rows = 500, 200, 40000
user_pref = rng.integers(0, 5, n_users) # favourite category per user
item_cat = rng.integers(0, 5, n_items)
item_pop = rng.beta(2, 5, n_items)
rows = pd.DataFrame({"user": rng.integers(0, n_users, n_rows), "item": rng.integers(0, n_items, n_rows),
"day": rng.integers(0, 30, n_rows)})
rows["cat_match"] = (user_pref[rows["user"]] == item_cat[rows["item"]]).astype(int)
rows["popularity"] = item_pop[rows["item"]]
rows["hour"] = rng.integers(0, 24, n_rows)
p_click = 1 / (1 + np.exp(-(-3 + 2.0 * rows["cat_match"] + 3 * rows["popularity"])))
rows["click"] = (rng.random(n_rows) < p_click).astype(int)
features = ["cat_match", "popularity", "hour"]
train, test = rows[rows["day"] < 25], rows[rows["day"] >= 25] # time-based split
model = HistGradientBoostingClassifier(random_state=42).fit(train[features], train["click"])
test = test.assign(score=model.predict_proba(test[features])[:, 1])
print("ROC AUC:", round(roc_auc_score(test["click"], test["score"]), 3))
# Ranking quality per user: NDCG@10 over each user's test impressions
ndcgs = [ndcg_score([g["click"].to_numpy()], [g["score"].to_numpy()], k=10)
for _, g in test.groupby("user") if g["click"].sum() > 0 and len(g) > 1]
print("mean NDCG@10:", round(float(np.mean(ndcgs)), 3))
Common Pitfalls
Feedback loops: logged data only contains items the old system showed; consider exploration and position-bias correction.
Random splits that leak future interactions; split by time.
Optimizing clicks alone, which can promote clickbait; include longer-term signals.
44
How is supervised learning applied in healthcare diagnostics?
KnownRevise
In healthcare diagnostics, supervised learning trains models on patient data labelled with confirmed diagnoses or outcomes to predict disease presence or risk for new patients. Common inputs are medical images (X-rays, CT, retinal photos, pathology slides), signals (ECG), lab results and electronic health records. Models act as decision support, flagging likely cases, prioritizing worklists or estimating risk, and must meet stricter standards than most ML: high sensitivity for serious conditions, calibrated risk estimates, validation on external populations, fairness across patient groups and regulatory approval for clinical use.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Applications
Area
Input
Model
Radiology
Chest X-ray, CT
CNNs / vision transformers (often pretrained)
Ophthalmology
Retinal fundus images
CNNs for diabetic retinopathy
Pathology
Whole-slide images
Patch-based CNNs, multiple-instance learning
Cardiology
ECG
1D CNNs
Clinical risk
EHR (labs, vitals, history)
Logistic regression, gradient boosting
Laboratory
Blood panels
Tree ensembles
Workflow
Define the clinical question and the action the prediction triggers.
Collect data with confirmed labels; split by patient (and ideally by site or time).
Choose the threshold with clinicians based on error costs.
Validate externally, check subgroups, and document limitations.
Deploy with monitoring and a human in the loop.
Code Example: Patient-Level Grouped Validation
import numpy as np
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import recall_score, roc_auc_score
from sklearn.model_selection import GroupKFold, KFold, cross_val_predict
rng = np.random.default_rng(42)
n_patients, visits = 400, 5
patient = np.repeat(np.arange(n_patients), visits)
baseline = rng.normal(0, 1, n_patients)[patient] # patient-specific signal
disease = (rng.random(n_patients) < 0.2).astype(int)[patient]
X = np.column_stack([baseline + rng.normal(0, 0.1, len(patient)), # nearly identical across a patient's visits
0.6 * disease + rng.normal(0, 1, len(patient)),
rng.normal(0, 1, len(patient))])
y = disease
model = HistGradientBoostingClassifier(max_depth=3, random_state=42)
for name, cv, groups in [("random K-fold (leaky)", KFold(5, shuffle=True, random_state=42), None),
("GroupKFold by patient", GroupKFold(5), patient)]:
p = cross_val_predict(model, X, y, cv=cv, groups=groups, method="predict_proba")[:, 1]
print(f"{name:22s} ROC AUC {roc_auc_score(y, p):.3f}, sensitivity at 0.3: {recall_score(y, (p >= 0.3).astype(int)):.3f}")
With random splits the model can recognize each patient’s fingerprint feature from their other visits, so its scores look better than they would for new patients; grouping by patient gives the honest estimate.
Common Pitfalls
Image-level splits when one patient has many images.
Optimizing accuracy for rare conditions; prioritize sensitivity at an acceptable specificity.
45
What role does supervised learning play in natural language processing?
KnownRevise
Supervised learning is central to NLP: tasks such as sentiment analysis, spam detection, topic classification, named-entity recognition, question answering and machine translation are all trained on labelled text pairs. Classic approaches use TF-IDF features with linear models; modern approaches fine-tune pretrained transformers (themselves trained with self-supervised objectives) on a labelled task dataset, and large language models are further refined with supervised instruction tuning on curated prompt-response pairs. Label quality, task definition and evaluation remain the deciding factors.
Read full answerHide full answer
Applies toscikit-learn 1.5+, Hugging Face Transformers 4.4x (optional) (Python 3.12)
Supervised NLP Tasks
Task
Input → label
Typical model
Text classification
Document → category
TF-IDF + linear model; fine-tuned BERT-style encoder
Sentiment analysis
Review → positive/negative/neutral
Same
Named-entity recognition
Token → entity tag (BIO)
Token-classification transformer, CRF
Question answering
Question + passage → answer span
Encoder with span head
Translation, summarization
Text → text
Encoder-decoder (T5, BART)
Instruction tuning
Prompt → desired response
Supervised fine-tuning of LLMs
Approaches Compared
Approach
Labelled data needed
Accuracy
Cost
TF-IDF + logistic regression / linear SVM
Hundreds to thousands
Strong baseline
Very low
Fine-tuned transformer
Hundreds to thousands
Usually best
GPU training
Zero/few-shot LLM prompting
None to a few
Good for many tasks
Inference cost, less control
Code Example: Classic Supervised Baseline
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.pipeline import make_pipeline
cats = ["rec.sport.baseball", "sci.space", "talk.politics.mideast"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))
model = make_pipeline(TfidfVectorizer(sublinear_tf=True, ngram_range=(1, 2), min_df=2),
LogisticRegression(max_iter=2000, C=10))
model.fit(train.data, train.target)
print(classification_report(test.target, model.predict(test.data), target_names=train.target_names, digits=3))
A transformer fine-tune (for example AutoModelForSequenceClassification from Hugging Face with a pretrained checkpoint) is the next step when the baseline is not enough.
Common Pitfalls
Inconsistent annotation guidelines, which cap model accuracy; measure inter-annotator agreement.
Leakage through duplicated or near-duplicated texts across splits.
Domain shift (training on news, deploying on social media).
46
Implement linear regression from scratch in Python.
KnownRevise
Linear regression from scratch fits weights w and bias b so that predictions X w + b minimize the mean squared error. There are two standard approaches: the closed-form least-squares solution (solve the normal equations, done stably with lstsq), and gradient descent, which repeatedly moves the weights against the gradient of the error. The implementation below provides both, adds optional L2 regularization, and checks the results against scikit-learn.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+ (for data and comparison) (Python 3.12)
Math in Words
Prediction: weighted sum of features plus bias.
Loss: average of squared differences between predictions and targets.
Gradient of the loss with respect to w: (2/n) Xᵀ(Xw + b − y); with respect to b: (2/n) × sum(Xw + b − y).
Closed form: w = (XᵀX)⁻¹Xᵀy on features augmented with a column of ones; never invert the matrix directly in practice, use a least-squares solver.
Code Example
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
class LinearRegressionScratch:
def __init__(self, method="closed_form", lr=0.05, epochs=5000, l2=0.0):
self.method, self.lr, self.epochs, self.l2 = method, lr, epochs, l2
def fit(self, X, y):
n, d = X.shape
if self.method == "closed_form":
Xb = np.c_[np.ones(n), X]
reg = self.l2 * np.eye(d + 1)
reg[0, 0] = 0 # do not penalize the bias
theta = np.linalg.lstsq(Xb.T @ Xb + reg, Xb.T @ y, rcond=None)[0]
self.b, self.w = theta[0], theta[1:]
else:
self.w, self.b = np.zeros(d), 0.0
for _ in range(self.epochs):
error = X @ self.w + self.b - y
self.w -= self.lr * (2 / n * X.T @ error + 2 * self.l2 / n * self.w)
self.b -= self.lr * (2 / n * error.sum())
return self
def predict(self, X):
return X @ self.w + self.b
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train) # gradient descent needs scaled features
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
closed = LinearRegressionScratch("closed_form").fit(X_train, y_train)
gd = LinearRegressionScratch("gradient_descent").fit(X_train, y_train)
ref = LinearRegression().fit(X_train, y_train)
print("closed form R²:", round(r2_score(y_test, closed.predict(X_test)), 4))
print("grad descent R²:", round(r2_score(y_test, gd.predict(X_test)), 4))
print("sklearn R²: ", round(ref.score(X_test, y_test), 4))
print("closed form matches sklearn:", np.allclose(closed.w, ref.coef_))
print("largest weight difference, GD vs exact:", round(float(np.abs(gd.w - ref.coef_).max()), 3))
Gradient descent approaches the exact solution gradually; strongly correlated features in this dataset (such as the cholesterol measures) slow the final convergence, so its weights can still differ somewhat even when its R² is nearly identical.
Common Pitfalls
Unscaled features with gradient descent, which needs a tiny learning rate and many epochs.
Inverting XᵀX explicitly, which is numerically unstable with correlated features.
Penalizing the bias term in the regularized version.
47
Write a logistic regression classifier using scikit-learn.
KnownRevise
A logistic regression classifier in scikit-learn is built with LogisticRegression, ideally inside a Pipeline with StandardScaler so features are scaled consistently in training and prediction. The workflow is: split the data, compare against a dummy baseline, tune the regularization strength C with cross-validation, evaluate on the test set with suitable metrics, choose a decision threshold if errors have different costs, and interpret the coefficients as odds ratios.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2,
stratify=data.target, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
print("baseline CV accuracy:", round(cross_val_score(DummyClassifier(), X_train, y_train, cv=cv).mean(), 3))
pipe = Pipeline([("scale", StandardScaler()), ("logreg", LogisticRegression(max_iter=1000))])
grid = GridSearchCV(pipe, {"logreg__C": np.logspace(-3, 2, 11), "logreg__class_weight": [None, "balanced"]},
cv=cv, scoring="roc_auc")
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV ROC AUC:", round(grid.best_score_, 4))
model = grid.best_estimator_
proba = model.predict_proba(X_test)[:, 1]
pred = model.predict(X_test)
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=data.target_names, digits=3))
# Interpret: odds ratio per one standard deviation increase
coefs = model.named_steps["logreg"].coef_[0]
top = np.argsort(-np.abs(coefs))[:5]
for i in top:
print(f"{data.feature_names[i]:22s} coef {coefs[i]:+.2f} odds ratio {np.exp(coefs[i]):.2f}")
Useful Options
Option
Purpose
C
Inverse regularization strength (small C = stronger penalty)
penalty="l1" with solver="liblinear" or "saga"
Sparse coefficients
class_weight="balanced"
Imbalanced classes
max_iter
Raise if you see convergence warnings
Multi-class targets
Handled automatically with a multinomial (softmax) model
Common Pitfalls
Skipping scaling, which slows convergence and makes the penalty uneven.
Using predict (threshold 0.5) when costs or class balance call for another threshold.
Interpreting coefficients of correlated features as independent effects.
48
Build and visualize a decision tree classifier on a dataset.
KnownRevise
A decision tree classifier splits the data with a sequence of feature thresholds chosen to make each group as pure as possible, and its structure can be shown directly as a diagram or as text rules, which makes it one of the most explainable models. The example trains a depth-limited tree on the iris dataset, chooses the depth with cross-validation, evaluates it, and visualizes it with plot_tree and export_text, plus the decision regions for two features.
import matplotlib
matplotlib.use("Agg") # write images to files
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text, plot_tree
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3,
stratify=iris.target, random_state=42)
# Choose depth and leaf size with cross-validation
search = GridSearchCV(DecisionTreeClassifier(random_state=42),
{"max_depth": [1, 2, 3, 4, 5, None], "min_samples_leaf": [1, 3, 5]}, cv=5)
search.fit(X_train, y_train)
tree = search.best_estimator_
print("best:", search.best_params_)
print(classification_report(y_test, tree.predict(X_test), target_names=iris.target_names))
# 1. Text rules
print(export_text(tree, feature_names=list(iris.feature_names)))
# 2. Tree diagram
fig, ax = plt.subplots(figsize=(14, 7))
plot_tree(tree, feature_names=iris.feature_names, class_names=iris.target_names,
filled=True, rounded=True, impurity=True, ax=ax)
plt.savefig("iris_tree.png", dpi=120, bbox_inches="tight")
# 3. Decision regions using the two petal features
two = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train[:, 2:], y_train)
fig, ax = plt.subplots(figsize=(6, 5))
DecisionBoundaryDisplay.from_estimator(two, X_train[:, 2:], response_method="predict", alpha=0.3, ax=ax,
xlabel="petal length (cm)", ylabel="petal width (cm)")
ax.scatter(X_train[:, 2], X_train[:, 3], c=y_train, edgecolors="k", s=25)
plt.savefig("iris_tree_regions.png", dpi=120, bbox_inches="tight")
Reading the Diagram
Each internal node shows the split rule, the impurity (Gini), the number of samples and the class counts (value).
Colour intensity reflects the majority class and its purity.
Leaves give the predicted class; the path from root to leaf is an if-then rule.
Decision regions are axis-aligned rectangles, a hallmark of trees.
Common Pitfalls
Visualizing an unpruned tree with hundreds of nodes; limit depth for explanation.
Treating the tree as stable: retraining on a slightly different sample can change the rules.
49
Build and train a simple feedforward neural network with Keras.
KnownRevise
A feedforward neural network (multi-layer perceptron) passes inputs through stacked fully connected layers with non-linear activations to an output layer suited to the task. In Keras you define it with keras.Sequential, compile it with an optimizer, loss and metrics, and train it with fit, using a validation split and early stopping to avoid overfitting. The example classifies breast-cancer biopsies, with scaling inside the model via a Normalization layer.
Read full answerHide full answer
Applies toTensorFlow 2.16+ with Keras 3, scikit-learn 1.5+ (Python 3.12)
Code Example
import keras
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
keras.utils.set_random_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X = X.astype("float32")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=42)
norm = keras.layers.Normalization()
norm.adapt(X_tr) # scaling statistics from the training split only
model = keras.Sequential([
keras.Input(shape=(X.shape[1],)),
norm,
keras.layers.Dense(32, activation="relu"),
keras.layers.Dropout(0.2),
keras.layers.Dense(16, activation="relu"),
keras.layers.Dense(1, activation="sigmoid"), # probability of the positive class
])
model.compile(optimizer=keras.optimizers.Adam(1e-3), loss="binary_crossentropy",
metrics=["accuracy", keras.metrics.AUC(name="auc")])
model.summary()
history = model.fit(X_tr, y_tr, validation_data=(X_val, y_val), epochs=200, batch_size=32, verbose=0,
callbacks=[keras.callbacks.EarlyStopping(monitor="val_auc", mode="max", patience=15,
restore_best_weights=True)])
print("stopped after", len(history.history["loss"]), "epochs")
proba = model.predict(X_test, verbose=0).ravel()
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))
print(classification_report(y_test, (proba >= 0.5).astype(int), digits=3))
model.save("ffn_breast_cancer.keras")
Design Choices
Choice
Reason
Normalization layer inside the model
Same scaling at training and inference
ReLU hidden layers
Fast, avoids vanishing gradients
Sigmoid output + binary cross-entropy
Binary classification
Dropout + early stopping
Limits overfitting on a small dataset
Separate validation split
Early stopping without touching the test set
For multi-class problems use Dense(K, activation="softmax") with sparse_categorical_crossentropy; for regression use Dense(1) with mse or mae.
Common Pitfalls
Fitting the normalization on all data, including validation and test rows.
Monitoring a metric name that does not exist (e.g. val_auc without naming the metric “auc”).
On small tabular datasets, a neural network may not beat logistic regression or gradient boosting; compare.
50
Write a Python function to compute the F1 score from a confusion matrix.
KnownRevise
The F1 score is the harmonic mean of precision and recall. From a binary confusion matrix it is 2·TP / (2·TP + FP + FN), which equals 2PR / (P + R) with precision P = TP / (TP + FP) and recall R = TP / (TP + FN). For multi-class confusion matrices, compute per-class F1 by treating each class as positive in turn, then average them: macro (simple mean), weighted (by class support) or micro (from pooled counts). The function below handles all of these and zero-division edge cases.
Swapping FP and FN by reading the matrix with the wrong orientation.
Averaging per-class precision and recall first, then computing F1 from the averages; that is not macro F1.
51
How do you deal with imbalanced datasets in classification tasks?
KnownRevise
Imbalanced datasets, where one class is much rarer than another (fraud, disease, defects), make standard training and metrics favour the majority class. The fix has three parts: evaluate properly (precision, recall, F1, PR AUC instead of accuracy), adjust learning (class weights, resampling such as SMOTE or undersampling, or algorithms robust to imbalance), and choose the decision threshold from validation data according to the cost of each error. Often class weights plus threshold tuning are enough; resampling helps in some cases but must stay inside cross-validation.
import numpy as np
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline as imb_pipeline
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, f1_score, precision_recall_curve
from sklearn.model_selection import cross_val_predict, train_test_split, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=20000, n_features=20, n_informative=6, weights=[0.98, 0.02],
flip_y=0.005, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
models = {
"plain": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"class_weight": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, class_weight="balanced")),
"SMOTE": imb_pipeline(StandardScaler(), SMOTE(random_state=42), LogisticRegression(max_iter=1000)),
}
for name, model in models.items():
# Out-of-fold probabilities on training data are used to pick the threshold
oof = cross_val_predict(model, X_train, y_train, cv=cv, method="predict_proba")[:, 1]
prec, rec, thr = precision_recall_curve(y_train, oof)
f1 = 2 * prec[:-1] * rec[:-1] / np.maximum(prec[:-1] + rec[:-1], 1e-12)
best_thr = thr[np.argmax(f1)]
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
print(f"{name:13s} PR AUC {average_precision_score(y_test, proba):.3f} | F1 @0.5 "
f"{f1_score(y_test, (proba >= 0.5).astype(int)):.3f} | F1 @tuned {best_thr:.2f}: "
f"{f1_score(y_test, (proba >= best_thr).astype(int)):.3f}")
PR AUC measures ranking quality and is unaffected by the threshold; the F1 columns show how much the threshold choice matters.
Common Pitfalls
Resampling before splitting or outside the CV pipeline, which leaks synthetic copies of test-like points.
Tuning the threshold on the test set.
Assuming SMOTE always helps; compare it against class weights.
52
How are SVMs used with non-linear kernels?
KnownRevise
Non-linear kernels let an SVM learn curved decision boundaries while keeping the same maximum-margin optimization. A kernel function computes the similarity of two points as if they had been mapped into a higher-dimensional feature space, where a linear separator may exist; the SVM never computes that mapping explicitly (the kernel trick). The RBF (Gaussian) kernel is the default choice for non-linear problems, with the polynomial kernel used when feature interactions of a known degree matter; both require scaled features and joint tuning of C with the kernel parameters.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Kernels
Kernel
Similarity
Parameters
Boundary
Linear
Dot product
C
Straight hyperplane
Polynomial
(gamma × dot product + coef0)^degree
degree, gamma, coef0, C
Polynomial curves
RBF
exp(−gamma × squared distance)
gamma, C
Flexible, local
Sigmoid
tanh(gamma × dot product + coef0)
gamma, coef0, C
Rarely best
The decision function becomes a weighted sum of kernel similarities to the support vectors plus a bias, so prediction cost grows with the number of support vectors.
Tuning
C: penalty for margin violations (large = less regularization).
gamma (RBF): reach of each support vector; large gamma gives wiggly boundaries that overfit, small gamma gives smooth ones that may underfit.
Search both on log scales together.
Code Example
import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_circles(n_samples=600, factor=0.5, noise=0.12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
param_grid = [
{"svc__kernel": ["linear"], "svc__C": [0.1, 1, 10]},
{"svc__kernel": ["poly"], "svc__degree": [2, 3], "svc__coef0": [0, 1], "svc__C": [0.1, 1, 10]},
{"svc__kernel": ["rbf"], "svc__gamma": np.logspace(-2, 1, 4), "svc__C": [0.1, 1, 10, 100]},
]
search = GridSearchCV(make_pipeline(StandardScaler(), SVC()), param_grid, cv=5, n_jobs=-1)
search.fit(X_train, y_train)
for kernel in ("linear", "poly", "rbf"):
mask = np.array([p["svc__kernel"] == kernel for p in search.cv_results_["params"]])
print(f"{kernel:7s} best CV accuracy {search.cv_results_['mean_test_score'][mask].max():.3f}")
print("overall best:", search.best_params_, "| test accuracy:", round(search.score(X_test, y_test), 3))
Concentric circles cannot be separated by a line, so the linear kernel should sit near 50% while polynomial (degree 2) and RBF kernels separate them well.
When to Use Non-Linear Kernels
Low- to medium-dimensional data with curved class boundaries.
Up to tens of thousands of samples (training is roughly O(n²) to O(n³)).
For larger data, approximate the kernel (Nystroem, RBFSampler) and use a linear SVM.
Common Pitfalls
No feature scaling, which distorts distances.
Large gamma and C together, which memorizes the training set.
Using RBF on very high-dimensional sparse text, where a linear kernel is usually as accurate and much faster.
53
How are decision trees pruned?
KnownRevise
Pruning reduces a decision tree’s size to prevent overfitting. Pre-pruning (early stopping) stops growth during construction using limits such as maximum depth, minimum samples per leaf or minimum impurity decrease. Post-pruning grows a large tree first and then removes branches that add little predictive value; scikit-learn implements minimal cost-complexity pruning, which trades the tree’s training error against its number of leaves with a parameter ccp_alpha chosen by cross-validation.
A split must reduce impurity by at least this much
Cost-Complexity Pruning
For each subtree T, cost = total leaf impurity (training error) + alpha × number of leaves.
As alpha increases, the weakest links (branches whose removal increases error the least per removed leaf) are pruned first, producing a nested sequence of smaller trees.
cost_complexity_pruning_path returns the effective alphas; choose the best one with cross-validation.
Other post-pruning methods include reduced-error pruning (remove nodes if validation accuracy doesn’t drop) and C4.5’s pessimistic error pruning.
Code Example
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
full = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
path = full.cost_complexity_pruning_path(X_train, y_train)
alphas = np.unique(path.ccp_alphas[:-1]) # drop the alpha that prunes to a single node
search = GridSearchCV(DecisionTreeClassifier(random_state=42), {"ccp_alpha": alphas}, cv=5)
search.fit(X_train, y_train)
pruned = search.best_estimator_
pre = GridSearchCV(DecisionTreeClassifier(random_state=42),
{"max_depth": [2, 3, 4, 5, 6], "min_samples_leaf": [1, 5, 10]}, cv=5).fit(X_train, y_train)
for name, tree in [("unpruned", full), ("post-pruned (ccp)", pruned), ("pre-pruned", pre.best_estimator_)]:
print(f"{name:18s} leaves {tree.get_n_leaves():3d}, depth {tree.get_depth():2d}, "
f"train {tree.score(X_train, y_train):.3f}, test {tree.score(X_test, y_test):.3f}")
print("chosen ccp_alpha:", round(search.best_params_["ccp_alpha"], 5))
The pruned trees should be much smaller with similar or better test accuracy than the unpruned tree, which has perfect training accuracy.
Common Pitfalls
Choosing ccp_alpha or depth by test accuracy.
Over-pruning, which underfits.
Pruning trees inside random forests: forests rely on deep, low-bias trees and averaging instead.
54
What is AdaBoost, and how does its algorithm work?
KnownRevise
AdaBoost (Adaptive Boosting) builds a strong classifier from many weak learners, usually decision stumps (one-split trees), trained one after another. After each round it increases the weights of misclassified training examples so the next learner focuses on them, and it gives each learner a vote proportional to its accuracy. The final prediction is the weighted vote of all learners. AdaBoost was the first practical boosting algorithm (Freund and Schapire, 1997) and minimizes an exponential loss.
Weighted error: err_m = sum of weights of misclassified points.
Learner weight: alpha_m = 0.5 × ln((1 − err_m) / err_m); accurate learners get large votes.
Update: multiply weights of misclassified points by exp(alpha_m) and correct ones by exp(−alpha_m), then normalize.
Predict sign(sum_m alpha_m h_m(x)).
A learner with error 0.5 (random) gets alpha = 0; lower error gives a larger vote.
Code Example: From Scratch vs scikit-learn
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y01 = make_classification(n_samples=2000, n_features=10, n_informative=5, random_state=42)
y = np.where(y01 == 1, 1, -1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
def adaboost_fit(X, y, rounds=100):
n = len(y)
w = np.full(n, 1 / n)
learners, alphas = [], []
for _ in range(rounds):
stump = DecisionTreeClassifier(max_depth=1, random_state=42).fit(X, y, sample_weight=w)
pred = stump.predict(X)
err = np.clip(w[pred != y].sum(), 1e-10, 1 - 1e-10)
alpha = 0.5 * np.log((1 - err) / err)
w *= np.exp(-alpha * y * pred) # up-weights mistakes, down-weights correct points
w /= w.sum()
learners.append(stump)
alphas.append(alpha)
return learners, np.array(alphas)
def adaboost_predict(learners, alphas, X):
return np.sign(sum(a * h.predict(X) for h, a in zip(learners, alphas)))
learners, alphas = adaboost_fit(X_train, y_train)
print("scratch AdaBoost accuracy:", round((adaboost_predict(learners, alphas, X_test) == y_test).mean(), 3))
print("first learner weights:", alphas[:5].round(3))
ref = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=100, random_state=42).fit(X_train, y_train)
print("scikit-learn AdaBoost accuracy:", round(ref.score(X_test, y_test), 3))
print("single stump accuracy:", round(DecisionTreeClassifier(max_depth=1).fit(X_train, y_train).score(X_test, y_test), 3))
scikit-learn’s SAMME variant uses alpha = ln((1 − err)/err) + ln(K − 1) with a learning rate, which for two classes is twice the value above; predictions are the same.
Strengths and Weaknesses
Strengths
Weaknesses
Simple, few hyperparameters
Sensitive to label noise and outliers (their weights grow exponentially)
Turns weak learners into a strong model
Sequential, so harder to parallelize
Often resists overfitting in practice
Usually outperformed by gradient boosting today
Common Pitfalls
Using strong base learners (deep trees), which overfit quickly.
Applying AdaBoost to very noisy labels.
55
How does gradient boosting differ from AdaBoost?
KnownRevise
Both AdaBoost and gradient boosting build an ensemble of weak learners sequentially, but they correct errors differently. AdaBoost reweights training examples, increasing the weight of misclassified points, and combines learners with accuracy-based votes; it effectively minimizes an exponential loss. Gradient boosting fits each new tree to the negative gradient of any differentiable loss (for squared error, simply the residuals), then adds it with a small learning rate. This generality, plus regularization options, makes gradient boosting (XGBoost, LightGBM, CatBoost, scikit-learn’s HistGradientBoosting) the stronger, more widely used method today.
Add the tree’s predictions times the learning rate.
Repeat; each tree corrects what is left.
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
# Manual gradient boosting for regression
rng = np.random.default_rng(42)
x = rng.uniform(0, 6, (300, 1))
y = np.sin(x).ravel() + rng.normal(0, 0.2, 300)
pred = np.full_like(y, y.mean())
for m in range(100):
residual = y - pred # negative gradient of squared error
tree = DecisionTreeRegressor(max_depth=2, random_state=42).fit(x, residual)
pred += 0.1 * tree.predict(x)
if m in (0, 9, 99):
print(f"after {m + 1:3d} trees: training MSE {np.mean((y - pred) ** 2):.4f}")
# Compare on noisy classification data
X, yc = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.15, random_state=42)
for name, model in [("AdaBoost (stumps)", AdaBoostClassifier(DecisionTreeClassifier(max_depth=1), n_estimators=300,
random_state=42)),
("Gradient boosting", GradientBoostingClassifier(n_estimators=300, learning_rate=0.05,
max_depth=3, subsample=0.8, random_state=42))]:
print(f"{name:18s} CV accuracy {cross_val_score(model, X, yc, cv=5).mean():.3f}")
Common Pitfalls
A large learning rate with many trees, which overfits; use small rates with early stopping.
Describing AdaBoost and gradient boosting as unrelated: AdaBoost is gradient boosting with exponential loss and a specific step size.
56
How would you handle missing data in a supervised learning problem?
KnownRevise
Handling missing data starts with understanding why values are missing: completely at random (MCAR), dependent on observed data (MAR), or dependent on the missing value itself (MNAR). Then choose a strategy: drop rows or columns only when the loss is small, impute values (mean, median, most frequent, k-NN or iterative model-based imputation), add missing-indicator features when missingness itself carries information, or use models that handle missing values natively, such as scikit-learn’s histogram gradient boosting and XGBoost. Imputation must be fitted on training data only, inside the pipeline.
SimpleImputer(strategy="most_frequent") or "constant"
Missing indicator
Missingness is informative (“no income reported”)
add_indicator=True, MissingIndicator
k-NN imputation
Similar rows have similar values
KNNImputer
Iterative / model-based
Features predict each other
IterativeImputer (experimental import)
Native support
Tree boosting
HistGradientBoosting*, XGBoost, LightGBM
Domain rules
Known meaning (e.g. missing = 0 purchases)
Explicit fill
Code Example
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.experimental import enable_iterative_imputer # noqa: F401 (enables IterativeImputer)
from sklearn.impute import IterativeImputer, KNNImputer, SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
rng = np.random.default_rng(42)
X_missing = X.copy()
mask = rng.random(X.shape) < 0.2 # 20% of values missing at random
X_missing[mask] = np.nan
models = {
"median imputer + ridge": make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), Ridge()),
"median + indicators + ridge": make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
StandardScaler(), Ridge()),
"KNN imputer + ridge": make_pipeline(StandardScaler(), KNNImputer(n_neighbors=5), Ridge()),
"iterative imputer + ridge": make_pipeline(IterativeImputer(random_state=42), StandardScaler(), Ridge()),
"native NaN (gradient boosting)": HistGradientBoostingRegressor(random_state=42),
}
print("no missing data, ridge:", round(-cross_val_score(make_pipeline(StandardScaler(), Ridge()), X, y, cv=5,
scoring="neg_mean_absolute_error").mean(), 1))
for name, model in models.items():
mae = -cross_val_score(model, X_missing, y, cv=5, scoring="neg_mean_absolute_error").mean()
print(f"{name:32s} CV MAE {mae:.1f}")
StandardScaler ignores NaNs when fitting and keeps them, so it can run before KNNImputer, which needs scaled distances.
Common Pitfalls
Imputing with statistics from the full dataset before splitting.
Dropping rows when missingness is related to the target, which biases the model.
Imputing the target variable; drop rows with missing labels instead.
Forgetting that production data may have missing values in new places; test the pipeline on them.
57
What steps prepare a dataset for a supervised learning model?
KnownRevise
Preparing a dataset turns raw records into a clean, leakage-free training table whose features will also be available at prediction time. The steps are: define the target and prediction moment, collect and join data, check data quality, split the data before fitting any transformation, clean and impute, encode categories, scale where needed, engineer features, handle imbalance, and wrap all fitted steps in a Pipeline so they are applied identically in training, validation and production.
One-hot, ordinal or target encoding for categories
8. Scale
Standardize for linear, distance-based and neural models
9. Engineer features
Ratios, aggregates, date parts, text vectors
10. Handle imbalance
Class weights, resampling inside CV
11. Package
Pipeline + ColumnTransformer; version the data and code
Code Example
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler
rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
"signup_date": pd.Timestamp("2025-01-01") + pd.to_timedelta(rng.integers(0, 365, n), unit="D"),
"plan": rng.choice(["basic", "plus", "pro", None], n, p=[0.5, 0.3, 0.15, 0.05]),
"monthly_spend": rng.lognormal(3, 0.8, n),
"support_tickets": rng.poisson(1.5, n).astype(float),
})
df.loc[rng.random(n) < 0.1, "monthly_spend"] = np.nan
df = pd.concat([df, df.sample(50, random_state=42)]) # duplicates to clean
y_all = ((df["support_tickets"] > 2) | (df["plan"] == "basic")).astype(int)
# Clean before splitting only what doesn't learn from data: drop exact duplicates, derive date parts
df = df.drop_duplicates()
y_all = y_all.loc[df.index]
df = df.assign(signup_month=df["signup_date"].dt.month).drop(columns="signup_date")
X_train, X_test, y_train, y_test = train_test_split(df, y_all, test_size=0.2, stratify=y_all, random_state=42)
numeric = ["monthly_spend", "support_tickets", "signup_month"]
categorical = ["plan"]
prep = ColumnTransformer([
("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
FunctionTransformer(np.log1p), StandardScaler()), numeric),
("cat", make_pipeline(SimpleImputer(strategy="constant", fill_value="missing"),
OneHotEncoder(handle_unknown="ignore")), categorical),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
print("CV ROC AUC:", round(cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean(), 3))
model.fit(X_train, y_train)
print("test ROC AUC:", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 3))
Common Pitfalls
Fitting imputers, scalers or encoders before the split.
Features created from information after the prediction moment (target leakage).
Duplicates of the same entity in train and test.
58
How would you handle text data in a supervised learning problem?
KnownRevise
Text must be converted into numeric features before most supervised models can use it. The classic route is to clean and tokenize the text, then represent each document as a sparse vector of word or character n-gram counts weighted by TF-IDF, and train a linear model (logistic regression or linear SVM). The modern route uses pretrained language models, either as frozen embedding extractors feeding a simple classifier or fine-tuned end to end. Text columns often sit alongside numeric and categorical features, which a ColumnTransformer combines.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
rng = np.random.default_rng(42)
pos = ["fast delivery and great quality", "love it, works perfectly", "excellent support, quick refund"]
neg = ["arrived broken, very disappointed", "terrible quality, stopped working", "slow support, no refund"]
n = 600
label = rng.integers(0, 2, n)
df = pd.DataFrame({
"review": [rng.choice(pos if l else neg) + rng.choice(["", " again", " overall", " honestly"]) for l in label],
"channel": rng.choice(["web", "app", "store"], n),
"order_value": rng.lognormal(3.5, 0.6, n),
})
prep = ColumnTransformer([
("text", TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=2), "review"), # a column name, not a list
("cat", OneHotEncoder(handle_unknown="ignore"), ["channel"]),
("num", StandardScaler(), ["order_value"]),
])
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000, C=5))])
print("CV accuracy:", round(cross_val_score(model, df, label, cv=5).mean(), 3))
model.fit(df, label)
vocab = model.named_steps["prep"].named_transformers_["text"].get_feature_names_out()
coefs = model.named_steps["clf"].coef_[0][:len(vocab)]
print("most positive terms:", list(vocab[np.argsort(coefs)[-5:]]))
The synthetic reviews are easy by construction; real text needs more data and an honest held-out test set.
Preprocessing Choices
Lowercasing and removing markup usually help; aggressive stop-word removal can delete meaning (“not”).
Keep the vectorizer inside the pipeline so its vocabulary and IDF weights come only from training folds.
Common Pitfalls
Passing a list (["review"]) instead of a string to the text vectorizer in ColumnTransformer, which feeds it a 2D array.
Leakage through duplicate texts across splits.
Very long documents truncated by transformer models without checking where the signal is.
59
When is dimensionality reduction useful, and how is it done?
KnownRevise
Dimensionality reduction reduces the number of input features while keeping as much useful information as possible. It is useful when there are many correlated or noisy features, when the number of features is large relative to samples, when models are slow, when distance-based methods suffer from the curse of dimensionality, and for visualizing data in 2D or 3D. It is done by feature selection (keeping a subset of original features) or feature extraction (creating new combined features, such as PCA components or learned embeddings).
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
When It Helps
Situation
Benefit
Many correlated features
Removes redundancy, stabilizes linear models
p close to or larger than n
Reduces overfitting
k-NN, clustering, SVM in high dimensions
Distances become more meaningful
Slow training or inference
Fewer features to process
Visualization
2D/3D views of structure
Noisy measurements
Low-variance components often contain noise
It may not help tree ensembles much, which already pick informative features, and it can hurt interpretability.
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True) # 64 pixel features
pca = PCA().fit(StandardScaler().fit_transform(X))
n95 = int((pca.explained_variance_ratio_.cumsum() < 0.95).sum()) + 1
print(f"components for 95% variance: {n95} of {X.shape[1]}")
configs = {
"k-NN, all 64 features": make_pipeline(StandardScaler(), KNeighborsClassifier()),
"k-NN, PCA(20)": make_pipeline(StandardScaler(), PCA(n_components=20, random_state=42), KNeighborsClassifier()),
"logreg, SelectKBest(30)": make_pipeline(StandardScaler(), SelectKBest(f_classif, k=30),
LogisticRegression(max_iter=2000)),
"logreg, PCA(0.95)": make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression(max_iter=2000)),
}
for name, model in configs.items():
print(f"{name:26s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
PCA(n_components=0.95) keeps as many components as needed to explain 95% of the variance; tune the number of components with cross-validation when accuracy matters.
Common Pitfalls
Fitting PCA or feature selection on all data before cross-validation.
Forgetting to scale before PCA, so large-scale features dominate the components.
Assuming high-variance directions are the most predictive; supervised selection may do better.
60
Which data augmentation techniques improve supervised learning models?
KnownRevise
Data augmentation creates new training examples by applying label-preserving transformations to existing ones, which increases data variety, reduces overfitting and makes models robust to variations they will meet in practice. Techniques depend on the data type: geometric and colour transforms for images, synonym replacement and back-translation for text, noise, time-shifting and masking for audio and time series, and SMOTE-style interpolation or noise injection for tabular data. Augmentation is applied only to training data and must not change the correct label.
Synonym replacement, random insertion/deletion/swap, back-translation, paraphrasing with a language model
Audio
Time shift, speed and pitch change, added noise, SpecAugment (time/frequency masking)
Time series
Jittering, scaling, window slicing, time warping, magnitude warping
Tabular
SMOTE for minority classes, Gaussian noise on numeric features, mixup
Code Example: Image and Tabular Augmentation
import numpy as np
import torch
from torchvision.transforms import v2
torch.manual_seed(42)
# Images: transforms applied fresh each time a training sample is loaded
train_transform = v2.Compose([
v2.RandomResizedCrop(size=224, scale=(0.7, 1.0), antialias=True),
v2.RandomHorizontalFlip(p=0.5),
v2.RandomRotation(degrees=10),
v2.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
v2.ToDtype(torch.float32, scale=True),
v2.RandomErasing(p=0.25),
])
test_transform = v2.Compose([v2.Resize(256, antialias=True), v2.CenterCrop(224),
v2.ToDtype(torch.float32, scale=True)]) # no randomness at evaluation
img = torch.randint(0, 256, (3, 256, 256), dtype=torch.uint8)
print("augmented:", train_transform(img).shape, "| test:", test_transform(img).shape)
# Mixup for a batch: blend pairs of examples and their one-hot labels
def mixup(x, y_onehot, alpha=0.2, rng=np.random.default_rng(42)):
lam = rng.beta(alpha, alpha)
perm = torch.randperm(x.size(0))
return lam * x + (1 - lam) * x[perm], lam * y_onehot + (1 - lam) * y_onehot[perm]
xb, yb = torch.randn(8, 3, 32, 32), torch.nn.functional.one_hot(torch.randint(0, 10, (8,)), 10).float()
xm, ym = mixup(xb, yb)
print("mixed label rows sum to 1:", torch.allclose(ym.sum(1), torch.ones(8)))
# Tabular: Gaussian noise on standardized numeric features (training rows only)
rng = np.random.default_rng(42)
X_train = rng.normal(size=(100, 5))
X_aug = np.vstack([X_train, X_train + rng.normal(0, 0.05, X_train.shape)])
print("tabular rows before/after:", len(X_train), len(X_aug))
Guidelines
Choose transformations that reflect real variation: flipping street signs or digits can change their meaning.
Tune augmentation strength like a hyperparameter.
Keep validation and test data unaugmented (test-time augmentation is a separate, deliberate technique).
For tabular data, augmentation helps less than for images or audio; class weights or more data often matter more.
Common Pitfalls
Augmenting before splitting, so variants of a test example appear in training.
Transformations that change the label (e.g. rotating a “6” into a “9”).
61
What role does transfer learning play in supervised models?
KnownRevise
Transfer learning reuses a model trained on a large source task (e.g. ImageNet images or web-scale text) as the starting point for a new supervised task with less labelled data. Early layers learn general features (edges, textures, word meanings) that transfer well, so you either use the pretrained network as a frozen feature extractor with a new classifier on top, or fine-tune some or all of its layers at a low learning rate. It dramatically reduces the data, compute and time needed and is the default approach for images, text and audio today.
Source and target domains are related (natural images → product photos).
Less helpful when domains differ greatly (natural images → spectrograms), though pretrained weights are still often better than random.
Code Example: Feature Extraction Then Fine-Tuning
import torch
from torch import nn
from torchvision import models
torch.manual_seed(42)
NUM_CLASSES = 5
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT) # downloads ImageNet weights
for p in model.parameters():
p.requires_grad = False # phase 1: freeze the backbone
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES) # new trainable head
head_opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
x, y = torch.randn(16, 3, 224, 224), torch.randint(0, NUM_CLASSES, (16,)) # stand-in batch
loss = nn.functional.cross_entropy(model(x), y)
loss.backward()
head_opt.step()
# Phase 2: unfreeze the last block and fine-tune with a lower learning rate for the backbone
for p in model.layer4.parameters():
p.requires_grad = True
optimizer = torch.optim.AdamW([
{"params": model.layer4.parameters(), "lr": 1e-4},
{"params": model.fc.parameters(), "lr": 1e-3},
])
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable parameters: {trainable:,} of {sum(p.numel() for p in model.parameters()):,}")
In real training, loop over a DataLoader with the pretrained model’s preprocessing (ResNet18_Weights.DEFAULT.transforms()), and validate after each epoch.
Common Pitfalls
Using different input preprocessing from the pretraining (resolution, normalization).
Fine-tuning everything with a high learning rate, which destroys pretrained features.
BatchNorm statistics drifting when fine-tuning with tiny batches; keep those layers in eval mode or freeze them.
62
What is end-to-end learning in deep learning?
KnownRevise
End-to-end learning trains a single model to map raw inputs directly to the final output, optimizing all components jointly with one loss, instead of chaining separately designed stages such as hand-crafted feature extraction followed by a classifier. Examples include speech recognition from audio waveforms or spectrograms straight to text, image classification from pixels, and translation from source to target sentences. It removes manual feature engineering and lets every layer adapt to the task, but it needs much more labelled data and compute, and it is harder to debug and interpret.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
Pipeline Versus End-to-End
Aspect
Traditional pipeline
End-to-end
Stages
Separate (features → model → post-processing)
One differentiable model
Feature design
Manual, domain expertise
Learned
Optimization
Each stage separately
Jointly for the final loss
Data needs
Lower
High (or pretrained models)
Debugging
Inspect each stage
Harder; intermediate representations are implicit
Example (speech)
MFCC features → acoustic model → pronunciation lexicon → language model
Audio → transformer → characters/tokens
When It Works Well
Large labelled datasets, or strong pretrained models to fine-tune.
Tasks where hand-crafted features lose information (vision, speech, language).
When intermediate labels are unavailable or unnecessary.
When a Pipeline Is Better
Little data, but good domain knowledge for features.
Requirements for interpretability or verifiable intermediate steps (e.g. safety-critical systems).
When intermediate outputs (detected objects, recognized entities) are useful in their own right.
Code Example: Learned Features vs Hand-Crafted Features
import numpy as np
import torch
from sklearn.linear_model import LogisticRegression
from torch import nn
torch.manual_seed(42)
rng = np.random.default_rng(42)
# Signals: class 1 contains a short burst at a random position
def make(n):
x = rng.normal(0, 1, (n, 128)).astype("float32")
y = rng.integers(0, 2, n)
for i in np.where(y == 1)[0]:
s = rng.integers(0, 112)
x[i, s:s + 16] += np.sin(np.linspace(0, 4 * np.pi, 16)) * 2
return x, y
X_tr, y_tr = make(2000)
X_te, y_te = make(500)
# Pipeline: hand-crafted features (mean, std, max) + classifier
feats = lambda X: np.c_[X.mean(1), X.std(1), X.max(1)]
pipe = LogisticRegression().fit(feats(X_tr), y_tr)
print("hand-crafted features accuracy:", round(pipe.score(feats(X_te), y_te), 3))
# End-to-end: a 1D CNN learns its own features from raw samples
model = nn.Sequential(nn.Conv1d(1, 16, 9, padding=4), nn.ReLU(), nn.AdaptiveMaxPool1d(1),
nn.Flatten(), nn.Linear(16, 1))
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
xt, yt = torch.tensor(X_tr)[:, None], torch.tensor(y_tr, dtype=torch.float32)
for epoch in range(200):
opt.zero_grad()
loss = nn.functional.binary_cross_entropy_with_logits(model(xt).squeeze(1), yt)
loss.backward()
opt.step()
with torch.no_grad():
acc = ((model(torch.tensor(X_te)[:, None]).squeeze(1) > 0).numpy() == y_te).mean()
print("end-to-end CNN accuracy:", round(float(acc), 3))
The CNN learns a filter that matches the burst shape, which the summary statistics capture only partly.
Common Pitfalls
Expecting end-to-end models to work with little data and no pretraining.
Losing useful structure (e.g. known physics or rules) that a hybrid approach could keep.
63
What is multitask learning, and what are its benefits?
KnownRevise
Multitask learning trains one model to perform several related tasks at once, usually with a shared representation (common layers) and task-specific heads, optimizing a combined loss. Learning tasks together lets each benefit from the others’ signal: shared features become more general, the model is regularized against overfitting to one task, and a single model is cheaper to serve than several. Examples include predicting both click and purchase in recommendations, or detecting objects and estimating depth from the same image.
Read full answerHide full answer
Applies toPyTorch 2.x, NumPy 2.x (Python 3.12)
Benefits and Risks
Benefits
Risks
Better generalization through shared representations
Negative transfer when tasks conflict
Data efficiency: a data-rich task helps a data-poor one
Loss balancing is tricky (tasks on different scales)
Regularization: harder to overfit any single task
Gradients from tasks can point in opposite directions
One model to train, deploy and run
More complex evaluation and debugging
Architectures
Type
Description
Hard parameter sharing
Shared backbone + separate heads (most common)
Soft parameter sharing
Separate networks with constraints that keep weights similar
Mixture-of-experts (e.g. MMoE)
Shared experts with task-specific gating (recommender systems)
Loss Weighting
Total loss = sum of w_t × loss_t. Weights can be fixed and tuned, normalized by each loss’s scale, or learned (e.g. uncertainty weighting). Gradient-surgery methods such as PCGrad reduce conflicts between tasks.
Compare against single-task models trained the same way to check that sharing actually helps.
Common Pitfalls
Summing losses with very different scales so one task dominates.
Grouping unrelated tasks, which causes negative transfer.
64
How can reinforcement learning be framed as a supervised learning problem?
KnownRevise
Reinforcement learning (RL) can be partly recast as supervised learning whenever you can create (input, target) pairs from experience. The main framings are: behaviour cloning, which learns to predict an expert’s action from the state (classification or regression); fitted value iteration, where a regressor learns Q-values from bootstrapped targets r + γ max Q(s', a') computed on logged transitions (fitted Q-iteration, and DQN’s update is a regression step); reward-weighted or return-conditioned learning (e.g. Decision Transformer), which predicts actions conditioned on desired returns; and learning environment models (predict next state and reward). The supervised view simplifies training but must still handle distribution shift, because the agent’s own actions change the data it sees.
Compounding errors in unseen states (DAgger helps)
Fitted Q-iteration / DQN updates
(State, action) → r + γ max Q(next)
Targets move; needs stable training tricks
Return-conditioned policies
(State, desired return) → action
Needs diverse offline data
Model learning
(State, action) → next state, reward
Model errors compound in planning
Reward modelling (RLHF)
Pair of outputs → which is preferred
Reward hacking if the model is wrong
Code Example: Fitted Q-Iteration on a Small Chain
States 0-9 on a line; actions left/right; reaching state 9 gives reward 1. A regressor learns Q-values from a fixed batch of random transitions.
import numpy as np
from sklearn.ensemble import ExtraTreesRegressor
rng = np.random.default_rng(42)
N, GAMMA = 10, 0.9
# Logged random transitions (offline data)
s = rng.integers(0, N - 1, 5000)
a = rng.integers(0, 2, 5000) # 0 = left, 1 = right
s_next = np.clip(s + np.where(a == 1, 1, -1), 0, N - 1)
r = (s_next == N - 1).astype(float)
done = s_next == N - 1
X = np.c_[s, a]
q_model = None
for iteration in range(30):
if q_model is None:
target = r # first iteration: immediate reward
else:
q_next = np.max([q_model.predict(np.c_[s_next, np.full_like(s_next, act)]) for act in (0, 1)], axis=0)
target = r + GAMMA * q_next * (~done) # supervised target from the Bellman equation
q_model = ExtraTreesRegressor(n_estimators=50, random_state=42).fit(X, target)
policy = [int(np.argmax([q_model.predict([[state, act]])[0] for act in (0, 1)])) for state in range(N - 1)]
print("greedy action per state (1 = right):", policy)
print("Q(0, right) ≈", round(q_model.predict([[0, 1]])[0], 3), "| theory:", round(GAMMA ** 8, 3))
Each iteration is an ordinary regression problem; the learned greedy policy should move right everywhere, and Q(0, right) should approach γ⁸ because the reward arrives after nine steps.
Common Pitfalls
Treating behaviour cloning as solved by high validation accuracy; evaluate with rollouts.
Offline data that never visits important states, which no regressor can fix.
65
What role do attention mechanisms play in neural networks?
KnownRevise
Attention mechanisms let a neural network decide, for each position or output, which parts of the input to focus on, by computing weights over all inputs and taking a weighted combination of them. Introduced for machine translation to overcome the fixed-length bottleneck of RNN encoders, attention became the core of the Transformer, where self-attention relates every token to every other token in parallel. It captures long-range dependencies, handles variable-length inputs, and powers modern language models, vision transformers and multimodal models; attention weights also give some insight into what the model uses.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
How Scaled Dot-Product Attention Works
Each input is projected into a query, a key and a value vector.
Score each query against every key with a dot product, divided by the square root of the key dimension.
Apply softmax so the scores become weights that sum to 1.
Output = weighted sum of the values.
Multi-head attention runs several of these in parallel with different projections and concatenates the results.
Types
Type
Queries from
Keys/values from
Use
Self-attention
Same sequence
Same sequence
Transformer encoders/decoders
Cross-attention
Decoder
Encoder output
Translation, image captioning
Causal (masked) self-attention
Sequence
Earlier positions only
Language model generation
Additive (Bahdanau)
Decoder state
Encoder states
Early seq2seq models
Code Example
import torch
import torch.nn.functional as F
from torch import nn
torch.manual_seed(42)
batch, seq_len, d_model = 2, 6, 16
x = torch.randn(batch, seq_len, d_model)
# Manual scaled dot-product self-attention
Wq, Wk, Wv = (nn.Linear(d_model, d_model, bias=False) for _ in range(3))
q, k, v = Wq(x), Wk(x), Wv(x)
scores = q @ k.transpose(-2, -1) / d_model ** 0.5 # (batch, seq, seq)
causal = torch.triu(torch.ones(seq_len, seq_len, dtype=torch.bool), diagonal=1)
weights = scores.masked_fill(causal, float("-inf")).softmax(dim=-1)
manual = weights @ v
print("first row of weights (only attends to itself):", weights[0, 0].detach().numpy().round(2))
# PyTorch's fused implementation gives the same result
fused = F.scaled_dot_product_attention(q, k, v, is_causal=True)
print("matches fused version:", torch.allclose(manual, fused, atol=1e-5))
# Multi-head attention module
mha = nn.MultiheadAttention(embed_dim=d_model, num_heads=4, batch_first=True)
out, attn = mha(x, x, x, need_weights=True)
print("output:", tuple(out.shape), "| averaged attention weights:", tuple(attn.shape))
Why It Matters
Direct connections between any two positions: no information has to pass through many recurrent steps.
Parallel computation across positions, unlike RNNs.
Cost grows with the square of sequence length, which motivates efficient variants (FlashAttention, sparse and linear attention).
Common Pitfalls
Forgetting padding or causal masks.
Reading attention weights as full explanations; they show where information flows, not necessarily why a prediction was made.
66
How would you design a supervised model to predict customer churn?
KnownRevise
A churn model predicts which customers are likely to stop using a product within a defined window, so the business can intervene. A good design starts with a precise churn definition and prediction time, builds features from behaviour up to that time (usage trends, payments, support contacts, tenure, plan), trains a classifier (logistic regression and gradient boosting are strong choices), validates on a later time period, and turns predicted probabilities into actions by targeting the customers where retention offers have the highest expected value.
Leakage from features recorded after the cut-off (e.g. “cancellation reason”).
Random splits across snapshots of the same customer.
Predicting churn but not whether an intervention changes it; uplift modelling targets persuadable customers.
67
What is your strategy for building a supervised sentiment analysis model?
KnownRevise
A supervised sentiment model is built by defining the label scheme (positive/negative, or with neutral, or a 1-5 rating), collecting and annotating representative text from the target domain, establishing a fast baseline (TF-IDF with logistic regression), then fine-tuning a pretrained transformer if the baseline is not good enough. Evaluation uses macro F1 on a held-out set from the same domain, with error analysis on negation, sarcasm and mixed opinions, followed by monitoring after deployment because language and topics drift.
Read full answerHide full answer
Applies toscikit-learn 1.5+, Hugging Face Transformers 4.4x (optional) (Python 3.12)
Strategy
Step
Details
1. Labels
Clear guidelines with examples; decide how to treat mixed or neutral text
2. Data
Domain-matched texts; double-annotate a sample to measure agreement (Cohen’s kappa)
3. Split
Stratified; de-duplicate; split by user or product if they repeat
Inspect misclassified examples; fix labels or add data
8. Deployment
Batch or API; monitor class distribution and sampled accuracy
Code Example: Baseline with Error Analysis
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import FeatureUnion, make_pipeline
rng = np.random.default_rng(42)
pos = ["great", "excellent", "love", "fantastic", "works well"]
neg = ["terrible", "awful", "hate", "broken", "waste of money"]
templates = ["the product is {}", "{} experience overall", "honestly {}", "really {} purchase"]
texts, labels = [], []
for _ in range(1500):
label = rng.integers(0, 2)
word = rng.choice(pos if label else neg)
text = rng.choice(templates).format(word)
if rng.random() < 0.15: # negation flips the sentiment
text, label = "not " + text, 1 - label
texts.append(text)
labels.append(label)
X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.25, stratify=labels, random_state=42)
features = FeatureUnion([("words", TfidfVectorizer(ngram_range=(1, 2))),
("chars", TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)))])
model = make_pipeline(features, LogisticRegression(max_iter=2000, C=5)).fit(X_train, y_train)
pred = model.predict(X_test)
print(classification_report(y_test, pred, target_names=["negative", "positive"], digits=3))
errors = [t for t, p, y in zip(X_test, pred, y_test) if p != y]
print("examples of errors:", errors[:5])
print("share of errors containing 'not':", round(np.mean(["not" in e for e in errors]), 2) if errors else 0)
The synthetic data shows the typical weak spot: negated phrases. Bigrams such as “not great” help a linear model; transformers handle negation better.
Common Pitfalls
Training on product reviews and deploying on tweets or support chats.
Removing stop words like “not” and “no”.
Treating neutral or mixed text as noise instead of defining it in the label scheme.
68
What supervised learning approach would you propose for transaction fraud detection?
KnownRevise
Transaction fraud detection can be framed as supervised binary classification: predict, at authorization time, the probability that a transaction is fraudulent, using labels from confirmed chargebacks and investigations. The proposal: rich behavioural features (amount relative to the customer’s history, velocity counts, merchant and device risk, geography), a gradient-boosted tree model with class weighting, strict time-based validation, and a decision threshold chosen from the cost of missed fraud versus declined good customers. Because fraud patterns change and labels arrive late, combine it with rules and anomaly detection, and retrain frequently.
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score
rng = np.random.default_rng(42)
n = 60000
df = pd.DataFrame({
"week": np.sort(rng.integers(0, 12, n)),
"amount": rng.lognormal(3.5, 1.0, n),
"amount_vs_avg": rng.lognormal(0, 0.5, n),
"tx_last_hour": rng.poisson(0.3, n),
"new_device": rng.binomial(1, 0.05, n),
"foreign": rng.binomial(1, 0.08, n),
})
logit = (-7 + 1.2 * np.log(df["amount_vs_avg"] + 0.1) + 0.8 * df["tx_last_hour"]
+ 2.0 * df["new_device"] + 1.2 * df["foreign"])
df["fraud"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
features = ["amount", "amount_vs_avg", "tx_last_hour", "new_device", "foreign"]
train, test = df[df["week"] < 9], df[df["week"] >= 9] # time-based split
model = HistGradientBoostingClassifier(class_weight="balanced", max_depth=4, random_state=42)
model.fit(train[features], train["fraud"])
p = model.predict_proba(test[features])[:, 1]
print("fraud rate:", round(test["fraud"].mean(), 4), "| PR AUC:", round(average_precision_score(test["fraud"], p), 3))
# Choose the threshold that minimizes expected cost on the test weeks (use a validation period in practice)
fraud_amount = test["amount"].to_numpy()
best = min(
((t, (fraud_amount * ((p < t) & (test["fraud"] == 1))).sum() + 5.0 * ((p >= t) & (test["fraud"] == 0)).sum())
for t in np.linspace(0.05, 0.95, 19)),
key=lambda pair: pair[1])
print(f"threshold {best[0]:.2f} with expected cost {best[1]:,.0f}")
The cost uses the transaction amount for missed fraud and a fixed 5.0 friction cost for each wrongly declined good transaction.
Common Pitfalls
Random splits that leak future fraud patterns into training.
Features computed with information from after the transaction (e.g. later chargeback counts).
Optimizing ROC AUC alone when fraud is extremely rare.
69
How would you build a predictive maintenance system with supervised learning?
KnownRevise
A predictive maintenance system uses sensor and operational data to predict equipment failures early enough to schedule maintenance before breakdowns. With supervised learning, you label historical data with failure events and frame either classification (“will this machine fail in the next N days?”) or regression (“remaining useful life, RUL”). The design centres on feature engineering over time windows (rolling statistics, trends), labelling relative to failure times, splitting by machine and time, and choosing alert thresholds that balance missed failures against unnecessary maintenance.
Sensor time series (vibration, temperature, pressure), operating settings, maintenance logs, failure records
Label
1 if a failure occurs within the next N cycles/days (classification) or cycles remaining (RUL regression)
Features
Rolling mean, std, min/max, slopes over several windows; time since last maintenance; usage counters
Split
Hold out whole machines, and later time periods
Model
Gradient boosting or random forest; sequence models (1D CNN/LSTM) with lots of data
Metrics
Recall of failures at an acceptable false-alarm rate; lead time; for RUL, MAE weighted toward late predictions
Action
Maintenance work orders when risk exceeds the threshold
Code Example
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, recall_score
from sklearn.model_selection import GroupShuffleSplit
rng = np.random.default_rng(42)
frames = []
for machine in range(120):
life = rng.integers(150, 300) # cycles until failure
t = np.arange(life)
wear = (t / life) ** 3 # degradation accelerates near the end
frames.append(pd.DataFrame({
"machine": machine, "cycle": t,
"vibration": 1 + 2 * wear + rng.normal(0, 0.15, life),
"temperature": 60 + 15 * wear + rng.normal(0, 1.0, life),
"fail_within_20": (life - t <= 20).astype(int),
}))
df = pd.concat(frames, ignore_index=True)
# Rolling features per machine (past values only)
g = df.groupby("machine")
for col in ("vibration", "temperature"):
df[f"{col}_mean10"] = g[col].transform(lambda s: s.rolling(10, min_periods=1).mean())
df[f"{col}_slope10"] = g[col].transform(lambda s: s.diff(10) / 10)
df = df.dropna()
features = [c for c in df.columns if c.endswith(("mean10", "slope10"))] + ["vibration", "temperature"]
split = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42)
train_idx, test_idx = next(split.split(df, groups=df["machine"])) # unseen machines in test
train, test = df.iloc[train_idx], df.iloc[test_idx]
model = HistGradientBoostingClassifier(class_weight="balanced", random_state=42)
model.fit(train[features], train["fail_within_20"])
p = model.predict_proba(test[features])[:, 1]
print("PR AUC:", round(average_precision_score(test["fail_within_20"], p), 3))
print("recall of pre-failure cycles at threshold 0.5:", round(recall_score(test["fail_within_20"], (p >= 0.5).astype(int)), 3))
# Lead time: first alert before failure for each test machine
test = test.assign(alert=p >= 0.5)
lead = test[test["alert"]].groupby("machine")["cycle"].min()
life_end = test.groupby("machine")["cycle"].max()
print("median warning (cycles before failure):", float((life_end.loc[lead.index] - lead).median()))
Common Pitfalls
Random row splits, so the same machine’s neighbouring cycles appear in train and test.
Rolling features that include future readings (centred windows).
Too few failures: consider anomaly detection or survival models, and collect more labelled failure data.
70
How would you build a model to automate medical image diagnosis?
KnownRevise
Building a medical image diagnosis model starts with a precise clinical task (e.g. detect pneumonia on chest X-rays, or grade diabetic retinopathy), a dataset with reliable reference-standard labels, and patient-level splits with an external test set from other hospitals. The usual model is a pretrained CNN or vision transformer fine-tuned on the images, with augmentation suited to the modality. Evaluation focuses on sensitivity, specificity, ROC AUC and calibration, subgroup performance and failure analysis; deployment is as decision support with human review, monitoring and, for clinical use, regulatory clearance.
Saliency maps (e.g. Grad-CAM) for review, not as proof
Deployment
PACS integration, human-in-the-loop, drift monitoring, regulatory process
Code Example: Fine-Tuning Skeleton
import torch
from torch import nn
from torchvision import models
from torchvision.transforms import v2
torch.manual_seed(42)
train_tf = v2.Compose([
v2.RandomResizedCrop(224, scale=(0.85, 1.0), antialias=True),
v2.RandomRotation(7), # small rotations only; no flips for chest X-rays
v2.ColorJitter(brightness=0.1, contrast=0.1),
v2.ToDtype(torch.float32, scale=True),
v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
model = models.densenet121(weights=models.DenseNet121_Weights.DEFAULT) # downloads ImageNet weights
model.classifier = nn.Linear(model.classifier.in_features, 1) # binary finding
pos_weight = torch.tensor([4.0]) # e.g. negatives/positives ratio
loss_fn = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=1e-4)
# One training step on a stand-in batch (grayscale X-rays repeated to 3 channels)
images = torch.randint(0, 256, (8, 3, 256, 256), dtype=torch.uint8)
labels = torch.tensor([0, 1, 0, 0, 1, 0, 0, 0], dtype=torch.float32)
batch = torch.stack([train_tf(img) for img in images])
model.train()
loss = loss_fn(model(batch).squeeze(1), labels)
loss.backward()
optimizer.step()
print("loss:", round(loss.item(), 4))
In a real project, data loaders read DICOM files, patients are split with GroupShuffleSplit, and the threshold is chosen on validation data to reach the required sensitivity.
Common Pitfalls
Image-level instead of patient-level splits.
Shortcut learning from text markers, devices or hospital-specific artefacts.
Labels mined from reports without checking their accuracy.