Skip to content
Machine Learning & Data Science

Optimization interview questions and answers

50 questions, 50 with code. About 2 h 59 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is optimization in the context of machine learning?

In machine learning, optimization is the process of finding the model parameters θ that minimize an objective function, usually the average loss over the training data plus a regularization term, J(θ) = (1/n) Σ L(f(xᵢ; θ), yᵢ) + λR(θ); training a linear regression, a logistic regression, a gradient-boosted model or a neural network is, at its core, solving such a minimization problem. The problem can be convex (linear and logistic regression, SVMs), where any local minimum is the global one and methods converge reliably, or non-convex (neural networks), where we settle for good local minima and must deal with saddle points and flat regions. Solvers range from closed-form solutions (the normal equations) and second-order methods (Newton, L-BFGS) for small smooth problems, to first-order stochastic methods (SGD, momentum, Adam) for large data, and constrained methods (Lagrange multipliers, KKT conditions) for problems like SVMs. A second layer of optimization tunes hyperparameters (learning rate, regularization strength, tree depth) using validation performance, with grid, random or Bayesian search. The goal isn’t just a low training loss: it is good performance on new data, which is why regularization, early stopping and validation are part of the process.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+

Ingredients

Part Example
Parameters θ Weights and biases
Loss L Squared error, cross-entropy, hinge
Regularizer R L2 (‖θ‖²), L1 (‖θ‖₁)
Constraints θ ≥ 0, margin constraints in SVMs
Algorithm Gradient descent, Adam, L-BFGS, coordinate descent
Hyperparameters Learning rate, λ, batch size
Evaluation Validation loss/metric

Code Example: The Same Problem Solved Three Ways

import numpy as np
from scipy.optimize import minimize

rng = np.random.default_rng(42)
X = np.column_stack([np.ones(200), rng.normal(size=(200, 2))])
true_w = np.array([1.0, 2.0, -3.0])
y = X @ true_w + rng.normal(0, 0.5, 200)
lam = 0.1

def objective(w):
    return np.mean((X @ w - y) ** 2) + lam * np.sum(w[1:] ** 2)    # loss + L2 penalty (no penalty on bias)

def gradient(w):
    g = 2 * X.T @ (X @ w - y) / len(y)
    g[1:] += 2 * lam * w[1:]
    return g

penalty = np.diag([0.0, lam, lam]) * len(y)
w_closed = np.linalg.solve(X.T @ X + penalty, X.T @ y)          # closed form (normal equations)

w_gd = np.zeros(3)
for _ in range(500):                                             # gradient descent
    w_gd -= 0.1 * gradient(w_gd)

result = minimize(objective, np.zeros(3), jac=gradient, method="L-BFGS-B")   # quasi-Newton
print(w_closed.round(4), w_gd.round(4), result.x.round(4), result.nit)
print(np.allclose(w_closed, w_gd, atol=1e-4), np.allclose(w_closed, result.x, atol=1e-4))
# True True: all three reach the same minimum of this convex objective

The closed form multiplies the penalty by n because the objective uses the mean squared error; with that scaling, all three methods agree to four decimals.

Common Pitfalls

  • Treating a low training loss as success without validation.

  • Using a method that doesn’t fit the problem size (Newton on millions of parameters).

  • Forgetting that features need scaling for gradient methods to converge well.

Interview Tips

  • Define the objective as loss plus regularization over parameters.

  • Mention convex vs non-convex problems, solver families and hyperparameter optimization.

2

What is the difference between a loss function and an objective function?

A loss function measures the error of the model on a single example, L(ŷ, y), such as the squared error (ŷ − y)² or the cross-entropy −y log ŷ − (1 − y) log(1 − ŷ); the objective function is the complete quantity the optimizer minimizes (or maximizes), usually the average loss over the training set plus any regularization and penalty terms, J(θ) = (1/n) Σ L(f(xᵢ; θ), yᵢ) + λR(θ). The terms overlap in everyday use, and “cost function” usually means the averaged loss. The distinction matters because the objective encodes choices beyond fitting the data: regularization strength, class weights, constraints converted into penalties, or the evidence lower bound in variational models, and in maximization problems (likelihood, reward) the objective is maximized while losses are minimized. The metric used to judge a model (accuracy, F1, AUC) is a third, separate thing, often not differentiable, which is why we optimize a smooth surrogate loss such as cross-entropy and evaluate with the metric.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Terms

Term Scope Example
Loss One example (ŷ − y)², log loss, hinge
Cost / empirical risk Average over the data MSE, mean log loss
Objective What the optimizer minimizes Cost + λ‖w‖² (+ constraints)
Metric What you report Accuracy, F1, AUC, RMSE

Code Example

import numpy as np

y = np.array([1, 0, 1, 1])
p = np.array([0.9, 0.2, 0.6, 0.4])                 # predicted P(y=1)
w = np.array([0.5, -1.5])                           # model weights
lam = 0.1

per_example_loss = -(y * np.log(p) + (1 - y) * np.log(1 - p))
cost = per_example_loss.mean()
objective = cost + lam * np.sum(w ** 2)
accuracy = np.mean((p >= 0.5) == y)

print(per_example_loss.round(4))   # [0.1054 0.2231 0.5108 0.9163]
print(round(float(cost), 4), round(float(objective), 4), accuracy)
# 0.4389 0.6889 0.75

The objective adds 0.1 · (0.25 + 2.25) = 0.25 to the cost of 0.4389. Accuracy is 3 of 4 because the last example (p = 0.4 for a positive) is misclassified, while its loss of 0.92 tells the optimizer by how much.

Common Pitfalls

  • Optimizing one objective and reporting a different metric without checking that they agree.

  • Forgetting that regularization is part of the objective but not of the reported loss.

  • Comparing losses computed with and without averaging or with different reductions.

Interview Tips

  • Define loss per example and objective as the full minimized quantity.

  • Mention surrogate losses vs evaluation metrics.

3

What is the role of gradients in optimization?

The gradient ∇J(θ) is the vector of partial derivatives of the objective with respect to each parameter; it points in the direction in which the objective increases fastest, and its length says how steep the surface is, so moving the parameters a small step in the opposite direction, θ ← θ − η∇J(θ), is the locally best way to decrease the objective, which is the basis of gradient descent and all its variants (SGD, momentum, Adam). At a minimum (or any stationary point) the gradient is zero, which gives optimality conditions and stopping criteria; its magnitude also signals problems such as vanishing gradients (training stalls) or exploding gradients (training diverges). For large models, gradients are computed efficiently by backpropagation (reverse-mode automatic differentiation) at roughly the cost of one forward pass, and in practice estimated on minibatches, which makes them noisy but cheap. Second-order methods add curvature (the Hessian) to choose better step sizes and directions, while gradient-free methods (grid search, evolutionary algorithms, Bayesian optimization) are used when gradients aren’t available, such as for hyperparameters.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

What the Gradient Tells You

Property Meaning
Direction Steepest ascent; step the other way to descend
Magnitude Steepness; tiny = flat region or near optimum
Zero gradient Stationary point (minimum, maximum or saddle)
Per-parameter entries How sensitive the objective is to each parameter
Noisy minibatch estimate Cheap, unbiased, adds exploration

Code Example: Analytic vs Numerical Gradient, and Descent

import numpy as np

def f(w):
    return (w[0] - 3) ** 2 + 10 * (w[1] + 1) ** 2

def grad_f(w):
    return np.array([2 * (w[0] - 3), 20 * (w[1] + 1)])

def numerical_grad(func, w, h=1e-6):
    g = np.zeros_like(w)
    for i in range(len(w)):
        e = np.zeros_like(w)
        e[i] = h
        g[i] = (func(w + e) - func(w - e)) / (2 * h)       # central difference
    return g

w = np.array([0.0, 0.0])
print(grad_f(w), numerical_grad(f, w).round(6))
# [-6. 20.] [-6. 20.]

for step in range(300):
    w = w - 0.04 * grad_f(w)
print(w.round(4), round(float(f(w)), 8))
# [ 3. -1.] 0.0   the minimum is at (3, -1)

With step size 0.04, the first coordinate’s error shrinks by a factor 1 − 0.04·2 = 0.92 per step and the second by 1 − 0.04·20 = 0.2, so the steep direction converges much faster; a step above 0.1 would make the second coordinate overshoot and diverge (|1 − 20η| > 1).

Common Pitfalls

  • Bugs in hand-written gradients; check them with finite differences.

  • Step sizes chosen without regard to curvature, causing divergence in steep directions.

  • Assuming a zero gradient means a minimum (it can be a saddle point).

Interview Tips

  • Explain direction of steepest ascent and the update rule.

  • Mention backpropagation, stationary points and gradient checks.

4

Why is convexity important in optimization problems?

A function is convex if the straight line between any two points on its graph never lies below the graph, f(tx + (1 − t)y) ≤ t f(x) + (1 − t) f(y), and a convex optimization problem minimizes a convex function over a convex set; this matters because every local minimum of a convex problem is a global minimum, the set of minimizers is convex, and a zero gradient (or the KKT conditions with constraints) guarantees optimality. As a result, convex problems can be solved reliably and efficiently, with convergence guarantees and rates for gradient descent, Newton’s method and interior-point methods, results that don’t depend on initialization, and tools such as duality that certify how close a solution is to optimal. Many classical models are convex, including linear and ridge regression, Lasso, logistic regression and SVMs, which is part of why they are dependable; deep neural networks are non-convex, so their training relies on heuristics like good initialization, momentum and learning-rate schedules, and results can depend on random seeds. Strong convexity (curvature bounded below) gives even faster, linear convergence.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+

Convex vs Non-Convex

Convex Non-convex
Local minima All are global Many, with different values
Initialization Doesn’t affect the optimum Matters
Guarantees Convergence and rates Usually only to stationary points
Optimality check Gradient = 0 / KKT Necessary but not sufficient
Examples Linear/logistic regression, Lasso, SVM Neural networks, matrix factorization, k-means

Code Example: Starting Point Matters Only for Non-Convex Functions

import numpy as np
from scipy.optimize import minimize

def convex(x):
    return (x[0] - 2) ** 2 + 0.5 * x[0] ** 4

def non_convex(x):
    return np.sin(3 * x[0]) + 0.1 * x[0] ** 2

for name, f in [("convex", convex), ("non-convex", non_convex)]:
    minima = sorted({round(float(minimize(f, [start]).x[0]), 3) for start in np.linspace(-4, 4, 9)})
    print(f"{name:10s}: distinct minimizers found from 9 starts: {minima}")
# convex    : one value, [1.0]
# non-convex: several different local minima

def is_convex_on_samples(f, low=-4, high=4, trials=10_000, seed=42):
    """Check the chord condition on random pairs (a test, not a proof)."""
    rng = np.random.default_rng(seed)
    a, b, t = rng.uniform(low, high, trials), rng.uniform(low, high, trials), rng.random(trials)
    lhs = np.array([f([v]) for v in t * a + (1 - t) * b])
    rhs = t * np.array([f([v]) for v in a]) + (1 - t) * np.array([f([v]) for v in b])
    return bool(np.all(lhs <= rhs + 1e-9))

print(is_convex_on_samples(convex), is_convex_on_samples(non_convex))
# True False

For the convex function, setting the derivative 2(x − 2) + 2x³ to zero gives x³ + x − 2 = 0, whose only real root is x = 1 (x³ + x is strictly increasing), so every start ends at the same point.

Common Pitfalls

  • Assuming a model is convex because its loss is (a convex loss of a neural network’s output is non-convex in the weights).

  • Treating non-convex results as unique or reproducible without fixing seeds.

  • Breaking convexity by adding non-convex penalties without noticing.

Interview Tips

  • Define convexity and the “local is global” property.

  • List convex ML models and explain how deep learning copes with non-convexity.

5

What is the difference between local minima and global minima?

A global minimum is the point where the objective takes its lowest value over the entire domain; a local minimum is a point that is lower than every nearby point but not necessarily the lowest overall, like the bottom of a small valley in a mountain range when a deeper valley exists elsewhere. Both have zero gradient and non-negative curvature (a positive semi-definite Hessian), so gradient methods can’t tell them apart locally and stop at whichever one their starting point and step sizes lead to. In convex problems every local minimum is global; in non-convex ones (neural networks, k-means, matrix factorization) many local minima exist. For large neural networks, research suggests most local minima have losses close to the global one and that saddle points and flat regions are the bigger obstacle, while “flat” minima tend to generalize better than sharp ones. Strategies for avoiding poor local minima include multiple random restarts, good initialization, stochastic gradients and momentum, learning-rate schedules (warm restarts), and global methods such as simulated annealing or evolutionary search for small problems.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+

Comparison

Local minimum Global minimum
Definition Lowest in a neighbourhood Lowest overall
Gradient Zero Zero
How many Possibly many One value (possibly several points)
Found by gradient descent Depends on the start Guaranteed only for convex problems

Code Example: Restarts Find the Global Minimum

import numpy as np
from scipy.optimize import basinhopping, minimize

def f(x):
    x = x[0]
    return x ** 4 - 4 * x ** 2 + x                 # two valleys of different depth

local_results = [minimize(f, [start]) for start in (-3.0, 3.0)]
for start, res in zip((-3.0, 3.0), local_results):
    print(f"start {start:+.0f}: x = {res.x[0]:+.3f}, f = {res.fun:.3f}")
# start -3: x = -1.473, f = -5.444   (global minimum)
# start +3: x = +1.347, f = -2.619   (local minimum)

rng = np.random.default_rng(42)
best = min((minimize(f, [s]) for s in rng.uniform(-3, 3, 10)), key=lambda r: r.fun)
print(f"best of 10 restarts: x = {best.x[0]:+.3f}")
# best of 10 restarts: x = -1.473

hop = basinhopping(f, [3.0], niter=50, seed=42)    # global search that hops between basins
print(f"basin hopping from +3: x = {hop.x[0]:+.3f}")
# basin hopping from +3: x = -1.473

The stationary points solve 4x³ − 8x + 1 = 0; the two minima are near x ≈ −1.473 and x ≈ 1.347, and the extra +x term makes the left valley deeper.

Common Pitfalls

  • Concluding a model can’t do better after one run from one seed.

  • Confusing saddle points (zero gradient, some downhill directions) with minima.

  • Spending effort on global search for convex problems, where it isn’t needed.

Interview Tips

  • Define both and explain why gradient descent can stop at a local minimum.

  • Mention convexity, restarts and the modern view of neural-network minima.

6

What is a hyperparameter and how does it relate to optimization?

A hyperparameter is a setting chosen before or around training that the training algorithm does not learn from the data by itself: the learning rate, batch size, number of epochs and optimizer type, the regularization strength λ, the depth of a tree or number of trees, the number of layers and units, the dropout rate, or k in k-nearest neighbours. Model parameters (weights, coefficients, split thresholds) are learned by minimizing the training objective; hyperparameters define that objective (λ), the model’s capacity (depth, width) and how the optimizer moves (learning rate, momentum), so they decide both how well the optimization converges and how well the result generalizes. That creates a two-level problem: an inner optimization fits parameters for given hyperparameters, and an outer optimization chooses hyperparameters by validation performance, using grid search, random search, Bayesian optimization (Optuna, scikit-optimize), successive halving or Hyperband, with cross-validation to avoid overfitting the validation set and a final untouched test set for reporting.

Read full answer

Applies toscikit-learn 1.5+, Optuna 3.x/4.x

Parameters vs Hyperparameters

Parameters Hyperparameters
Examples Weights, biases, split points Learning rate, λ, depth, k
Set by Training (inner optimization) You or a search (outer optimization)
Objective Training loss Validation score
Typical method Gradient descent, closed form Grid/random/Bayesian search

Code Example: Inner and Outer Optimization

import optuna
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

def objective(trial):
    C = trial.suggest_float("C", 1e-3, 1e2, log=True)              # hyperparameter: 1 / regularization strength
    model = make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=5000))
    return cross_val_score(model, X_train, y_train, cv=5).mean()   # inner fit happens inside each fold

optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=30)
print(f"best C = {study.best_params['C']:.4f}, CV accuracy = {study.best_value:.3f}")

final = make_pipeline(StandardScaler(), LogisticRegression(C=study.best_params["C"], max_iter=5000))
final.fit(X_train, y_train)
print(f"test accuracy = {final.score(X_test, y_test):.3f}")
print("learned parameters (first 3 coefficients):", final[-1].coef_[0][:3].round(3))

Common Pitfalls

  • Tuning hyperparameters on the test set.

  • Searching learning rates or λ on a linear scale instead of a log scale.

  • Tuning too many hyperparameters with too little data (overfitting the validation set).

Interview Tips

  • Contrast learned parameters with chosen hyperparameters.

  • Describe the inner/outer optimization and search methods.

7

Explain the concept of a learning rate

The learning rate η is the step size of gradient-based optimization: each update moves the parameters by η times the gradient, θ ← θ − η∇J(θ), so it controls how far the optimizer travels in the downhill direction at every step. Too small a learning rate makes training slow and can leave it stuck on plateaus; too large a rate overshoots the minimum, so the loss oscillates or diverges (for a quadratic with curvature L, plain gradient descent diverges when η > 2/L); a well-chosen rate converges quickly and stably. Because the best value changes during training, practitioners use schedules: warmup at the start (common for Transformers and large batches), then step decay, exponential decay or cosine annealing, and adaptive optimizers (AdaGrad, RMSprop, Adam) scale the step per parameter. Good practice is to search the learning rate on a log scale (1e-5 to 1), use a learning-rate range test, scale it with batch size, and watch the training loss curve for divergence or stalls.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Effects

Learning rate Behaviour
Too small Slow, may stall on plateaus
Good Fast, steady decrease
Slightly too large Oscillation, noisy loss
Far too large Divergence (loss → ∞ or NaN)

Code Example

import numpy as np

def run_gd(lr, steps=30, x0=10.0):
    x = x0
    for _ in range(steps):
        x -= lr * 2 * x                 # gradient of f(x) = x**2 is 2x
    return x

for lr in [0.01, 0.1, 0.5, 0.9, 1.1]:
    print(f"lr={lr:<4}: x after 30 steps = {run_gd(lr):.4g}")
# lr=0.01: 5.455    (slow: x shrinks by 0.98 per step)
# lr=0.1 : 0.01238  (factor 0.8)
# lr=0.5 : 0        (factor 0: jumps straight to the minimum)
# lr=0.9 : 0.01238  (factor -0.8: oscillates around 0 while shrinking)
# lr=1.1 : 2374     (factor -1.2: diverges)

def cosine_schedule(step, total, lr_max=0.1, lr_min=0.001, warmup=5):
    if step < warmup:
        return lr_max * (step + 1) / warmup
    progress = (step - warmup) / max(1, total - warmup)
    return lr_min + 0.5 * (lr_max - lr_min) * (1 + np.cos(np.pi * progress))

print([round(float(cosine_schedule(s, 50)), 4) for s in [0, 4, 5, 27, 49]])
# [0.02, 0.1, 0.1, 0.0522, 0.0011]

For f(x) = x² the update is x ← (1 − 2η)x, so the factor per step is 0.98, 0.8, 0, −0.8 and −1.2 for the five rates; |1 − 2η| < 1, that is η < 1 (= 2/L with L = 2), is the condition for convergence.

Common Pitfalls

  • Tuning everything else while leaving a poor learning rate in place.

  • Changing the batch size without adjusting the learning rate.

  • Interpreting a diverging loss (NaN) as a data problem when the rate is too high.

Interview Tips

  • Define the step size and show too-small vs too-large behaviour.

  • Mention the 2/L stability bound, schedules, warmup and adaptive methods.

8

What is the bias-variance trade-off in model optimization?

A model’s expected prediction error on new data can be split into bias² (error from wrong or overly simple assumptions, so the model misses real patterns and underfits), variance (error from sensitivity to the particular training sample, so the model fits noise and overfits) and irreducible noise: E[(y − f̂(x))²] = Bias² + Variance + σ². Making the model more flexible (more features, deeper trees, more layers, less regularization, longer training) usually lowers bias but raises variance, and the best model sits where their sum is smallest, which you find with validation data rather than training loss. In optimization terms, the levers are the objective (regularization strength λ, which trades a slightly higher training loss for lower variance), the capacity of the model, and the optimization process itself (early stopping, which halts before the model fits noise; noise from small batches; dropout). More training data reduces variance without increasing bias. Modern over-parameterized networks can show “double descent”, where test error falls again past the interpolation point, but regularization and validation remain the practical tools.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+

Symptoms and Fixes

High bias (underfitting) High variance (overfitting)
Training error High Low
Validation error High, close to training Much higher than training
Fixes More features/capacity, less regularization, train longer More data, more regularization, simpler model, early stopping, ensembling

Code Example: Measuring Bias and Variance by Simulation

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(42)
x_test = np.linspace(0, 1, 50)[:, None]
true_f = np.sin(2 * np.pi * x_test).ravel()

def bias_variance(degree, alpha, n_sets=200, n=30, noise=0.3):
    preds = []
    for _ in range(n_sets):
        x = rng.uniform(0, 1, (n, 1))
        y = np.sin(2 * np.pi * x).ravel() + rng.normal(0, noise, n)
        model = make_pipeline(PolynomialFeatures(degree), Ridge(alpha=alpha)).fit(x, y)
        preds.append(model.predict(x_test))
    preds = np.array(preds)
    bias2 = np.mean((preds.mean(axis=0) - true_f) ** 2)
    variance = np.mean(preds.var(axis=0))
    return bias2, variance

for degree, alpha in [(1, 1e-6), (4, 1e-6), (15, 1e-6), (15, 1e-3)]:
    b2, var = bias_variance(degree, alpha)
    print(f"degree {degree:>2}, alpha {alpha:g}: bias^2 {b2:.3f}, variance {var:.3f}, sum {b2 + var:.3f}")
# degree 1: high bias, low variance; degree 15 with tiny alpha: low bias, high variance;
# degree 4, or degree 15 with more regularization, gives the smallest total

Common Pitfalls

  • Judging models by training error.

  • Thinking regularization always helps; too much causes underfitting.

  • Ignoring that more data mostly attacks variance, not bias.

Interview Tips

  • Write the decomposition and describe the two failure modes.

  • Name the optimization levers: λ, capacity, early stopping, data.

9

What is gradient descent and how does it work?

Gradient descent is an iterative method for minimizing a differentiable objective J(θ): start from an initial guess θ₀, compute the gradient ∇J(θ), which points uphill, take a step of size η (the learning rate) in the opposite direction, θ ← θ − η∇J(θ), and repeat until the gradient is tiny, the loss stops improving, or a step budget runs out. For convex, smooth objectives with a suitable learning rate it converges to the global minimum, and for non-convex ones to a stationary point, usually a local minimum. The batch version uses all training examples for each gradient, stochastic gradient descent (SGD) uses one example, and mini-batch SGD uses a small batch, the standard choice in deep learning because it is cheap per step and its noise helps escape saddle points. Its behaviour depends on the learning rate (too small is slow, too large diverges), feature scaling (badly scaled features create long narrow valleys where it zigzags) and the curvature of the problem; momentum, adaptive learning rates (Adam) and schedules are improvements built on the same idea.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Algorithm

  1. Initialize θ (zeros or small random values).

  2. Compute the gradient ∇J(θ) (analytically or by automatic differentiation).

  3. Update θ ← θ − η∇J(θ).

  4. Stop when ‖∇J‖ < tolerance, the loss plateaus, or after a maximum number of steps.

Code Example: Linear Regression by Gradient Descent

import numpy as np

rng = np.random.default_rng(42)
n = 500
X = np.column_stack([np.ones(n), rng.normal(size=(n, 2))])     # bias column + 2 standardized features
true_w = np.array([4.0, 2.0, -1.0])
y = X @ true_w + rng.normal(0, 0.5, n)

def mse(w):
    return np.mean((X @ w - y) ** 2)

def gradient(w):
    return 2 * X.T @ (X @ w - y) / n

w = np.zeros(3)
lr, history = 0.1, []
for step in range(1000):
    g = gradient(w)
    if np.linalg.norm(g) < 1e-6:                    # stopping rule
        break
    w -= lr * g
    history.append(mse(w))

w_exact = np.linalg.lstsq(X, y, rcond=None)[0]
print(f"stopped after {step} steps, w = {w.round(3)}, exact = {w_exact.round(3)}")
print(f"loss: start {mse(np.zeros(3)):.2f}, after 10 steps {history[9]:.3f}, final {history[-1]:.4f}")
# w is close to [4, 2, -1]; the final loss is about the noise variance 0.25

Variants

Variant Gradient from Pros Cons
Batch GD All data Smooth, exact gradient Expensive per step
SGD One example Very cheap, escapes saddles Very noisy
Mini-batch SGD 32–4096 examples GPU-efficient, balanced noise Needs tuning
Momentum / Nesterov Past gradients Faster in ravines Extra hyperparameter
Adam / RMSprop Per-parameter scaling Robust defaults Can generalize worse than SGD in some tasks

Common Pitfalls

  • Unscaled features, which make convergence slow.

  • A learning rate that is too high (divergence) or too low (stalling).

  • Stopping on a fixed step count without checking the loss curve.

Interview Tips

  • Write the update rule and explain each part.

  • Compare batch, stochastic and mini-batch versions and mention scaling.

10

What is SGD and what are its benefits over standard gradient descent?

Stochastic gradient descent (SGD) updates the parameters using the gradient computed on a single random example or, as almost always in practice, a small random mini-batch, θ ← θ − η∇J_B(θ), instead of the gradient over the full training set used by standard (batch) gradient descent; because the mini-batch gradient is an unbiased estimate of the full gradient, it moves in the right direction on average while each step costs only a tiny fraction of a full pass. The benefits are that each update is far cheaper, so SGD makes many updates per epoch and reaches a good solution sooner on large datasets; it works with data that doesn’t fit in memory or arrives as a stream; mini-batches map well onto GPU parallelism; and its gradient noise helps escape saddle points and sharp minima, which is linked to better generalization in deep learning. The costs are noisy, non-monotonic loss curves and the need to decrease the learning rate (or use schedules or averaging) to converge precisely, plus extra hyperparameters such as batch size; momentum and Adam reduce the noise’s impact.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Batch GD vs SGD

Batch gradient descent Mini-batch SGD
Gradient Exact, full data Noisy estimate from B examples
Cost per update O(n) O(B)
Updates per epoch 1 n/B
Memory Whole dataset One batch
Convergence Smooth Noisy; needs decaying rate
Large data Slow Fast
Generalization — Noise often helps

Code Example: Same Budget, More Progress

import numpy as np

rng = np.random.default_rng(42)
n, d = 50_000, 20
X = rng.normal(size=(n, d))
true_w = rng.normal(size=d)
y = X @ true_w + rng.normal(0, 0.1, n)

def loss(w):
    return float(np.mean((X @ w - y) ** 2))

def batch_gd(epochs, lr=0.1):
    w = np.zeros(d)
    for _ in range(epochs):
        w -= lr * 2 * X.T @ (X @ w - y) / n            # one update per pass over the data
    return w

def minibatch_sgd(epochs, batch=64, lr=0.02, seed=42):
    r = np.random.default_rng(seed)
    w = np.zeros(d)
    for epoch in range(epochs):
        order = r.permutation(n)
        for start in range(0, n, batch):
            idx = order[start:start + batch]
            w -= lr * 2 * X[idx].T @ (X[idx] @ w - y[idx]) / len(idx)
        lr *= 0.5                                        # decay to reduce noise near the optimum
    return w

for epochs in [1, 3]:
    print(f"{epochs} epoch(s): batch GD loss {loss(batch_gd(epochs)):.4f}, "
          f"mini-batch SGD loss {loss(minibatch_sgd(epochs)):.4f}")
# With the same number of passes over the data, SGD makes about 780 updates per epoch instead of 1,
# so it is already close to the noise floor (0.01) while batch GD has barely started

Common Pitfalls

  • A constant learning rate that keeps SGD bouncing around the minimum.

  • Not shuffling the data each epoch, which biases the updates.

  • Comparing methods per update rather than per unit of computation.

Interview Tips

  • Explain the unbiased noisy gradient and cheaper, more frequent updates.

  • Mention noise for generalization and the need for learning-rate decay.

11

Describe the momentum method in optimization

Momentum speeds up gradient descent by giving the parameters a “velocity” that accumulates past gradients: v ← βv + ∇J(θ), then θ ← θ − ηv, with β typically 0.9, so the update is an exponentially weighted sum of recent gradients rather than only the current one. In directions where gradients keep pointing the same way, steps add up and the optimizer accelerates (up to about 1/(1 − β) = 10 times the plain step for β = 0.9); in directions where gradients flip sign, as across a narrow valley, they cancel and the zigzag is damped. That makes momentum much faster on ill-conditioned problems (long, narrow ravines), helps carry the optimizer through flat regions and small bumps, and smooths the noise of mini-batch gradients. Nesterov momentum evaluates the gradient at the look-ahead point θ − ηβv, which corrects the step earlier and is often slightly better. Momentum is built into SGD in PyTorch (momentum=0.9, nesterov=True) and, as the first-moment estimate, into Adam.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Variants

Method Update
Plain gradient descent θ ← θ − η∇J(θ)
Heavy-ball momentum v ← βv + ∇J(θ); θ ← θ − ηv
Nesterov v ← βv + ∇J(θ − ηβv); θ ← θ − ηv
Adam (first moment) m ← β₁m + (1 − β₁)g, with bias correction

Code Example: An Ill-Conditioned Quadratic

import numpy as np

A = np.diag([1.0, 100.0])                     # condition number 100: a long narrow valley

def grad(theta):
    return A @ theta

def optimize(method, lr=0.009, beta=0.9, steps=300):
    theta, v = np.array([10.0, 1.0]), np.zeros(2)
    for step in range(steps):
        if method == "gd":
            theta = theta - lr * grad(theta)
        elif method == "momentum":
            v = beta * v + grad(theta)
            theta = theta - lr * v
        else:                                  # nesterov
            v = beta * v + grad(theta - lr * beta * v)
            theta = theta - lr * v
        if np.linalg.norm(theta) < 1e-3:
            return step + 1
    return None

for method in ["gd", "momentum", "nesterov"]:
    steps = optimize(method)
    print(f"{method:9s}: {'did not reach 1e-3 in 300 steps' if steps is None else f'{steps} steps'}")
# Plain GD crawls along the flat direction (its error there shrinks by only 0.991 per step);
# momentum and Nesterov reach the tolerance in far fewer steps

With a learning rate of 0.009 (just under the stability limit 2/100 for the steep direction), plain gradient descent shrinks the flat coordinate by 1 − 0.009 = 0.991 per step, so reaching 1e-3 from 10 would take about 1,000 steps; momentum effectively multiplies the step along the flat direction.

Common Pitfalls

  • Keeping the learning rate unchanged when adding momentum; the effective step grows by about 1/(1 − β).

  • β too close to 1, which causes large overshoots and oscillation.

  • Forgetting to reset or adapt momentum after a big learning-rate change.

Interview Tips

  • Write the velocity update and explain acceleration and damping.

  • Mention Nesterov and Adam’s use of momentum.

12

What role do second-order methods play in optimization?

Second-order methods use curvature, the Hessian matrix H of second derivatives, in addition to the gradient: Newton’s method minimizes the local quadratic approximation of the objective and steps θ ← θ − H⁻¹∇J(θ), which automatically chooses both the direction and the step size, corrects for badly scaled or correlated parameters, and converges quadratically (the number of correct digits roughly doubles per step) near a minimum. First-order methods (gradient descent, SGD, Adam) use only the gradient, so they are cheap per step (O(d)) but need a tuned learning rate and many iterations on ill-conditioned problems. The price of second-order information is computing and inverting a d × d Hessian, O(d²) memory and O(d³) time, which is infeasible for models with millions of parameters, and Newton steps can go wrong on non-convex functions where the Hessian isn’t positive definite (they may head toward saddle points). Practical compromises are quasi-Newton methods such as BFGS and L-BFGS, which build a Hessian approximation from gradient differences (L-BFGS stores only a few vectors), Gauss-Newton and Levenberg-Marquardt for least squares, Hessian-free/conjugate-gradient methods using Hessian-vector products, and diagonal or block approximations (K-FAC, Shampoo) in deep learning. In scikit-learn, logistic regression’s default L-BFGS solver and the Newton-CG solver are second-order style methods.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+

Comparison

First-order Newton Quasi-Newton (L-BFGS)
Uses Gradient Gradient + Hessian Gradient + curvature estimate
Cost per step O(d) O(d³) (solve with H) O(md), m ≈ 10
Memory O(d) O(d²) O(md)
Convergence Linear (or slower) Quadratic near optimum Superlinear
Tuning Learning rate Little Little
Scale Huge models, stochastic Small d Up to millions, full-batch

Code Example: Rosenbrock with Different Methods

import numpy as np
from scipy.optimize import minimize, rosen, rosen_der, rosen_hess

x0 = np.array([-1.2, 1.0])
for method, kwargs in [("CG", {"jac": rosen_der}),
                       ("BFGS", {"jac": rosen_der}),
                       ("L-BFGS-B", {"jac": rosen_der}),
                       ("Newton-CG", {"jac": rosen_der, "hess": rosen_hess})]:
    res = minimize(rosen, x0, method=method, **kwargs)
    print(f"{method:9s}: x = {res.x.round(5)}, iterations {res.nit:>4}, f = {res.fun:.2e}")
# All reach (1, 1); the methods that use curvature need far fewer iterations than plain gradient descent would

x = x0.copy()
for step in range(6):                              # pure Newton steps, no line search
    x = x - np.linalg.solve(rosen_hess(x), rosen_der(x))
print("pure Newton after 6 steps:", x.round(6))

Plain gradient descent on the Rosenbrock function needs thousands of iterations because its curved valley is badly conditioned; pure Newton steps from this start get close to (1, 1) within a few iterations, but without a line search or trust region they can fail from other starting points.

Common Pitfalls

  • Using full Newton on large models (memory and time).

  • Newton steps on non-convex problems without safeguards (line search, trust region, Hessian modification).

  • L-BFGS with noisy mini-batch gradients, which breaks its curvature estimates.

Interview Tips

  • Contrast gradient-only steps with curvature-aware Newton steps.

  • Explain cost trade-offs and quasi-Newton methods.

13

How does AdaGrad work and what problem does it address?

AdaGrad (Adaptive Gradient, Duchi, Hazan and Singer, 2011) gives every parameter its own learning rate by dividing the step by the square root of the sum of all its past squared gradients: Gₜ = Gₜ₋₁ + gₜ², then θ ← θ − η·gₜ / (√Gₜ + ε), element-wise. Parameters that receive large or frequent gradients get smaller steps, and parameters with rare or small gradients get relatively larger steps, which addresses the problem of a single global learning rate when features have very different frequencies or scales, as in sparse problems like text classification with bag-of-words features, recommendation with rare items, or word embeddings, where rare but informative features would otherwise barely learn. Its weakness is that G only grows, so the effective learning rate keeps shrinking and can become too small to make progress in long training runs, especially for deep, non-convex networks; RMSprop and Adam fix this by using an exponentially decaying average of squared gradients instead of a sum.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Update

Quantity Formula
Accumulator Gₜ = Gₜ₋₁ + gₜ² (per parameter)
Step Δθ = −η·gₜ / (√Gₜ + ε)
Effective rate η / √(Σ past g²)
Typical η 0.01–0.1

Code Example: Rare vs Frequent Features

import numpy as np

rng = np.random.default_rng(42)
n, d = 5_000, 2
X = np.zeros((n, d))
X[:, 0] = rng.normal(size=n)                        # frequent feature: present in every row
rare = rng.random(n) < 0.02
X[rare, 1] = rng.normal(size=rare.sum())            # rare feature: non-zero in 2% of rows
true_w = np.array([1.0, 3.0])
y = X @ true_w + rng.normal(0, 0.1, n)

def train(method, lr, epochs=3):
    w, G = np.zeros(d), np.zeros(d)
    for _ in range(epochs):
        for i in rng.permutation(n):
            g = 2 * (X[i] @ w - y[i]) * X[i]
            if method == "sgd":
                w -= lr * g
            else:
                G += g ** 2
                w -= lr * g / (np.sqrt(G) + 1e-8)
    return w

print("SGD     :", train("sgd", lr=0.01).round(3))
print("AdaGrad :", train("adagrad", lr=0.5).round(3))
# Both learn the frequent weight (about 1.0); AdaGrad gets the rare weight much closer to 3.0,
# because the rare feature's accumulator stays small and its steps stay large

steps = 0.5 / np.sqrt(np.cumsum(np.ones(10_000)))   # effective rate for a constant unit gradient
print(steps[[0, 99, 9_999]].round(4))
# [0.5 0.05 0.005]   the rate keeps shrinking like 1/sqrt(t)

Common Pitfalls

  • Long training runs where AdaGrad’s learning rate decays toward zero.

  • Leaving ε too large, which removes the adaptivity, or too small with float16 (division issues).

  • Expecting AdaGrad to work well for deep non-convex networks; RMSprop or Adam usually work better.

Interview Tips

  • Write the accumulator and update and explain per-parameter rates.

  • Name sparse-feature problems as the motivation and the decaying rate as the weakness.

14

Can you explain RMSprop?

RMSprop (Root Mean Square Propagation, proposed by Geoffrey Hinton in his 2012 Coursera lectures) is an adaptive learning-rate method that divides each parameter’s gradient by a running root-mean-square of its recent gradients: E[g²]ₜ = ρ·E[g²]ₜ₋₁ + (1 − ρ)·gₜ², then θ ← θ − η·gₜ / (√E[g²]ₜ + ε), with ρ around 0.9 and η around 1e-3. Like AdaGrad it shrinks steps for parameters with large gradients and enlarges them for parameters with small ones, which normalizes very different gradient scales across layers and parameters, but because it uses an exponentially decaying average rather than a total sum, old gradients are forgotten and the learning rate doesn’t decay to zero, so it keeps learning in long, non-convex and non-stationary problems; it was popular for recurrent networks and reinforcement learning. Adding momentum to RMSprop and correcting the bias of the moving averages gives Adam, which is why Adam’s second-moment term is essentially RMSprop.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and PyTorch 2.x

AdaGrad vs RMSprop vs Adam

Second-moment estimate Learning-rate decay Momentum
AdaGrad Sum of all g² Always decreasing No
RMSprop Exponential average of g² Adapts up and down Optional
Adam Exponential average of g² (bias-corrected) Adapts Yes (first moment)

Code Example

import numpy as np

def rmsprop_step(theta, grad, state, lr=0.01, rho=0.9, eps=1e-8):
    state = rho * state + (1 - rho) * grad ** 2
    return theta - lr * grad / (np.sqrt(state) + eps), state

def loss_grad(theta):
    # Badly scaled quadratic: gradient scales differ by a factor of 1000
    return np.array([2 * 0.001 * theta[0], 2 * 1.0 * theta[1]])

theta_rms, state = np.array([5.0, 5.0]), np.zeros(2)
theta_gd = np.array([5.0, 5.0])
for _ in range(500):
    theta_rms, state = rmsprop_step(theta_rms, loss_grad(theta_rms), state)
    theta_gd = theta_gd - 0.5 * loss_grad(theta_gd)       # largest stable plain-GD rate here is below 1.0
print("RMSprop:", theta_rms.round(3))
print("plain GD:", theta_gd.round(3))
# RMSprop moves both coordinates toward 0 at a similar pace; plain GD solves the steep
# coordinate at once but barely moves the flat one (it shrinks by only 0.999 per step)

g = np.array([3.0, 0.5])                                  # equal-sized steps from unequal gradients
_, s = rmsprop_step(np.zeros(2), g, np.zeros(2), rho=0.0)
print((0.01 * g / np.sqrt(s)).round(4))
# [0.01 0.01]

With ρ = 0 the running average equals the current g², so each step is η·g/|g| = η·sign(g), the same size for every parameter; that normalization is the core idea.

In PyTorch: torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.99, momentum=0.0) (PyTorch calls ρ alpha).

Common Pitfalls

  • Using AdaGrad-style large learning rates with RMSprop; start near 1e-3.

  • Forgetting ε matters with float16 or tiny gradients.

  • Assuming adaptivity removes the need for learning-rate schedules.

Interview Tips

  • Write the moving-average update and contrast it with AdaGrad’s sum.

  • Mention its role inside Adam.

15

Discuss the Adam optimization algorithm and its key features

Adam (Adaptive Moment Estimation, Kingma and Ba, 2015) combines momentum and RMSprop: it keeps an exponentially decaying average of past gradients (the first moment, mₜ = β₁mₜ₋₁ + (1 − β₁)gₜ) and of past squared gradients (the second moment, vₜ = β₂vₜ₋₁ + (1 − β₂)gₜ²), corrects both for their initialization bias toward zero (m̂ = m/(1 − β₁ᵗ), v̂ = v/(1 − β₂ᵗ)), and updates θ ← θ − η·m̂ / (√v̂ + ε). Its key features are per-parameter adaptive step sizes (robust to different gradient scales across layers), momentum that smooths noisy mini-batch gradients, steps whose size is roughly bounded by η regardless of gradient magnitude, low memory (two extra values per parameter), and defaults that work well across many problems (η = 1e-3, β₁ = 0.9, β₂ = 0.999, ε = 1e-8), which made it the standard optimizer for deep learning. Known issues are that Adam with L2 regularization couples weight decay with the adaptive scaling, which AdamW fixes by applying decoupled weight decay (the usual choice for Transformers), that it can generalize slightly worse than well-tuned SGD with momentum on some vision tasks, and that it still benefits from warmup and learning-rate decay.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and PyTorch 2.x

Hyperparameters

Parameter Default Role
η (lr) 1e-3 Step size (most important to tune)
β₁ 0.9 Momentum (first moment) decay
β₂ 0.999 Second-moment decay
ε 1e-8 Numerical stability
weight_decay (AdamW) 0.01–0.1 Decoupled L2-style shrinkage

Code Example: Adam from Scratch vs PyTorch

import numpy as np
import torch

def adam(grad_fn, theta, lr=0.1, b1=0.9, b2=0.999, eps=1e-8, steps=200):
    m, v = np.zeros_like(theta), np.zeros_like(theta)
    for t in range(1, steps + 1):
        g = grad_fn(theta)
        m = b1 * m + (1 - b1) * g
        v = b2 * v + (1 - b2) * g ** 2
        m_hat, v_hat = m / (1 - b1 ** t), v / (1 - b2 ** t)        # bias correction
        theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
    return theta

def grad_fn(theta):                                   # f = (x - 3)^2 + 100 (y + 2)^2
    return np.array([2 * (theta[0] - 3), 200 * (theta[1] + 2)])

mine = adam(grad_fn, np.array([0.0, 0.0]))

param = torch.zeros(2, dtype=torch.float64, requires_grad=True)
opt = torch.optim.Adam([param], lr=0.1)
for _ in range(200):
    opt.zero_grad()
    loss = (param[0] - 3) ** 2 + 100 * (param[1] + 2) ** 2
    loss.backward()
    opt.step()
print(mine.round(4), param.detach().numpy().round(4))
print("match:", np.allclose(mine, param.detach().numpy(), atol=1e-6))
# match: True; both end near (3, -2) even though the two gradients differ in scale by 100x

first_step = adam(grad_fn, np.array([0.0, 0.0]), steps=1)
print(first_step.round(4))
# [ 0.1 -0.1]   the first step is about lr in each coordinate, whatever the gradient size

With bias correction, the first step is m̂/√v̂ = g/|g| = ±1 per coordinate, so the parameters move by exactly η in the downhill direction of each coordinate.

Adam vs AdamW

Adam + L2 in the loss AdamW
Weight decay Scaled by the adaptive denominator Applied directly: θ ← θ − ηλθ
Effect Weaker, uneven regularization Consistent shrinkage
PyTorch Adam(weight_decay=...) AdamW(weight_decay=...)

Common Pitfalls

  • Using Adam(weight_decay=...) and expecting AdamW behaviour.

  • Too high a learning rate for large models without warmup.

  • Assuming Adam needs no tuning; the learning rate still matters most.

Interview Tips

  • Write the two moment updates, bias correction and the step.

  • Mention defaults, AdamW and the generalization debate with SGD.

16

When would you choose a conjugate gradient method?

Use the conjugate gradient (CG) method when you need to solve a large linear system Ax = b with a symmetric positive-definite matrix A, or equivalently minimize the quadratic ½xᵀAx − bᵀx, and A is too big to factor but cheap to multiply by a vector, for example a sparse matrix from a graph or mesh, a kernel or Gram matrix applied implicitly, or Hessian-vector products in a Newton step. CG builds search directions that are conjugate (A-orthogonal) to each other, so it never undoes progress, finds the exact solution in at most n iterations in exact arithmetic, and in practice reaches good accuracy in far fewer, at a rate governed by √κ (the square root of A’s condition number), which preconditioning improves. It needs only matrix-vector products and a few vectors of memory. Nonlinear CG (Fletcher-Reeves, Polak-Ribière) extends the idea to smooth unconstrained optimization, with memory like gradient descent and often faster convergence, though L-BFGS usually beats it now. CG is the inner solver in Newton-CG and Hessian-free optimization and in scikit-learn’s newton-cg logistic-regression solver. It is not suited to non-symmetric or indefinite systems (use GMRES, MINRES or BiCGSTAB), nor to noisy stochastic gradients.

Read full answer

Applies toNumPy 2.x, SciPy 1.13+

When It Fits

Situation CG?
Large sparse SPD linear system Yes
Matrix only available as a function (A @ v) Yes
Newton step with Hessian-vector products Yes (Newton-CG)
Small dense system No: use a direct solver
Non-symmetric/indefinite matrix No: GMRES, MINRES, BiCGSTAB
Noisy mini-batch gradients No: use SGD/Adam

Code Example

import numpy as np
from scipy.optimize import minimize, rosen, rosen_der
from scipy.sparse import diags
from scipy.sparse.linalg import LinearOperator, cg

n = 10_000
A = diags([-1.0, 2.01, -1.0], [-1, 0, 1], shape=(n, n), format="csr")   # sparse SPD (1-D Laplacian + shift)
rng = np.random.default_rng(42)
x_true = rng.normal(size=n)
b = A @ x_true

iterations = []
x, info = cg(A, b, rtol=1e-10, maxiter=2000, callback=lambda xk: iterations.append(1))
print(f"converged: {info == 0}, iterations: {len(iterations)}, error: {np.linalg.norm(x - x_true):.1e}")
# converges in a few hundred iterations, far fewer than n = 10,000, without ever forming a dense matrix

operator = LinearOperator((n, n), matvec=lambda v: A @ v)                # only matrix-vector products needed
x2, _ = cg(operator, b, rtol=1e-10, maxiter=2000)
print(np.allclose(x, x2))
# True

res = minimize(rosen, np.array([-1.2, 1.0]), jac=rosen_der, method="CG")   # nonlinear CG
print(res.x.round(4), res.nit)

Common Pitfalls

  • Applying CG to matrices that aren’t symmetric positive definite.

  • Skipping preconditioning on badly conditioned systems (slow convergence).

  • Forming the dense matrix when only products are needed.

Interview Tips

  • Name large sparse SPD systems and Hessian-vector products as the use cases.

  • Mention conjugate directions, the √κ rate and preconditioning.

17

What is regularization and why is it used in optimization?

Regularization is any modification of the learning problem that discourages overly complex solutions so the model generalizes better to new data; in optimization terms it usually means adding a penalty to the objective, J(θ) = training loss + λ·R(θ), where R measures complexity, for example the L2 norm ‖θ‖² (ridge, weight decay) or the L1 norm ‖θ‖₁ (Lasso), and λ controls how strongly simplicity is preferred over fitting the training data. It is used because minimizing training loss alone lets flexible models fit noise (overfitting, high variance); the penalty trades a little training error for a large reduction in variance. It also helps the optimization itself: L2 makes ill-conditioned or underdetermined problems (more features than samples, collinear features) strictly convex with a unique, stable solution, and L1 performs feature selection by driving some weights exactly to zero. Other regularizers act through the training process rather than the objective: early stopping, dropout, data augmentation, noise injection, label smoothing, batch normalization’s noise, and constraints such as max-norm. From a Bayesian view, L2 corresponds to a Gaussian prior on the weights and L1 to a Laplace prior. λ is a hyperparameter chosen by cross-validation.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+

Common Regularizers

Regularizer Penalty / mechanism Effect
L2 (ridge, weight decay) λ‖θ‖₂² Shrinks all weights smoothly
L1 (Lasso) λ‖θ‖₁ Sparse weights (feature selection)
Elastic net λ(α‖θ‖₁ + (1 − α)‖θ‖₂²) Sparse and stable with correlated features
Early stopping Stop when validation loss rises Limits effective capacity
Dropout Randomly zero activations Prevents co-adaptation
Data augmentation Transformed copies of inputs Builds in invariances

Code Example: More Features than Samples

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(42)
n, d = 50, 200
X = rng.normal(size=(n, d))
w_true = np.zeros(d)
w_true[:5] = [3, -2, 1.5, 1, -1]
y = X @ w_true + rng.normal(0, 0.5, n)

for name, model in [("no regularization", LinearRegression()),
                    ("ridge alpha=1", Ridge(alpha=1.0)),
                    ("ridge alpha=10", Ridge(alpha=10.0))]:
    model.fit(X, y)
    train_r2 = model.score(X, y)
    cv_r2 = cross_val_score(model, X, y, cv=5, scoring="r2").mean()
    print(f"{name:18s}: train R2 {train_r2:.3f}, CV R2 {cv_r2:.3f}, weight norm {np.linalg.norm(model.coef_):.2f}")
# Without regularization the model fits the training data perfectly (R2 = 1) with 200 weights for 50 rows;
# ridge gives up a little training fit for a smaller, more stable weight vector

print("X^T X invertible:", np.linalg.matrix_rank(X.T @ X) == d)                       # False (rank 50)
print("X^T X + lambda I invertible:", np.linalg.matrix_rank(X.T @ X + 1.0 * np.eye(d)) == d)   # True

Common Pitfalls

  • Penalizing the bias term, which shifts predictions without reducing complexity.

  • Regularizing unscaled features, which penalizes features unevenly.

  • Choosing λ on the test set.

Interview Tips

  • Write loss + λR(θ) and explain the bias-variance trade.

  • Mention L1 vs L2, conditioning benefits, non-penalty regularizers and the Bayesian view.

18

Explain L1 and L2 regularization and their effect on model complexity

L2 regularization adds λ·Σwⱼ² to the objective: it shrinks all weights smoothly toward zero in proportion to their size, rarely making any exactly zero, spreads weight across correlated features, and keeps the problem smooth and strictly convex (ridge regression, weight decay in neural networks); L1 regularization adds λ·Σ|wⱼ|: its constant pull toward zero, independent of the weight’s size, drives many weights exactly to zero, producing sparse models that perform feature selection (Lasso), but it tends to pick one feature arbitrarily from a group of correlated ones and is non-differentiable at zero, so it needs solvers such as coordinate descent or proximal methods. Both reduce model complexity, L2 by limiting the magnitude of weights (a smaller effective number of parameters) and L1 by reducing the number of non-zero weights. Geometrically, the L1 constraint region is a diamond with corners on the axes, where the loss contours tend to touch it first, while the L2 region is a circle with no corners. With orthonormal features, L2 scales each least-squares weight by 1/(1 + λ), and L1 soft-thresholds it, w = sign(ŵ)·max(|ŵ| − λ/2, 0) for the objective ‖y − Xw‖² + λ‖w‖₁. Elastic net combines them.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+

Comparison

L1 (Lasso) L2 (Ridge)
Penalty λΣ|w| λΣw²
Effect on weights Many exactly zero Small, rarely zero
Feature selection Yes No
Correlated features Picks one Shares weight
Differentiable No (at 0) Yes
Solution Coordinate descent, proximal Closed form, any gradient method
Bayesian prior Laplace Gaussian

Code Example

import numpy as np
from sklearn.linear_model import Lasso, Ridge
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n, d = 200, 10
X = StandardScaler().fit_transform(rng.normal(size=(n, d)))
w_true = np.array([4.0, -3.0, 2.0, 0, 0, 0, 0, 0, 0, 0])
y = X @ w_true + rng.normal(0, 1.0, n)

for alpha in [0.01, 0.1, 0.5]:
    lasso = Lasso(alpha=alpha).fit(X, y)
    ridge = Ridge(alpha=alpha * n).fit(X, y)       # rough matching of penalty scales
    print(f"alpha={alpha}: lasso non-zero {np.sum(lasso.coef_ != 0):>2}, ridge non-zero {np.sum(ridge.coef_ != 0):>2}, "
          f"lasso |w| sum {np.abs(lasso.coef_).sum():.2f}, ridge |w|^2 sum {np.sum(ridge.coef_ ** 2):.2f}")
# Lasso zeroes out more of the 7 irrelevant weights as alpha grows; ridge keeps all 10 non-zero

w_ls = np.array([3.0, 0.4, -1.0])                  # least-squares weights for orthonormal features
lam = 1.0
print("L2 shrink:", (w_ls / (1 + lam)).round(3))
print("L1 soft-threshold:", (np.sign(w_ls) * np.maximum(np.abs(w_ls) - lam / 2, 0)).round(3))
# L2 shrink: [ 1.5   0.2  -0.5 ]   L1 soft-threshold: [ 2.5  0.  -0.5]

The closed forms show the difference directly: L2 halves every weight when λ = 1, while L1 subtracts 0.5 from each magnitude and sets the small weight 0.4 to exactly zero.

Common Pitfalls

  • Not standardizing features, so the penalty hits features unevenly.

  • Reading Lasso’s selected features as the only important ones when features are correlated.

  • Comparing α values across libraries with different objective scalings.

Interview Tips

  • Contrast sparsity vs smooth shrinkage and the diamond vs circle geometry.

  • Mention correlated features, solvers and elastic net.

19

What is early stopping in machine learning?

Early stopping is a regularization technique for iterative training (gradient descent on neural networks, boosting rounds, iterative linear solvers): you monitor a validation metric during training and stop when it hasn’t improved for a set number of evaluations (the patience), then restore the parameters from the best epoch. It works because, as training continues, the model first learns general patterns and later starts fitting noise in the training set, so training loss keeps falling while validation loss reaches a minimum and rises; stopping near that minimum picks the best point on the bias-variance curve for free. For linear models trained by gradient descent from zero, early stopping has been shown to behave much like L2 regularization, with the number of steps playing the role of 1/λ. It also saves compute. Practical choices are the monitored metric (validation loss or the business metric), patience (for example 5–20 epochs, or 50 boosting rounds), a minimum improvement threshold, checkpointing the best weights, and keeping a separate test set because the validation set has now been used for model selection. It is built into Keras (EarlyStopping), PyTorch Lightning, XGBoost/LightGBM (early_stopping_rounds) and scikit-learn (early_stopping=True in SGD, MLP and HistGradientBoosting models).

Read full answer

Applies togeneral concepts; examples with scikit-learn 1.5+

Settings

Setting Typical value Purpose
Monitored metric Validation loss What “better” means
Patience 5–20 epochs / 50 rounds Tolerate noisy plateaus
min_delta Small positive number Ignore tiny improvements
restore_best_weights True Return the best model, not the last
Validation split 10–20% Independent of training data

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

X, y = make_classification(n_samples=3000, n_features=40, n_informative=8, flip_y=0.1, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.25, random_state=42)

def train_with_early_stopping(max_epochs=300, patience=10):
    model = MLPClassifier(hidden_layer_sizes=(128, 128), learning_rate_init=1e-3, random_state=42)
    classes = np.unique(y_train)
    best_loss, best_epoch, best_coefs, waited = np.inf, 0, None, 0
    for epoch in range(max_epochs):
        model.partial_fit(X_train, y_train, classes=classes)
        p = np.clip(model.predict_proba(X_val)[:, 1], 1e-7, 1 - 1e-7)
        val_loss = -np.mean(y_val * np.log(p) + (1 - y_val) * np.log(1 - p))
        if val_loss < best_loss - 1e-4:
            best_loss, best_epoch, waited = val_loss, epoch, 0
            best_coefs = ([w.copy() for w in model.coefs_], [b.copy() for b in model.intercepts_])
        else:
            waited += 1
            if waited >= patience:
                break
    model.coefs_, model.intercepts_ = best_coefs                 # restore the best weights
    return model, best_epoch, epoch

model, best_epoch, last_epoch = train_with_early_stopping()
print(f"best epoch {best_epoch}, stopped at epoch {last_epoch}, val accuracy {model.score(X_val, y_val):.3f}")

hgb = HistGradientBoostingClassifier(max_iter=1000, early_stopping=True, validation_fraction=0.2,
                                     n_iter_no_change=20, random_state=42).fit(X_train, y_train)
print(f"boosting stopped after {hgb.n_iter_} of 1000 iterations")

Common Pitfalls

  • Using the test set to decide when to stop.

  • Patience too small for noisy validation curves, which stops too early.

  • Keeping the last weights instead of the best ones.

  • Comparing models stopped on the same validation data they are then evaluated on.

Interview Tips

  • Describe monitoring validation loss with patience and restoring the best weights.

  • Mention its link to L2 regularization and support in common libraries.

20

How does dropout work as a regularization technique in neural networks?

Dropout (Srivastava, Hinton and colleagues, 2014) randomly sets each unit’s output to zero with probability p (commonly 0.1–0.5) on every training step, and scales the surviving outputs by 1/(1 − p) (“inverted dropout”) so their expected value is unchanged; at inference time dropout is turned off and the full network is used. Because any unit may disappear, the network can’t rely on specific co-adapted combinations of units and must spread information redundantly, which reduces overfitting. It can be viewed as training an exponentially large ensemble of thinned subnetworks that share weights, with the full network at test time approximating their average, and as adding multiplicative noise to the activations. It is typically applied after fully connected and attention layers (Transformers use about 0.1), less often in convolutional layers (where spatial dropout or DropBlock drop whole feature maps or regions), and needs a carefully handled variant in RNNs (variational dropout with the same mask across time steps). Keeping dropout on at inference and averaging many predictions, Monte Carlo dropout, gives an estimate of predictive uncertainty. Too much dropout underfits and slows convergence, and it interacts with batch normalization, so many modern CNNs rely more on weight decay and augmentation.

Read full answer

Applies toPyTorch 2.x

Key Facts

Aspect Detail
Training Zero units with probability p, scale survivors by 1/(1 − p)
Inference No dropout (model.eval())
Typical p 0.1 (Transformers), 0.2–0.5 (MLP layers)
Interpretation Ensemble of subnetworks; activation noise
Variants Spatial dropout, DropBlock, DropConnect, variational dropout
Uncertainty MC dropout: average many stochastic passes

Code Example

import torch
from torch import nn

torch.manual_seed(42)
drop = nn.Dropout(p=0.5)
x = torch.ones(10_000)

drop.train()
out = drop(x)
print(f"share zeroed {float((out == 0).float().mean()):.3f}, surviving value {out[out > 0][0].item()}, "
      f"mean {out.mean().item():.3f}")
# about 0.5 zeroed, survivors scaled to 2.0, mean about 1.0 (expected value preserved)

drop.eval()
print(torch.equal(drop(x), x))
# True: dropout is the identity at inference

model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Dropout(0.3), nn.Linear(64, 1))
inputs = torch.randn(5, 20)
model.train()                                      # keep dropout active for MC dropout
with torch.no_grad():
    samples = torch.stack([model(inputs) for _ in range(100)])
print("MC dropout mean shape:", tuple(samples.mean(0).shape), "std of first prediction:",
      round(samples.std(0)[0].item(), 4))

Common Pitfalls

  • Forgetting model.eval() at inference, which leaves predictions random.

  • Too high a dropout rate on small networks, which underfits.

  • Applying standard dropout to recurrent connections with a new mask at every time step.

  • Placing dropout right before batch normalization, which shifts the statistics between training and inference.

Interview Tips

  • Explain random masking, inverted scaling and inference behaviour.

  • Mention the ensemble view, typical rates, variants and MC dropout.

21

What is the idea behind elastic net regularization?

Elastic net (Zou and Hastie, 2005) adds both an L1 and an L2 penalty to the loss, in scikit-learn’s form (1/2n)‖y − Xw‖² + α·ρ·‖w‖₁ + α·(1 − ρ)/2·‖w‖₂², where α sets the overall strength and the mixing ratio ρ (l1_ratio) moves between ridge (ρ = 0) and Lasso (ρ = 1). The idea is to keep the strengths of both: the L1 part produces sparse models and selects features, and the L2 part stabilizes the solution, makes the problem strictly convex, and fixes two known weaknesses of Lasso: with a group of highly correlated features Lasso tends to pick one of them almost arbitrarily (and the choice changes between samples), while elastic net tends to keep or drop the group together (the “grouping effect”); and when there are more features than samples (p > n), Lasso can select at most n features, while elastic net has no such limit. It is a good default for high-dimensional, correlated data such as genomics, text and many engineered features. Both α and ρ are tuned by cross-validation (ElasticNetCV, or LogisticRegression(penalty="elasticnet", solver="saga") for classification), with standardized features.

Read full answer

Applies toscikit-learn 1.5+

Comparison

Lasso Ridge Elastic net
Penalty L1 L2 L1 + L2
Sparse Yes No Yes
Correlated groups Picks one Shares weight Keeps groups together
p > n ≤ n features selected Fine Fine
Strictly convex No Yes Yes (ρ < 1)
Hyperparameters α α α, ρ

Code Example: Correlated Features

import numpy as np
from sklearn.linear_model import ElasticNetCV, LassoCV
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 300
signal = rng.normal(size=n)
group = np.column_stack([signal + rng.normal(0, 0.05, n) for _ in range(4)])   # 4 near-copies of one driver
noise = rng.normal(size=(n, 20))
X = StandardScaler().fit_transform(np.hstack([group, noise]))
y = 3 * signal + rng.normal(0, 0.5, n)

lasso = LassoCV(cv=5, random_state=42).fit(X, y)
enet = ElasticNetCV(l1_ratio=[0.2, 0.5, 0.8], cv=5, random_state=42).fit(X, y)
print("lasso weights on the group:", lasso.coef_[:4].round(2), "noise non-zero:", int(np.sum(lasso.coef_[4:] != 0)))
print("enet  weights on the group:", enet.coef_[:4].round(2), "noise non-zero:", int(np.sum(enet.coef_[4:] != 0)),
      "l1_ratio:", enet.l1_ratio_)
# Lasso tends to concentrate weight on one or two of the copies; elastic net spreads it more evenly
# across all four, and both shrink most noise features to zero

Common Pitfalls

  • Not standardizing features, so the penalties act unevenly.

  • Tuning only α and leaving ρ at a default that doesn’t suit the data.

  • Interpreting selected features causally when the group is correlated.

Interview Tips

  • Write the combined penalty and explain ρ.

  • Mention the grouping effect and the p > n limit of Lasso.

22

What is the vanishing gradient problem and how can it be mitigated?

The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through many layers (or many time steps in a recurrent network), so the earliest layers receive almost no learning signal and train very slowly or not at all; it happens because backpropagation multiplies a chain of local derivatives, and if most factors are below 1, for example sigmoid derivatives (at most 0.25) or weight matrices with small singular values, the product shrinks exponentially with depth. The mitigations target each factor: activation functions whose derivatives don’t saturate (ReLU, GELU, SiLU, leaky ReLU instead of sigmoid or tanh in hidden layers); careful initialization that keeps the variance of activations and gradients constant across layers (Xavier/Glorot for tanh, He/Kaiming for ReLU); normalization layers (batch norm, layer norm) that keep activations in a healthy range; residual (skip) connections, which give gradients an identity path around blocks and made 100+ layer networks trainable (ResNets, Transformers); gated recurrent units (LSTM, GRU), whose additive cell state carries gradients across long sequences; plus shorter effective depth, auxiliary losses, and adaptive optimizers that rescale small gradients. Monitoring per-layer gradient norms during training shows whether the problem is present.

Read full answer

Applies togeneral concepts; examples with PyTorch 2.x

Causes and Fixes

Cause Fix
Saturating activations (sigmoid, tanh) ReLU family, GELU
Poor initialization Xavier/He initialization
Deep plain stacks Residual connections
Shifting activation scales Batch/layer normalization
Long sequences in RNNs LSTM/GRU, attention, truncated BPTT

Code Example: Gradient Norms in a Deep Network

import torch
from torch import nn

def first_layer_grad(activation, depth=30, residual=False, init="default"):
    torch.manual_seed(42)
    layers = [nn.Linear(64, 64) for _ in range(depth)]
    if init == "he":
        for layer in layers:
            nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
            nn.init.zeros_(layer.bias)
    x = torch.randn(128, 64)
    h = x
    for layer in layers:
        out = activation(layer(h))
        h = h + out if residual else out
    loss = h.pow(2).mean()
    loss.backward()
    return layers[0].weight.grad.norm().item()

print(f"sigmoid, plain       : {first_layer_grad(torch.sigmoid):.2e}")
print(f"ReLU + He init, plain: {first_layer_grad(torch.relu, init='he'):.2e}")
print(f"sigmoid + residual   : {first_layer_grad(torch.sigmoid, residual=True):.2e}")
# The plain sigmoid network's first-layer gradient is many orders of magnitude smaller;
# ReLU with He initialization and residual connections keep it at a usable size

print(f"largest sigmoid derivative 0.25, after 30 layers: {0.25 ** 30:.1e}")
# 8.7e-19

Common Pitfalls

  • Using sigmoid or tanh in deep hidden layers without normalization or residuals.

  • Default initialization mismatched to the activation function.

  • Blaming the learning rate when early layers simply receive no gradient; check per-layer norms.

Interview Tips

  • Explain the chain-rule product and why it shrinks.

  • List activations, initialization, normalization, residuals and gated RNNs.

23

Explain the exploding gradient problem and potential solutions

Exploding gradients are the opposite of vanishing gradients: during backpropagation the product of many local derivatives (Jacobians) grows exponentially when their typical magnitude is above 1, so gradients for early layers or early time steps become huge, updates jump far across the loss surface, and training shows sudden loss spikes, NaN or infinite values, or weights that overflow. It is most common in recurrent networks over long sequences (the same weight matrix is multiplied at every step, so if its largest singular value exceeds 1 the gradient grows like that value to the power of the sequence length), in very deep networks with poor initialization, and with high learning rates or mixed-precision training. Solutions are gradient clipping (by global norm, clip_grad_norm_(params, 1.0), or by value), which caps the update size while keeping its direction; careful initialization (orthogonal for RNN weights, Xavier/He elsewhere); normalization layers; gated architectures (LSTM/GRU) and residual connections; smaller learning rates with warmup; weight decay or spectral constraints; truncated backpropagation through time; and loss scaling with overflow checks in float16 training. Logging the gradient norm per step is the simplest way to detect it.

Read full answer

Applies togeneral concepts; examples with PyTorch 2.x

Symptoms and Remedies

Symptom Remedy
Loss suddenly spikes or becomes NaN Gradient clipping, lower learning rate
Gradient norms growing over steps Check initialization, add normalization
RNN unstable on long sequences LSTM/GRU, clipping, truncated BPTT, orthogonal init
float16 overflow Dynamic loss scaling (torch.amp.GradScaler)

Code Example: Clipping Stabilizes an Unstable RNN

import torch
from torch import nn

def train(clip=None, steps=50):
    torch.manual_seed(42)
    rnn = nn.RNN(input_size=8, hidden_size=32, nonlinearity="relu", batch_first=True)
    with torch.no_grad():
        rnn.weight_hh_l0.copy_(1.5 * torch.eye(32))        # recurrent weights with singular values > 1
    head = nn.Linear(32, 1)
    params = list(rnn.parameters()) + list(head.parameters())
    opt = torch.optim.SGD(params, lr=1e-3)
    x, y = torch.rand(16, 40, 8), torch.rand(16, 1)
    norms = []
    for _ in range(steps):
        opt.zero_grad()
        out, _ = rnn(x)
        loss = nn.functional.mse_loss(head(out[:, -1]), y)
        loss.backward()
        total = torch.nn.utils.clip_grad_norm_(params, max_norm=clip if clip else float("inf"))
        norms.append(total.item())
        opt.step()
    return loss.item(), max(norms)

for clip in [None, 1.0]:
    loss, max_norm = train(clip)
    print(f"clip={clip}: final loss {loss:.3e}, largest gradient norm seen {max_norm:.2e}")
# Without clipping the gradient norm is astronomically large and the loss blows up to inf or nan;
# with clipping the update size is capped at lr * 1.0 and training stays finite

print(f"1.5 ** 40 = {1.5 ** 40:.2e}")
# 1.5 ** 40 = 1.11e+07

clip_grad_norm_ returns the norm before clipping, which is why the reported “largest gradient norm” can still be huge in the clipped run; the step actually taken uses the rescaled gradient.

Common Pitfalls

  • Clipping by value, which changes the gradient direction, when norm clipping was intended.

  • A clip threshold so low that every step is clipped, which slows learning.

  • Ignoring NaNs until many steps later; check for them and stop or skip the step.

Interview Tips

  • Explain the repeated multiplication with norms above 1 and where it occurs.

  • Name clipping, initialization, gated units, normalization and lower learning rates.

24

How does imbalanced data affect optimization and how can you address it?

When one class dominates (99% legitimate transactions, 1% fraud), the average loss that the optimizer minimizes is dominated by the majority class, so the easiest way to reduce it is to predict the majority almost always: gradients from the rare class are few and get swamped, the model learns a biased decision boundary, and a 99%-accurate model can miss nearly every positive. The fix starts with evaluation (precision, recall, F1, PR-AUC, or the actual business cost instead of accuracy, with stratified splits) and then changes the optimization or the data: reweight the loss so errors on the minority class cost more (class_weight="balanced", sample_weight, pos_weight in BCEWithLogitsLoss), use losses that focus on hard examples (focal loss), resample the training data (random undersampling of the majority, oversampling or SMOTE for the minority, balanced mini-batches), or keep the probabilities as trained and tune the decision threshold to the cost trade-off. Reweighting and resampling distort predicted probabilities, so recalibrate when probabilities matter; and for extreme imbalance, anomaly detection approaches can be a better fit.

Read full answer

Applies toscikit-learn 1.5+, PyTorch 2.x

Strategies

Level Strategy Notes
Evaluation PR-AUC, recall at precision, cost Never accuracy alone
Loss Class weights, pos_weight, focal loss Simple and effective
Data Undersample, oversample, SMOTE Only on training folds
Batches Balanced sampling Deep learning
Decision Tune the threshold Keeps the model unchanged
Probabilities Recalibrate after reweighting When outputs are used as risks

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_score, recall_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20_000, n_features=10, n_informative=4, weights=[0.99, 0.01],
                           class_sep=1.0, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
print(f"positives in training data: {y_tr.mean():.3%}")

def report(name, proba, threshold=0.5):
    pred = (proba >= threshold).astype(int)
    print(f"{name:28s} accuracy {np.mean(pred == y_te):.3f}  recall {recall_score(y_te, pred):.3f}  "
          f"precision {precision_score(y_te, pred, zero_division=0):.3f}  PR-AUC {average_precision_score(y_te, proba):.3f}")

plain = LogisticRegression(max_iter=1000).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
weighted = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
report("plain", plain)
report("class_weight='balanced'", weighted)
report("plain, threshold 0.1", plain, threshold=0.1)
# The plain model has high accuracy but low recall; weighting or a lower threshold trades precision for recall.
# PR-AUC (a ranking measure) changes little, because weighting mostly shifts the decision threshold

In PyTorch the equivalent of class weighting for binary problems is nn.BCEWithLogitsLoss(pos_weight=torch.tensor([neg / pos])).

Common Pitfalls

  • Reporting accuracy on imbalanced data.

  • Resampling before the train/test split, which leaks duplicated or synthetic points into the test set.

  • Using reweighted probabilities as calibrated risks.

  • Ignoring threshold tuning, often the simplest effective fix.

Interview Tips

  • Explain how the majority dominates the average loss and gradients.

  • Give loss weighting, resampling, threshold tuning and the right metrics.

25

What is overfitting and how can optimization techniques help prevent it?

Overfitting happens when a model fits the training data too closely, including its noise and quirks, so training error is low but error on new data is much higher; it is a sign that the optimizer has found parameters that are very good for the training sample but not for the underlying pattern, typically because the model has more capacity than the data can support or because training ran too long. Optimization techniques help in three ways. Change the objective: add regularization (L2/weight decay, L1, elastic net) so that simpler parameter settings are preferred, or use label smoothing for classifiers. Change the training process: early stopping on a validation metric, dropout and other noise injection, smaller batches or higher learning rates whose gradient noise biases SGD toward flatter minima, learning-rate schedules, and stochastic weight averaging. Change what is optimized over: data augmentation and more data, fewer features or a smaller model, or ensembling. Hyperparameter optimization then chooses the right regularization strength and capacity by cross-validated performance, never by training loss. Always confirm the diagnosis with learning curves (training vs validation loss over epochs or training-set size).

Read full answer

Applies togeneral concepts; examples with scikit-learn 1.5+

Techniques by Mechanism

Mechanism Techniques
Objective L2/weight decay, L1, elastic net, label smoothing
Training process Early stopping, dropout, noise injection, SGD noise, SWA
Data Augmentation, more data, cross-validation
Capacity Smaller models, feature selection, pruning
Selection Hyperparameter search on validation data

Code Example: Validation Curve for Regularization Strength

import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.neural_network import MLPRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
X = rng.uniform(-3, 3, (300, 1))
y = np.sin(X).ravel() + rng.normal(0, 0.3, 300)

model = make_pipeline(StandardScaler(), MLPRegressor(hidden_layer_sizes=(200, 200), max_iter=3000, random_state=42))
alphas = [1e-5, 1e-3, 1e-1, 1, 10]
train_scores, val_scores = validation_curve(model, X, y, param_name="mlpregressor__alpha",
                                            param_range=alphas, cv=5, scoring="r2")
for a, tr, va in zip(alphas, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"alpha {a:<6g}: train R2 {tr:.3f}, validation R2 {va:.3f}, gap {tr - va:.3f}")
# Small alpha: the largest train/validation gap (overfitting); very large alpha: both scores drop (underfitting);
# the best validation score lies in between

Common Pitfalls

  • Judging success by training loss.

  • Regularizing so heavily that the model underfits.

  • Tuning against the test set, which overfits the evaluation.

Interview Tips

  • Define overfitting by the train/validation gap.

  • Group remedies into objective, process, data and capacity.

26

What strategies help optimize non-convex loss functions?

Non-convex losses, the norm for neural networks, matrix factorization and clustering, have many local minima, saddle points and flat regions, so there’s no guarantee of finding the global optimum; the practical aim is to reach a good, preferably flat, minimum reliably, and the strategies fall into a few groups. Start well: sensible initialization (He/Xavier, pretrained weights) and normalized inputs. Move well: stochastic mini-batch gradients, whose noise helps escape saddle points and sharp minima; momentum or Adam to traverse flat regions and ravines; learning-rate warmup followed by decay (cosine, step), or cyclical rates and warm restarts to jump between basins; gradient clipping for stability. Shape the landscape: architecture choices such as residual connections and normalization layers that make the loss surface smoother, over-parameterization (wide networks have fewer bad minima), regularization, and curriculum or continuation methods that start from an easier problem. Search more broadly when the problem is small: multiple random restarts and keeping the best, ensembling different runs, or global methods such as simulated annealing, basin hopping, evolutionary strategies and Bayesian optimization. Sharpness-aware minimization (SAM) and stochastic weight averaging explicitly seek flatter regions, which tends to improve generalization.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+

Strategies

Goal Techniques
Good start He/Xavier init, pretraining, input normalization
Escape saddles/plateaus SGD noise, momentum, Adam
Explore basins Warm restarts, cyclical learning rates, restarts
Smoother landscape Residuals, normalization, over-parameterization
Flat minima SAM, SWA, small-batch noise
Global search (small problems) Basin hopping, annealing, evolutionary methods

Code Example: Restarts and Global Search on a Rugged Function

import numpy as np
from scipy.optimize import basinhopping, differential_evolution, minimize

def rastrigin(x):
    x = np.asarray(x)
    return 10 * len(x) + np.sum(x ** 2 - 10 * np.cos(2 * np.pi * x))   # global minimum 0 at the origin

rng = np.random.default_rng(42)
single = minimize(rastrigin, rng.uniform(-5, 5, 2))
restarts = min((minimize(rastrigin, rng.uniform(-5, 5, 2)) for _ in range(50)), key=lambda r: r.fun)
hopping = basinhopping(rastrigin, rng.uniform(-5, 5, 2), niter=200, seed=42)
evolution = differential_evolution(rastrigin, bounds=[(-5, 5)] * 2, seed=42)

for name, res in [("single local run", single), ("50 restarts", restarts),
                  ("basin hopping", hopping), ("differential evolution", evolution)]:
    print(f"{name:22s}: f = {res.fun:.4f} at {np.round(res.x, 3)}")
# A single run usually stops in a local minimum (f of a few units); restarts and global methods reach about 0

Common Pitfalls

  • Judging an architecture from one training run.

  • A constant large learning rate that never settles, or a tiny one that gets stuck.

  • Using global search methods on models with millions of parameters (far too expensive).

Interview Tips

  • Group strategies into initialization, optimizer dynamics, landscape shaping and global search.

  • Mention saddle points, flat minima, SAM/SWA and restarts.

27

What factors influence the convergence rate of an optimization algorithm?

The convergence rate depends on the problem, the algorithm and the data. Problem: the curvature of the objective, summarized by the condition number κ = L/μ (largest over smallest curvature); for gradient descent on a strongly convex function the error shrinks by about (1 − 1/κ) per step, so a badly conditioned problem with κ = 10,000 converges thousands of times slower than a well-conditioned one, while convexity, smoothness (a bounded L) and strong convexity (μ > 0) all decide which guarantees hold, and non-convexity, saddle points and plateaus slow things further. Algorithm: the learning rate (too small is slow, too large oscillates or diverges; η ≈ 1/L is the classic choice), momentum and Nesterov acceleration (improving the dependence to about √κ), adaptive per-parameter scaling (Adam, RMSprop), second-order information (Newton converges quadratically near the optimum; L-BFGS superlinearly), and learning-rate schedules. Data and setup: feature scaling and normalization (which directly reduce κ), gradient noise from small batches (SGD converges sublinearly, like 1/t, unless the step decays or variance is reduced as in SVRG), batch size, initialization, and numerical precision. Rates are measured in iterations, but wall-clock time also depends on the cost per iteration.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Typical Rates

Setting Method Error after t steps
Smooth convex Gradient descent O(1/t)
Smooth convex Nesterov O(1/t²)
Strongly convex Gradient descent O((1 − 1/κ)ᵗ) (linear)
Strongly convex Momentum (heavy ball / Nesterov) O((1 − 1/√κ)ᵗ)
Strongly convex, near optimum Newton Quadratic
Stochastic gradients, convex SGD with decaying step O(1/√t) (or O(1/t) strongly convex)

Code Example: Conditioning and Scaling

import numpy as np

def gd_steps(A, lr, tol=1e-6, max_steps=1_000_000):
    """Steps of gradient descent on f(x) = 0.5 x^T A x from x0 = (1, 1)."""
    x = np.ones(len(A))
    for step in range(max_steps):
        if np.linalg.norm(x) < tol:
            return step
        x = x - lr * (A @ x)
    return max_steps

for kappa in [1, 10, 100, 1000]:
    A = np.diag([1.0, float(kappa)])
    print(f"kappa {kappa:>4}: {gd_steps(A, lr=1 / kappa):>6} steps")
# kappa 1: 1 step; then roughly 14 * kappa steps: about 130, 1,400, 14,000

With η = 1/L = 1/κ the steep coordinate is solved immediately and the flat one shrinks by 1 − 1/κ per step, so reaching 1e-6 needs about ln(10⁶)·κ ≈ 13.8κ steps. Rescaling the features so both directions have equal curvature (κ = 1) would make it converge in one step, which is why standardization matters so much for gradient methods.

Common Pitfalls

  • Unscaled features (a high condition number) blamed on the optimizer.

  • Comparing iteration counts across methods with very different costs per iteration.

  • Expecting exact convergence from SGD with a constant learning rate.

Interview Tips

  • Lead with the condition number and the (1 − 1/κ) rate.

  • Mention learning rate, momentum, second-order methods, noise and scaling.

28

How does batch size affect the optimization process in SGD?

Batch size B controls how many examples each gradient estimate averages over, so it trades gradient noise against cost per step: the variance of the mini-batch gradient falls like 1/B, so small batches give noisy, cheap updates and many of them per epoch, while large batches give accurate, expensive updates and few of them. Effects follow from that. Speed: larger batches use GPUs more efficiently, and up to a “critical batch size” doubling B roughly halves the number of steps needed (with the learning rate scaled up), but beyond it extra examples mostly waste computation. Learning rate: larger batches allow, and need, a larger learning rate; the linear scaling rule (multiply η by k when B grows k-fold) plus warmup works well up to a point, and the square-root rule is used with Adam. Generalization: the noise of small batches acts as implicit regularization and tends to find flatter minima, so very large batches can generalize slightly worse unless compensated with tuning, warmup, longer training or optimizers like LARS/LAMB. Memory: batch size is limited by GPU memory, with gradient accumulation to simulate larger batches. Batch normalization also behaves poorly with very small batches. Typical choices are 32–512 for many tasks, and thousands to millions of tokens for large language models.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Small vs Large Batches

Small batch (e.g. 32) Large batch (e.g. 4096)
Gradient noise High Low
Updates per epoch Many Few
Hardware efficiency Lower Higher
Learning rate Smaller Larger (with warmup)
Generalization Often better (implicit regularization) May need tuning to match
Memory Low High (or gradient accumulation)

Code Example: Noise vs Batch Size

import numpy as np

rng = np.random.default_rng(42)
n, d = 100_000, 10
X = rng.normal(size=(n, d))
w_true = rng.normal(size=d)
y = X @ w_true + rng.normal(0, 1.0, n)
w = np.zeros(d)
full_grad = 2 * X.T @ (X @ w - y) / n

for B in [8, 64, 512, 4096]:
    errors = []
    for _ in range(300):
        idx = rng.integers(0, n, B)
        g = 2 * X[idx].T @ (X[idx] @ w - y[idx]) / B
        errors.append(np.linalg.norm(g - full_grad))
    print(f"batch {B:>5}: mean gradient error {np.mean(errors):.3f}")
# The error shrinks roughly like 1/sqrt(B): each 8x larger batch cuts it by about 2.8x

def sgd_to_tolerance(B, lr, tol=0.05, max_epochs=20):
    r = np.random.default_rng(0)
    w = np.zeros(d)
    steps = 0
    for epoch in range(max_epochs):
        for start in range(0, n, B):
            idx = r.integers(0, n, B)
            w -= lr * 2 * X[idx].T @ (X[idx] @ w - y[idx]) / B
            steps += 1
            if np.linalg.norm(w - w_true) < tol:
                return steps, steps * B
    return steps, steps * B

for B, lr in [(32, 0.005), (256, 0.04), (2048, 0.3)]:      # learning rate scaled roughly with B
    steps, examples = sgd_to_tolerance(B, lr)
    print(f"batch {B:>4}, lr {lr}: {steps} steps, {examples} examples processed")

Scaling the learning rate with the batch size keeps the number of examples processed roughly similar until the batch becomes so large that its gradients are nearly exact; then extra examples per step stop paying off.

Common Pitfalls

  • Changing batch size without retuning the learning rate.

  • Very small batches with batch normalization (unstable statistics).

  • Assuming a larger batch always trains faster in wall-clock time.

Interview Tips

  • Explain the 1/B variance, cost per step and the critical batch size.

  • Mention learning-rate scaling, warmup, generalization and gradient accumulation.

29

Why is feature scaling important for optimization algorithms?

Gradient-based optimizers converge fast when the loss surface is round and slowly when it is a long, narrow valley, and feature scales decide which you get: if one feature is in rupees (values up to 10⁶) and another is a ratio between 0 and 1, the curvature of the loss with respect to their weights differs by many orders of magnitude, the condition number of the problem becomes huge, and gradient descent must use a learning rate small enough for the steepest direction, so it crawls along the flat ones and zigzags across the valley. Standardizing features (zero mean, unit variance) or scaling them to similar ranges makes the curvature similar in all directions, so one learning rate suits all weights and convergence can be orders of magnitude faster; centring also decouples the bias from the other weights. Scaling matters for regularization too, because L1/L2 penalties treat all weights alike and would otherwise punish features unevenly, and for distance-based methods (kNN, k-means, SVMs with RBF kernels) and PCA. Tree-based models are invariant to monotonic scaling and don’t need it. Adaptive optimizers (Adam) reduce but don’t remove the problem, and in neural networks normalization layers (batch/layer norm) apply the same idea to hidden activations. Fit scalers on training data only, inside a pipeline.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Where Scaling Matters

Method Scaling needed?
Gradient descent / SGD / neural networks Yes
L1/L2-regularized linear models Yes
kNN, k-means, SVM (RBF) Yes
PCA Yes (unless units are shared)
Decision trees, random forests, boosting No
Closed-form least squares Not for the solution, but for numerical stability

Code Example

import numpy as np

rng = np.random.default_rng(42)
n = 1000
income = rng.uniform(1e4, 1e6, n)            # large scale
ratio = rng.uniform(0, 1, n)                 # small scale
X_raw = np.column_stack([np.ones(n), income, ratio])
y = 2.0 + 3e-6 * income + 4.0 * ratio + rng.normal(0, 0.1, n)

def gd_steps(X, y, tol=1e-6, max_steps=200_000):
    H = 2 * X.T @ X / len(y)
    lr = 1.0 / np.linalg.eigvalsh(H).max()                  # largest stable-ish step for this problem
    w = np.zeros(X.shape[1])
    target = np.linalg.lstsq(X, y, rcond=None)[0]
    for step in range(max_steps):
        if np.linalg.norm(X @ w - X @ target) / np.sqrt(len(y)) < tol:
            return step, np.linalg.cond(H)
        w -= lr * 2 * X.T @ (X @ w - y) / len(y)
    return max_steps, np.linalg.cond(H)

X_std = X_raw.copy()
X_std[:, 1:] = (X_raw[:, 1:] - X_raw[:, 1:].mean(axis=0)) / X_raw[:, 1:].std(axis=0)

for name, X in [("raw features", X_raw), ("standardized", X_std)]:
    steps, cond = gd_steps(X, y)
    print(f"{name:13s}: condition number {cond:.1e}, steps to converge {steps}")
# Raw: a condition number around 1e12 or more, and gradient descent hits the 200,000-step limit;
# standardized: a condition number near 1 and convergence in a few dozen steps

Common Pitfalls

  • Fitting the scaler on all data (leakage).

  • Scaling binary or one-hot columns unnecessarily (harmless but can hurt interpretability).

  • Forgetting to apply the same fitted scaler at prediction time.

  • Min-max scaling with outliers, which squashes the rest of the data.

Interview Tips

  • Explain elongated loss contours, the condition number and the shared learning rate.

  • Name the methods that need scaling and those that don’t.

30

How can optimization algorithms be parallelized?

Most ML optimization parallelizes the expensive part of each step, computing gradients, and the main patterns are data parallelism, model parallelism and parallel search. Data parallelism splits each mini-batch across workers (GPUs or machines) that hold a copy of the model, compute gradients on their shard and average them with an all-reduce before every worker applies the same update; it is synchronous SGD (PyTorch DistributedDataParallel, Horovod), scales well as long as the global batch and learning rate are adjusted, and can be made asynchronous (parameter servers, Hogwild!) at the cost of stale gradients. Model parallelism splits a model too large for one device: tensor parallelism shards individual layers, pipeline parallelism places layer groups on different devices and streams micro-batches through them, and sharded data parallelism (ZeRO, FSDP) partitions optimizer state, gradients and parameters across workers. For classical algorithms, gradients and sufficient statistics are sums over examples, so map-reduce works (distributed L-BFGS, coordinate descent per feature block, ADMM for consensus problems), and boosting libraries parallelize split finding across features and machines. Hyperparameter optimization is embarrassingly parallel (independent trials in grid or random search), with asynchronous methods like ASHA and Hyperband or parallel Bayesian optimization. Communication is usually the bottleneck, addressed with gradient compression, overlapping communication with computation, local SGD (sync every few steps) and large-batch optimizers (LARS, LAMB).

Read full answer

Applies togeneral concepts; examples with Python 3.12 and NumPy 2.x

Patterns

Pattern What is split Examples
Data parallel (sync) Mini-batch DDP, Horovod, all-reduce
Data parallel (async) Mini-batch, stale updates Parameter servers, Hogwild!
Tensor parallel Layers’ weight matrices Megatron-LM
Pipeline parallel Groups of layers GPipe, PipeDream
Sharded data parallel Optimizer state, gradients, weights ZeRO, FSDP
Map-reduce statistics Examples Distributed L-BFGS, Spark MLlib
Parallel search Hyperparameter trials Ray Tune, Optuna, ASHA

Code Example: Synchronous Data-Parallel Gradient Descent (Simulated Workers)

from concurrent.futures import ThreadPoolExecutor

import numpy as np

rng = np.random.default_rng(42)
n, d, workers = 40_000, 20, 4
X = rng.normal(size=(n, d))
w_true = rng.normal(size=d)
y = X @ w_true + rng.normal(0, 0.1, n)
shards = list(zip(np.array_split(X, workers), np.array_split(y, workers)))

def local_gradient(args):
    (Xs, ys), w = args
    return 2 * Xs.T @ (Xs @ w - ys), len(ys)            # sum of gradients on this shard

w = np.zeros(d)
with ThreadPoolExecutor(max_workers=workers) as pool:
    for step in range(100):
        results = list(pool.map(local_gradient, [(shard, w) for shard in shards]))
        grad = sum(g for g, _ in results) / sum(m for _, m in results)   # "all-reduce": average
        w -= 0.1 * grad                                                    # same update on every worker

full_grad_w = np.zeros(d)
for _ in range(100):                                                      # single-process reference
    full_grad_w -= 0.1 * 2 * X.T @ (X @ full_grad_w - y) / n
print("parallel equals serial:", np.allclose(w, full_grad_w))
# parallel equals serial: True
print("error vs true weights:", round(float(np.linalg.norm(w - w_true)), 4))

Synchronous data parallelism computes exactly the same full gradient as one machine, which is why results match; the speed-up comes from computing the shard gradients at the same time on different devices.

Common Pitfalls

  • Scaling to many workers without raising the batch size and learning rate (or warmup).

  • Communication costs that exceed the computation saved.

  • Asynchronous updates with very stale gradients, which hurt convergence.

  • Non-deterministic results from floating-point reduction order; fix seeds and accept small differences.

Interview Tips

  • Explain synchronous data parallelism with all-reduce first.

  • Mention model/pipeline/sharded parallelism, map-reduce for classical methods and parallel hyperparameter search.

31

How do you choose an optimization algorithm for a given problem?

Match the algorithm to the problem’s structure, size and constraints rather than defaulting to one optimizer: ask whether the objective is convex or not, smooth or not, whether gradients are available, how many parameters and examples there are, whether there are constraints, and whether you need an exact optimum or a good-enough model quickly. Small or medium smooth problems with full-batch gradients (logistic regression, GLMs, fitting a few physical parameters) suit quasi-Newton L-BFGS, Newton or trust-region methods, which need almost no tuning; L1-regularized and other non-smooth convex problems suit coordinate descent or proximal methods (as in scikit-learn’s Lasso and liblinear/saga solvers); constrained problems suit SLSQP, interior-point methods, or linear and quadratic programming solvers (linprog, SVM dual solvers); very large datasets and deep networks need stochastic first-order methods, AdamW as a robust default (especially for Transformers) and SGD with momentum and a schedule as a strong choice for vision; problems without usable gradients (hyperparameters, simulators, discrete choices) suit derivative-free methods: Bayesian optimization for expensive evaluations, CMA-ES or evolutionary methods, Nelder-Mead for a few variables, random search for many hyperparameters. Then validate empirically: compare a couple of candidates on wall-clock time to a target validation metric, not just iterations.

Read full answer

Applies togeneral concepts; examples with SciPy 1.13+ and scikit-learn 1.5+

Decision Guide

Problem Good choices
Smooth, small/medium, full batch L-BFGS, Newton/trust-region, BFGS
Least squares Normal equations/QR, Levenberg-Marquardt (least_squares)
L1 or other non-smooth convex terms Coordinate descent, proximal gradient, saga
Linear or quadratic constraints linprog, QP solvers, SLSQP, interior point
Huge data, deep networks Mini-batch SGD + momentum, Adam/AdamW, with schedules
Sparse, high-dimensional data SGD, AdaGrad, saga, FTRL
No gradients, expensive evaluations Bayesian optimization (Optuna, scikit-optimize)
No gradients, cheap evaluations CMA-ES, differential evolution, random search
Discrete/combinatorial Integer programming, simulated annealing, genetic algorithms

Code Example: Picking scipy.optimize Methods by Problem Type

import numpy as np
from scipy.optimize import differential_evolution, linprog, minimize, rosen, rosen_der

smooth = minimize(rosen, x0=[-1.2, 1.0], jac=rosen_der, method="L-BFGS-B")
print("smooth unconstrained (L-BFGS-B):", smooth.x.round(4), smooth.nit, "iterations")

constrained = minimize(lambda x: (x[0] - 2) ** 2 + (x[1] - 1) ** 2, x0=[0.0, 0.0], method="SLSQP",
                       constraints=[{"type": "ineq", "fun": lambda x: 1 - x[0] - x[1]}],   # x0 + x1 <= 1
                       bounds=[(0, None), (0, None)])
print("constrained (SLSQP):", constrained.x.round(4))
# [1. 0.]: the closest point to (2, 1) inside the region x0 + x1 <= 1, x >= 0

lp = linprog(c=[-3, -2], A_ub=[[1, 1], [1, 3]], b_ub=[4, 6], bounds=[(0, None), (0, None)])
print("linear program (linprog):", lp.x.round(4), "objective", round(-lp.fun, 4))
# [4. 0.] objective 12.0: maximize 3x + 2y subject to x + y <= 4, x + 3y <= 6

def rugged(x):
    return np.sum(x ** 2 - 10 * np.cos(2 * np.pi * x)) + 10 * len(x)

global_search = differential_evolution(rugged, bounds=[(-5, 5)] * 3, seed=42)
print("multimodal, no gradients (differential evolution):", global_search.x.round(3))
# about [0, 0, 0]

For the constrained example, the unconstrained optimum (2, 1) violates x₀ + x₁ ≤ 1; projecting onto the line x₀ + x₁ = 1 gives (1, 0), which also satisfies x ≥ 0. For the linear program, the corners of the feasible region are (0, 0), (4, 0), (3, 1) and (0, 2), with objective values 0, 12, 11 and 4.

Common Pitfalls

  • Using Adam for a small convex problem where L-BFGS converges in a few iterations.

  • Using L-BFGS with noisy mini-batch gradients.

  • Comparing optimizers on iterations rather than time to a target quality.

  • Ignoring constraints and clipping parameters afterwards.

Interview Tips

  • Walk through the questions: convexity, smoothness, size, gradients, constraints.

  • Give a default per category and say you would benchmark two candidates.

32

What methods can be used to tune hyperparameters effectively?

Effective hyperparameter tuning combines a sensible search space, an efficient search strategy, honest evaluation and a compute budget. The main strategies are grid search, which tries every combination and suits two or three discrete hyperparameters; random search (Bergstra and Bengio, 2012), which samples combinations and finds good settings far more efficiently than grids when only a few hyperparameters matter, especially with log-uniform ranges for learning rates and regularization strengths; Bayesian optimization (Gaussian processes, tree-structured Parzen estimators in Optuna or Hyperopt, scikit-optimize), which builds a model of score versus settings and chooses promising trials, best when each training run is expensive; multi-fidelity methods, successive halving and Hyperband (HalvingGridSearchCV, HalvingRandomSearchCV, Optuna pruners), which start many configurations on a small budget (few epochs, subsets of data) and give more resources only to the best; and population-based training and evolutionary methods for large deep-learning runs. Whatever the method, use cross-validation or a fixed validation set, keep a final test set untouched, prefer nested cross-validation when reporting tuned performance on small data, fix random seeds, log every trial, and tune the most influential hyperparameters first (learning rate, regularization, model size).

Read full answer

Applies toscikit-learn 1.5+, Optuna 3.x/4.x

Methods

Method Best for Library
Grid search Few discrete settings GridSearchCV
Random search Many hyperparameters, cheap runs RandomizedSearchCV
Bayesian optimization Expensive runs Optuna (TPE), scikit-optimize, Hyperopt
Successive halving / Hyperband Many configs, cheap early signal HalvingRandomSearchCV, Optuna pruners, Ray Tune
Population-based training Long deep-learning runs Ray Tune
Manual + learning curves Building intuition, final polish —

Code Example: Random Search, Successive Halving and Optuna

from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.experimental import enable_halving_search_cv  # noqa: F401
from sklearn.model_selection import HalvingRandomSearchCV, RandomizedSearchCV, cross_val_score

import optuna

X, y = load_breast_cancer(return_X_y=True)
space = {"learning_rate": loguniform(1e-3, 0.5), "max_leaf_nodes": randint(4, 64),
         "l2_regularization": loguniform(1e-4, 10)}
model = HistGradientBoostingClassifier(random_state=42)

rand = RandomizedSearchCV(model, space, n_iter=25, cv=5, random_state=42, n_jobs=-1).fit(X, y)
print(f"random search: {rand.best_score_:.3f} {rand.best_params_}")

halving = HalvingRandomSearchCV(model, space, n_candidates=60, factor=3, resource="max_iter",
                                max_resources=300, cv=5, random_state=42, n_jobs=-1).fit(X, y)
print(f"successive halving: {halving.best_score_:.3f} after {halving.n_iterations_} rounds")

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.5, log=True),
        "max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 4, 64),
        "l2_regularization": trial.suggest_float("l2_regularization", 1e-4, 10, log=True),
    }
    return cross_val_score(HistGradientBoostingClassifier(random_state=42, **params), X, y, cv=5).mean()

optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=25)
print(f"Optuna TPE: {study.best_value:.3f} {study.best_params}")

All three typically reach a cross-validated accuracy around 0.96–0.97 on this dataset; the differences between methods matter far more when each training run is expensive.

Common Pitfalls

  • Linear ranges for scale parameters such as learning rate and regularization.

  • Selecting and reporting on the same validation data (optimistic estimates).

  • Huge grids that waste computation on unimportant hyperparameters.

  • Forgetting to fix seeds, so differences between trials are partly noise.

Interview Tips

  • Compare grid, random, Bayesian and multi-fidelity methods with when to use each.

  • Mention log scales, cross-validation, nested CV and budgets.

33

How does the choice of optimization algorithm affect model interpretability?

The optimizer doesn’t change the type of model, but it changes which solution you get and how stable it is, which affects interpretation. For convex problems with a unique solution (ridge regression, logistic regression with L2), any converged solver gives the same coefficients, so interpretation is unaffected as long as it truly converged; an unconverged solver (too few iterations, a too-small tolerance budget) gives coefficients that are not the real optimum and may be misread. Where the solution isn’t unique, the algorithm effectively picks one: with perfectly correlated features or more features than samples, gradient descent started at zero converges to the minimum-norm solution, while other solvers can return different but equally fitting coefficients; Lasso solved by coordinate descent may choose different members of a correlated group depending on feature order or tolerance. Sparsity, often the basis of interpretation, also depends on the method: coordinate descent and proximal methods produce exact zeros for L1, while plain (sub)gradient descent produces tiny non-zero weights. For non-convex models such as neural networks, different optimizers, seeds and schedules reach different minima with different internal representations and feature attributions, so explanations (saliency maps, SHAP values) should be checked across runs. Practical safeguards are checking convergence warnings, standardizing features, using regularization to make solutions unique, and testing whether conclusions are stable across solvers, seeds and bootstrap samples.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x

Effects

Situation How the optimizer matters
Convex, unique optimum Same answer if converged; check convergence
Non-unique optimum (collinearity, p > n) Different solvers pick different solutions
L1 regularization Exact zeros only with coordinate/proximal methods
Early stopping Acts like regularization; changes coefficients
Non-convex networks Different minima, different attributions

Code Example

import warnings

import numpy as np
from sklearn.exceptions import ConvergenceWarning
from sklearn.linear_model import Lasso, LogisticRegression
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 500
x1 = rng.normal(size=n)
X = StandardScaler().fit_transform(np.column_stack([x1, x1 + rng.normal(0, 0.01, n), rng.normal(size=(n, 3))]))
y = 2 * x1 + 0.5 * X[:, 2] + rng.normal(0, 0.3, n)

lasso = Lasso(alpha=0.05).fit(X, y)

w, b = np.zeros(X.shape[1]), 0.0                     # same Lasso objective, plain subgradient descent
for _ in range(5000):
    err = X @ w + b - y
    w -= 0.01 * (X.T @ err / n + 0.05 * np.sign(w))
    b -= 0.01 * err.mean()

print("coordinate descent:", lasso.coef_.round(3), "exact zeros:", int(np.sum(lasso.coef_ == 0)))
print("subgradient       :", w.round(3), "exact zeros:", int(np.sum(w == 0)))
# Coordinate descent sets irrelevant weights exactly to zero; the subgradient method leaves them
# jittering around zero with tiny non-zero values, and splits the near-duplicate columns differently

y_cls = (y > 0).astype(int)
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always", ConvergenceWarning)
    early = LogisticRegression(max_iter=3, solver="lbfgs").fit(X, y_cls)
full = LogisticRegression(max_iter=1000, solver="lbfgs").fit(X, y_cls)
print("converged?", not any(issubclass(w.category, ConvergenceWarning) for w in caught))
# converged? False
print("3 iterations :", early.coef_.round(2))
print("converged fit:", full.coef_.round(2))

Common Pitfalls

  • Ignoring convergence warnings and interpreting unconverged coefficients.

  • Interpreting individual coefficients of highly correlated features.

  • Trusting one neural network’s explanations without checking other seeds.

Interview Tips

  • Separate unique-optimum cases from non-unique and non-convex ones.

  • Mention convergence checks, sparsity from the solver and stability across runs.

34

Implement basic gradient descent to minimize a quadratic function

For a quadratic f(x) = ½xᵀAx − bᵀx with a symmetric positive-definite matrix A, the gradient is ∇f(x) = Ax − b and the unique minimum is the solution of Ax = b, so gradient descent is just the loop x ← x − η(Ax − b), stopped when the gradient norm falls below a tolerance or a step limit is reached. A good implementation takes the gradient as a function, records the history of the objective, checks for divergence, and chooses a safe step size: gradient descent on this problem converges for any 0 < η < 2/λ_max(A), and η = 1/λ_max is a reliable default, while the speed depends on the condition number κ = λ_max/λ_min (the error shrinks by about 1 − 1/κ per step). Comparing the result with np.linalg.solve(A, b) verifies the implementation.

Read full answer

Applies toPython 3.12, NumPy 2.x

Code Example

import numpy as np

def gradient_descent(grad, x0, lr, tol=1e-8, max_iter=10_000, f=None):
    """Minimize with fixed-step gradient descent. Returns (x, iterations, history)."""
    x = np.asarray(x0, dtype=float)
    history = []
    for i in range(max_iter):
        g = grad(x)
        if f is not None:
            history.append(f(x))
        if np.linalg.norm(g) < tol:
            return x, i, history
        x = x - lr * g
        if not np.all(np.isfinite(x)):
            raise FloatingPointError(f"diverged at iteration {i}; reduce the learning rate")
    return x, max_iter, history

# One-dimensional warm-up: f(x) = (x - 3)^2, gradient 2(x - 3)
x, iters, _ = gradient_descent(lambda x: 2 * (x - 3), x0=[0.0], lr=0.1)
print(x.round(6), iters)
# [3.] 91

# Two-dimensional quadratic: f(x) = 0.5 x^T A x - b^T x
A = np.array([[3.0, 1.0], [1.0, 2.0]])
b = np.array([1.0, -1.0])

def f(x):
    return 0.5 * x @ A @ x - b @ x

def grad(x):
    return A @ x - b

eigs = np.linalg.eigvalsh(A)
lr = 1 / eigs.max()
x, iters, history = gradient_descent(grad, x0=[0.0, 0.0], lr=lr, f=f)
print("gradient descent:", x.round(6), "iterations:", iters)
print("exact solution  :", np.linalg.solve(A, b).round(6))
# both [ 0.6 -0.8]
print(f"condition number {eigs.max() / eigs.min():.2f}, f at minimum {history[-1]:.4f}")
# condition number 2.62, f at minimum -0.7000

try:
    gradient_descent(grad, x0=[0.0, 0.0], lr=2.5 / eigs.max(), max_iter=2_000)
except FloatingPointError as exc:
    print("too large a step:", exc)

The exact solution of Ax = b is x = (0.6, −0.8): 3·0.6 + 1·(−0.8) = 1 and 1·0.6 + 2·(−0.8) = −1. The minimum value is −½bᵀx = −½(0.6 + 0.8) = −0.7. In the one-dimensional case each step multiplies the error by 0.8, and the gradient 2·3·0.8ᵏ drops below 1e-8 after 91 steps.

Common Pitfalls

  • A learning rate above 2/λ_max, which diverges.

  • Stopping on a fixed iteration count without checking the gradient.

  • Using a non-symmetric or indefinite A, for which this “quadratic” has no minimum.

Interview Tips

  • Derive the gradient Ax − b and the stability condition on η.

  • Verify against the closed-form solution and mention the condition number.

35

Write a Python function to perform SGD on a sample dataset

A mini-batch stochastic gradient descent function for linear regression (or any differentiable model) shuffles the data at the start of every epoch, splits it into mini-batches, computes the gradient of the loss on each batch, and updates the weights with w ← w − η·g; good implementations add a learning-rate schedule (for example η/(1 + decay·epoch)), optional momentum and L2 regularization, a record of the training loss per epoch, and early stopping on a validation set. For mean squared error, the batch gradient is (2/B)·Xᵦᵀ(Xᵦw − yᵦ). Standardizing the features first and checking the result against a closed-form or library solution (np.linalg.lstsq or scikit-learn’s SGDRegressor) confirm that the function works.

Read full answer

Applies toPython 3.12, NumPy 2.x, scikit-learn 1.5+

Code Example

import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

def sgd_linear_regression(X, y, lr=0.05, epochs=50, batch_size=32, momentum=0.9, l2=0.0,
                          decay=0.01, X_val=None, y_val=None, patience=5, seed=42):
    """Mini-batch SGD with momentum, L2 and early stopping. Returns (w, b, history)."""
    rng = np.random.default_rng(seed)
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    vw, vb = np.zeros(d), 0.0
    history = {"train": [], "val": []}
    best = (np.inf, w.copy(), b)
    waited = 0
    for epoch in range(epochs):
        step = lr / (1 + decay * epoch)
        for idx in np.array_split(rng.permutation(n), max(1, n // batch_size)):
            err = X[idx] @ w + b - y[idx]
            gw = 2 * X[idx].T @ err / len(idx) + 2 * l2 * w
            gb = 2 * err.mean()
            vw, vb = momentum * vw + gw, momentum * vb + gb
            w, b = w - step * vw, b - step * vb
        history["train"].append(float(np.mean((X @ w + b - y) ** 2)))
        if X_val is not None:
            val = float(np.mean((X_val @ w + b - y_val) ** 2))
            history["val"].append(val)
            if val < best[0] - 1e-6:
                best, waited = (val, w.copy(), b), 0
            else:
                waited += 1
                if waited >= patience:
                    return best[1], best[2], history
    if X_val is not None:
        return best[1], best[2], history
    return w, b, history

X, y = make_regression(n_samples=2000, n_features=8, noise=10.0, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_tr)
X_tr, X_val = scaler.transform(X_tr), scaler.transform(X_val)

w, b, hist = sgd_linear_regression(X_tr, y_tr, lr=0.01, X_val=X_val, y_val=y_val)
exact = np.linalg.lstsq(np.column_stack([X_tr, np.ones(len(X_tr))]), y_tr, rcond=None)[0]
print(f"epochs run: {len(hist['train'])}, final validation MSE: {hist['val'][-1]:.1f}")
print("max difference from least squares:", float(np.abs(np.r_[w, b] - exact).max().round(3)))
# Validation MSE close to the noise variance (10^2 = 100) and weights close to the exact solution

Design Notes

Feature Why
Shuffle every epoch Unbiased batches, no order effects
Momentum Smooths noisy gradients
Decaying step Reduces noise near the optimum
L2 term Regularization
Early stopping with best weights Prevents overfitting, saves time
Standardized features Stable learning rate

Common Pitfalls

  • Forgetting to shuffle, especially when data is sorted by target or time.

  • A learning rate too large for the momentum setting (the effective step is about lr/(1 − momentum)).

  • Computing gradients as sums instead of means, which ties the learning rate to batch size.

Interview Tips

  • Show the batch loop, gradient formula and update.

  • Mention shuffling, schedules, momentum and validation.

36

Simulate how different learning rates affect convergence

A clear simulation runs the same optimizer from the same starting point on the same problem with several learning rates and plots the loss against iterations on a log scale; on a simple quadratic, the behaviour can be predicted exactly, which makes the plot easy to interpret: for f(x, y) = ½(x² + 10y²) the gradient step multiplies each coordinate by (1 − η·λ), with curvatures λ = 1 and λ = 10, so any η below 0.2 converges, η = 0.2 sits on the edge (the steep coordinate oscillates forever), and larger η diverges. Small rates converge slowly and steadily; moderate rates converge fastest (η ≈ 2/(λ_min + λ_max) ≈ 0.18 is optimal here); rates close to the limit converge but oscillate in the steep direction; rates beyond it explode. Running the same rates with mini-batch SGD on a regression problem adds the effect of noise: large rates plateau at a higher, noisier loss, which is why learning-rate decay is used.

Read full answer

Applies toPython 3.12, NumPy 2.x, Matplotlib 3.8+

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np

curvatures = np.array([1.0, 10.0])

def loss(p):
    return 0.5 * np.sum(curvatures * p ** 2)

def run(lr, steps=60):
    p = np.array([5.0, 2.0])
    losses = [loss(p)]
    for _ in range(steps):
        p = p - lr * curvatures * p
        losses.append(loss(p))
        if losses[-1] > 1e6:
            break
    return losses

rates = [0.01, 0.05, 0.18, 0.199, 0.21]
fig, ax = plt.subplots(figsize=(8, 5))
for lr in rates:
    losses = run(lr)
    ax.plot(losses, label=f"lr = {lr}")
    print(f"lr={lr:<5}: per-step factors {np.round(1 - lr * curvatures, 3)}, final loss {losses[-1]:.3g}")
ax.set(yscale="log", xlabel="iteration", ylabel="loss", title="Gradient descent with different learning rates")
ax.legend()
fig.tight_layout()
fig.savefig("learning_rates.png", dpi=120)
# lr=0.01 : factors [0.99 0.9]    slow
# lr=0.05 : factors [0.95 0.5]    faster
# lr=0.18 : factors [0.82 -0.8]   fastest overall (both factors about 0.8 in size)
# lr=0.199: factors [0.801 -0.99] the steep coordinate oscillates and decays very slowly
# lr=0.21 : factors [0.79 -1.1]   diverges

rng = np.random.default_rng(42)                        # noisy version: mini-batch SGD on linear regression
X = rng.normal(size=(5000, 5))
y = X @ rng.normal(size=5) + rng.normal(0, 0.5, 5000)
for lr in [0.001, 0.01, 0.1]:
    w = np.zeros(5)
    for step in range(3000):
        idx = rng.integers(0, 5000, 16)
        w -= lr * 2 * X[idx].T @ (X[idx] @ w - y[idx]) / 16
    print(f"SGD lr={lr}: final full-data MSE {np.mean((X @ w - y) ** 2):.4f}")
# Small rates may not have converged yet; large rates converge quickly but plateau above the noise floor 0.25

What to Look For

Learning rate Curve
Too small Smooth, slow straight-ish decline on the log plot
Good Steep decline
Near the limit Zigzag, slow in the steep direction
Too large Loss grows; NaN
SGD, large Fast start, noisy plateau

Common Pitfalls

  • Comparing rates over too few iterations, which hides slow convergence.

  • Plotting on a linear scale, which hides differences after the first few steps.

  • Using different random seeds per run, which mixes noise with the learning-rate effect.

Interview Tips

  • Explain the per-step factor 1 − ηλ and the 2/λ_max limit.

  • Show both the deterministic and the noisy (SGD) picture.

37

Implement momentum in a gradient descent optimizer

Momentum keeps a velocity vector that accumulates an exponentially decaying sum of past gradients and moves the parameters along it: v ← βv + g, θ ← θ − ηv (the PyTorch convention), or equivalently v ← βv − ηg, θ ← θ + v; Nesterov’s variant evaluates the gradient at the look-ahead position. An optimizer class needs to store the velocity per parameter, apply the update in a step(params, grads) method, and support β (typically 0.9), the learning rate, optional Nesterov, and optional weight decay. Testing it on an ill-conditioned problem, such as the Rosenbrock function or a narrow quadratic valley, shows the benefit: fewer iterations and less zigzagging than plain gradient descent with the same learning rate, and matching PyTorch’s SGD(momentum=0.9) step by step confirms correctness.

Read full answer

Applies toPython 3.12, NumPy 2.x, PyTorch 2.x

Code Example

import numpy as np
import torch

class MomentumSGD:
    """SGD with (Nesterov) momentum, using PyTorch's update convention."""

    def __init__(self, lr=0.01, momentum=0.9, nesterov=False, weight_decay=0.0):
        self.lr, self.momentum, self.nesterov, self.weight_decay = lr, momentum, nesterov, weight_decay
        self.velocity = None

    def step(self, params, grads):
        if self.velocity is None:
            self.velocity = [np.zeros_like(p) for p in params]
        new_params = []
        for p, g, v in zip(params, grads, self.velocity):
            g = g + self.weight_decay * p
            v *= self.momentum
            v += g                                        # v <- beta * v + g (in place)
            update = g + self.momentum * v if self.nesterov else v
            new_params.append(p - self.lr * update)
        return new_params

def rosenbrock_grad(p):
    x, y = p
    return np.array([-2 * (1 - x) - 400 * x * (y - x ** 2), 200 * (y - x ** 2)])

def minimize_with(opt, steps=5000, tol=1e-4):
    p = np.array([-1.2, 1.0])
    for i in range(steps):
        (p,) = opt.step([p], [rosenbrock_grad(p)])
        if np.linalg.norm(p - 1) < tol:
            return p, i + 1
    return p, steps

for name, opt in [("plain GD", MomentumSGD(lr=1e-3, momentum=0.0)),
                  ("momentum 0.9", MomentumSGD(lr=1e-3, momentum=0.9)),
                  ("Nesterov 0.9", MomentumSGD(lr=1e-3, momentum=0.9, nesterov=True))]:
    p, steps = minimize_with(opt)
    print(f"{name:13s}: reached {p.round(4)} in {steps} steps")
# Plain GD uses the whole 5000-step budget without getting within 1e-4 of (1, 1);
# with momentum the iterate gets much closer within the same budget (Nesterov is usually a little faster still)

torch_p = torch.tensor([-1.2, 1.0], dtype=torch.float64, requires_grad=True)
torch_opt = torch.optim.SGD([torch_p], lr=1e-3, momentum=0.9, nesterov=True)
mine = MomentumSGD(lr=1e-3, momentum=0.9, nesterov=True)
p = np.array([-1.2, 1.0])
for _ in range(100):
    torch_opt.zero_grad()
    loss = (1 - torch_p[0]) ** 2 + 100 * (torch_p[1] - torch_p[0] ** 2) ** 2
    loss.backward()
    torch_opt.step()
    (p,) = mine.step([p], [rosenbrock_grad(p)])
print("matches PyTorch after 100 steps:", np.allclose(p, torch_p.detach().numpy()))
# matches PyTorch after 100 steps: True

Common Pitfalls

  • Mixing the two velocity conventions and getting the learning-rate scaling wrong.

  • Using the same learning rate as plain GD with high momentum, which can overshoot.

  • Forgetting to keep velocity per parameter tensor.

Interview Tips

  • Write both update forms and the Nesterov look-ahead.

  • Verify against a library implementation.

38

Write a regularization function that penalizes large weights in linear regression

To penalize large weights, add an L2 (ridge) term to the mean squared error, J(w, b) = (1/n)‖Xw + b − y‖² + λ‖w‖², and include its gradient 2λw when optimizing; leave the intercept b unpenalized, because shifting predictions up or down isn’t model complexity. A clean implementation separates a penalty(w, kind, lam) function (returning the penalty and its gradient, supporting L2, L1 with the subgradient λ·sign(w), and elastic net) from the training routine, which can be closed form for L2, w = (XᵀX + nλI)⁻¹Xᵀy on centred data, or gradient descent for any penalty. Standardize features first so the penalty treats them equally, choose λ by cross-validation, and check against scikit-learn’s Ridge, remembering that Ridge(alpha) minimizes ‖y − Xw‖² + α‖w‖² (a sum, not a mean), so α = nλ.

Read full answer

Applies toPython 3.12, NumPy 2.x, scikit-learn 1.5+

Code Example

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler

def penalty(w, kind="l2", lam=0.1, l1_ratio=0.5):
    """Return (value, gradient) of a weight penalty. The intercept is not passed in."""
    if kind == "l2":
        return lam * np.sum(w ** 2), 2 * lam * w
    if kind == "l1":
        return lam * np.sum(np.abs(w)), lam * np.sign(w)          # subgradient at 0 is taken as 0
    if kind == "elasticnet":
        v1, g1 = penalty(w, "l1", lam * l1_ratio)
        v2, g2 = penalty(w, "l2", lam * (1 - l1_ratio))
        return v1 + v2, g1 + g2
    raise ValueError(kind)

def fit_regularized(X, y, kind="l2", lam=0.1, lr=0.05, steps=5000):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(steps):
        err = X @ w + b - y
        _, pen_grad = penalty(w, kind, lam)
        w -= lr * (2 * X.T @ err / n + pen_grad)
        b -= lr * 2 * err.mean()
    return w, b

def ridge_closed_form(X, y, lam):
    n, d = X.shape
    Xc, yc = X - X.mean(axis=0), y - y.mean()
    w = np.linalg.solve(Xc.T @ Xc + n * lam * np.eye(d), Xc.T @ yc)
    return w, y.mean() - X.mean(axis=0) @ w

rng = np.random.default_rng(42)
X = StandardScaler().fit_transform(rng.normal(size=(200, 5)))
y = X @ np.array([5.0, -3.0, 0.0, 0.0, 1.0]) + 10 + rng.normal(0, 1, 200)

lam = 0.5
w_gd, b_gd = fit_regularized(X, y, "l2", lam)
w_cf, b_cf = ridge_closed_form(X, y, lam)
sk = Ridge(alpha=lam * len(y)).fit(X, y)
print(np.allclose(w_gd, w_cf, atol=1e-4), np.allclose(w_cf, sk.coef_), round(b_cf, 3), round(float(sk.intercept_), 3))
# True True (intercepts equal, about 10)

for lam in [0.0, 0.1, 1.0, 10.0]:
    w, _ = ridge_closed_form(X, y, lam)
    print(f"lambda {lam:>4}: ||w|| = {np.linalg.norm(w):.3f}, w = {w.round(2)}")
# The weight norm shrinks as lambda grows; with standardized features each weight is about w_ls / (1 + lambda)

With standardized features, XcᵀXc ≈ nI, so the ridge solution is roughly w_ls / (1 + λ); for λ = 1 the weights are about halved.

Common Pitfalls

  • Penalizing the intercept.

  • Forgetting the n scaling when comparing with scikit-learn’s alpha.

  • Using plain gradient descent for L1 and expecting exact zeros (use coordinate descent or proximal steps).

  • Unscaled features, which make the penalty uneven.

Interview Tips

  • Write the penalized objective, its gradient and the closed form.

  • Mention the unpenalized intercept, scaling and λ selection.

39

Write a script that uses the Adam optimizer in PyTorch

A complete PyTorch training script with Adam defines a model (nn.Module), a loss function, and the optimizer torch.optim.Adam(model.parameters(), lr=1e-3) (or AdamW with decoupled weight decay, the usual choice today), then runs the standard loop: for each mini-batch, optimizer.zero_grad(), forward pass, loss.backward(), optional gradient clipping, optimizer.step(), and optionally a learning-rate scheduler step. Good scripts also fix random seeds, put the model in train() and eval() modes as appropriate, evaluate on a validation set without gradients (torch.no_grad()), move data and model to the available device, and save the best model’s state_dict. The key Adam hyperparameters are the learning rate (most important), betas (0.9, 0.999), eps and weight decay.

Read full answer

Applies toPyTorch 2.x, scikit-learn 1.5+ (for the sample data)

Code Example

import torch
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

X, y = make_classification(n_samples=4000, n_features=20, n_informative=10, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler().fit(X_tr)

def to_loader(X, y, shuffle):
    ds = TensorDataset(torch.tensor(scaler.transform(X), dtype=torch.float32),
                       torch.tensor(y, dtype=torch.float32))
    return DataLoader(ds, batch_size=64, shuffle=shuffle)

train_loader, val_loader = to_loader(X_tr, y_tr, True), to_loader(X_val, y_val, False)

model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Dropout(0.1), nn.Linear(64, 1)).to(device)
loss_fn = nn.BCEWithLogitsLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, betas=(0.9, 0.999), weight_decay=1e-2)
epochs = 30
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=epochs)

best_acc = 0.0
for epoch in range(epochs):
    model.train()
    for xb, yb in train_loader:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        loss = loss_fn(model(xb).squeeze(1), yb)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()
    scheduler.step()

    model.eval()
    correct = total = 0
    with torch.no_grad():
        for xb, yb in val_loader:
            preds = (model(xb.to(device)).squeeze(1) > 0).float()
            correct += (preds == yb.to(device)).sum().item()
            total += len(yb)
    acc = correct / total
    if acc > best_acc:
        best_acc = acc
        torch.save(model.state_dict(), "best_model.pt")
    if epoch % 10 == 0 or epoch == epochs - 1:
        print(f"epoch {epoch:>2}: last batch loss {loss.item():.4f}, val accuracy {acc:.3f}, "
              f"lr {scheduler.get_last_lr()[0]:.2e}")
print(f"best validation accuracy {best_acc:.3f}")

Swap AdamW for torch.optim.Adam(model.parameters(), lr=1e-3) to use plain Adam; with Adam, weight_decay adds an L2 term to the gradient instead of decoupled decay.

Common Pitfalls

  • Forgetting optimizer.zero_grad(), which accumulates gradients across steps.

  • Applying a sigmoid before BCEWithLogitsLoss (it already includes one).

  • Evaluating without model.eval() and torch.no_grad().

  • Calling scheduler.step() per batch for schedulers designed per epoch, or the reverse.

Interview Tips

  • Write the five-line training step from memory.

  • Mention AdamW vs Adam, schedulers, clipping and evaluation mode.

40

Show the difference between L1 and L2 regularization on a small dataset

A good demonstration fits Lasso (L1) and Ridge (L2) on the same small, standardized dataset where only a few features matter, then compares the coefficients across a range of regularization strengths: L1 drives irrelevant coefficients exactly to zero and eventually zeroes everything, giving a sparse model, while L2 shrinks all coefficients smoothly toward zero without making them exactly zero, and with correlated features L1 tends to keep one while L2 splits the weight between them. Plotting the regularization paths (coefficient value versus log α) makes the difference visible, and reporting the number of non-zero coefficients and the cross-validated error at each strength shows the trade-off between sparsity and accuracy. Because scikit-learn scales the objectives differently (Lasso uses a 1/(2n) factor on the squared error, Ridge doesn’t), compare the shape of the paths rather than equal α values.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import Lasso, Ridge, lasso_path
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

def compare_l1_l2(n=60, seed=42):
    rng = np.random.default_rng(seed)
    x0 = rng.normal(size=n)
    X = np.column_stack([
        x0,
        x0 + rng.normal(0, 0.1, n),               # feature 1: correlated copy of feature 0
        rng.normal(size=n),                        # feature 2: relevant
        rng.normal(size=(n, 5)),                   # features 3-7: irrelevant
    ])
    X = StandardScaler().fit_transform(X)
    y = 3 * x0 + 2 * X[:, 2] + rng.normal(0, 1, n)

    print(f"{'alpha':>6} | {'L1 non-zero':>11} {'L1 CV R2':>8} | {'L2 non-zero':>11} {'L2 CV R2':>8}")
    for alpha in [0.01, 0.1, 0.5, 1.0]:
        lasso, ridge = Lasso(alpha=alpha), Ridge(alpha=alpha * n)
        lasso.fit(X, y)
        ridge.fit(X, y)
        print(f"{alpha:>6} | {np.sum(lasso.coef_ != 0):>11} {cross_val_score(lasso, X, y, cv=5).mean():>8.3f} | "
              f"{np.sum(ridge.coef_ != 0):>11} {cross_val_score(ridge, X, y, cv=5).mean():>8.3f}")

    print("L1 coefficients (alpha=0.1):", Lasso(alpha=0.1).fit(X, y).coef_.round(2))
    print("L2 coefficients (alpha=6)  :", Ridge(alpha=6).fit(X, y).coef_.round(2))

    alphas_l1, coefs_l1, _ = lasso_path(X, y, alphas=np.logspace(-3, 0.5, 60))
    alphas_l2 = np.logspace(-1, 4, 60)
    coefs_l2 = np.array([Ridge(alpha=a).fit(X, y).coef_ for a in alphas_l2]).T
    fig, axes = plt.subplots(1, 2, figsize=(12, 4.5), sharey=True)
    for coef in coefs_l1:
        axes[0].plot(alphas_l1, coef)
    for coef in coefs_l2:
        axes[1].plot(alphas_l2, coef)
    axes[0].set(xscale="log", title="L1 (Lasso) path: coefficients hit zero", xlabel="alpha", ylabel="coefficient")
    axes[1].set(xscale="log", title="L2 (Ridge) path: smooth shrinkage", xlabel="alpha")
    fig.tight_layout()
    fig.savefig("l1_vs_l2_paths.png", dpi=120)

compare_l1_l2()
# L1 keeps only a few non-zero coefficients (often one of the two correlated features plus feature 2);
# L2 keeps all 8 non-zero, splitting weight between features 0 and 1

Common Pitfalls

  • Comparing α values directly between Lasso and Ridge (different objective scaling).

  • Skipping standardization, which makes the penalty favour features with large scales.

  • Concluding that a zeroed correlated feature is useless.

Interview Tips

  • Show the coefficient table and the regularization paths.

  • Explain sparsity vs shrinkage and the correlated-features behaviour.

41

How would you optimize a deep neural network for image recognition?

Start from a strong, proven setup and change one thing at a time while watching training and validation curves: use a pretrained backbone (ResNet, EfficientNet, ConvNeXt or a Vision Transformer) and fine-tune it, because transfer learning beats training from scratch unless you have millions of labelled images; train with SGD with momentum 0.9 and a cosine or one-cycle schedule with warmup (a strong choice for CNNs) or AdamW with weight decay (standard for ViTs), and find the learning rate with a short range test, using lower rates for the pretrained layers than for the new head. Fight overfitting with data augmentation (random resized crops, flips, colour jitter, RandAugment or TrivialAugment, Mixup/CutMix), weight decay, label smoothing, dropout or stochastic depth, and early stopping on validation accuracy. Make training efficient and stable with batch normalization, mixed-precision training (torch.autocast with bfloat16 or float16 plus a gradient scaler), the largest batch that fits (with the learning rate scaled), gradient clipping, and fast data loading (multiple workers, pinned memory). Finally tune the most influential hyperparameters (learning rate, weight decay, augmentation strength, epochs), use test-time augmentation or model ensembles if accuracy matters most, and analyse errors by class with a confusion matrix to decide what data or changes would help.

Read full answer

Applies toPyTorch 2.x, torchvision 0.18+

Checklist

Area Choices
Model Pretrained ResNet/EfficientNet/ConvNeXt/ViT
Optimizer SGD + momentum (CNNs), AdamW (ViTs)
Schedule Warmup + cosine or one-cycle
Regularization Augmentation, weight decay, label smoothing, stochastic depth, Mixup/CutMix
Efficiency Mixed precision, large batches, fast data loading, channels-last
Fine-tuning Freeze then unfreeze, discriminative learning rates
Evaluation Validation curves, per-class errors, TTA, ensembles

Code Example: Fine-Tuning Setup

import torch
from torch import nn
from torchvision import models, transforms

device = "cuda" if torch.cuda.is_available() else "cpu"
num_classes, epochs, steps_per_epoch = 10, 20, 100

model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)   # downloads pretrained weights
model.fc = nn.Linear(model.fc.in_features, num_classes)
model = model.to(device).to(memory_format=torch.channels_last)

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.TrivialAugmentWide(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

backbone = [p for name, p in model.named_parameters() if not name.startswith("fc.")]
optimizer = torch.optim.SGD([
    {"params": backbone, "lr": 0.01},                    # smaller rate for pretrained layers
    {"params": model.fc.parameters(), "lr": 0.1},        # larger rate for the new head
], momentum=0.9, weight_decay=1e-4, nesterov=True)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=[0.01, 0.1],
                                                total_steps=epochs * steps_per_epoch, pct_start=0.1)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
scaler = torch.amp.GradScaler(enabled=device == "cuda")

def train_step(images, labels):
    model.train()
    optimizer.zero_grad(set_to_none=True)
    with torch.autocast(device_type=device, dtype=torch.float16 if device == "cuda" else torch.bfloat16):
        loss = loss_fn(model(images.to(device, memory_format=torch.channels_last)), labels.to(device))
    scaler.scale(loss).backward()
    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    scaler.step(optimizer)
    scaler.update()
    scheduler.step()
    return loss.item()

images, labels = torch.randn(8, 3, 224, 224), torch.randint(0, num_classes, (8,))   # stand-in batch
print(f"one training step, loss {train_step(images, labels):.3f}")

Plug train_tf into an ImageFolder dataset and a DataLoader(num_workers=..., pin_memory=True) to train on real images.

Common Pitfalls

  • Training from scratch on small datasets.

  • Augmentation that changes the label (flipping digits or text).

  • A single learning rate for a pretrained backbone and a random head.

  • Tuning on the test set or ignoring per-class failures.

Interview Tips

  • Lead with transfer learning, optimizer and schedule, then augmentation and regularization.

  • Mention mixed precision, error analysis and an incremental, curve-driven workflow.

42

How would you handle vanishing gradients in RNNs?

In a plain RNN, the gradient with respect to an early time step is a product of one Jacobian per step, ∂hₜ/∂hₜ₋₁ = diag(f′)·W, so over long sequences it shrinks exponentially whenever those factors are below 1 (tanh derivatives are at most 1 and usually smaller), and the network can’t learn dependencies more than a few dozen steps back. The steps I would take: first confirm the problem by logging gradient norms per time step or per layer and testing on a task with known long-range dependencies; then switch to gated cells, LSTM or GRU, whose additive cell-state update (with the forget gate near 1) gives gradients a nearly constant path through time, and initialize the forget-gate bias to 1 or more so memory is kept by default; initialize recurrent weights orthogonally (or as an identity for ReLU RNNs, as in IRNN) so the Jacobians start with singular values near 1; add layer normalization inside the recurrence; use skip connections across time or layers and keep stacked RNNs shallow; clip gradients to handle the opposite, exploding case that often appears together; use truncated backpropagation through time with sensible window lengths; and if dependencies are very long, consider attention on top of the RNN, temporal convolutions, or a Transformer or state-space model, which connect distant positions directly.

Read full answer

Applies toPyTorch 2.x

Steps

Step Action
Diagnose Log gradient norms vs time step; synthetic long-dependency tests
Architecture LSTM or GRU instead of vanilla RNN
Initialization Orthogonal recurrent weights; forget-gate bias ≈ 1
Normalization Layer norm in the cell
Training Gradient clipping, truncated BPTT, sensible learning rate
Alternatives Attention, TCNs, Transformers, state-space models

Code Example: How Far Back Does the Gradient Reach?

import torch
from torch import nn

def gradient_by_timestep(cell_type, seq_len=100, hidden=64, seed=42):
    torch.manual_seed(seed)
    if cell_type == "rnn":
        net = nn.RNN(16, hidden, batch_first=True, nonlinearity="tanh")
    else:
        net = nn.LSTM(16, hidden, batch_first=True)
        for name, p in net.named_parameters():
            if "bias" in name:                           # forget-gate bias slice set to 1
                p.data[hidden:2 * hidden] = 1.0
    for name, p in net.named_parameters():
        if "weight_hh" in name:
            nn.init.orthogonal_(p)
    x = torch.randn(32, seq_len, 16, requires_grad=True)
    out, _ = net(x)
    out[:, -1].pow(2).sum().backward()                  # loss depends only on the last output
    g = x.grad.norm(dim=(0, 2))                         # gradient reaching each input time step
    return g[0].item(), g[seq_len // 2].item(), g[-1].item()

for cell in ["rnn", "lstm"]:
    first, middle, last = gradient_by_timestep(cell)
    print(f"{cell:4s}: gradient at t=0 {first:.2e}, t=50 {middle:.2e}, t=99 {last:.2e}, "
          f"ratio first/last {first / last:.1e}")
# The vanilla RNN's gradient at t=0 is many orders of magnitude smaller than at the last step;
# the LSTM keeps a much larger share of the signal across the 100 steps

Common Pitfalls

  • Treating long training times as an optimizer problem when gradients simply don’t reach early steps.

  • Forgetting gradient clipping, since exploding and vanishing gradients often coexist.

  • Truncated BPTT windows shorter than the dependencies the model must learn.

Interview Tips

  • Explain the product of Jacobians and why gating helps.

  • List diagnosis, LSTM/GRU, initialization, normalization, clipping and attention.

43

How would you optimize a recommendation system that deals with sparse data?

Sparse interaction data (each user has rated or clicked only a tiny fraction of items, often under 0.1% of the user-item matrix is filled) makes recommenders overfit the few observed entries and struggle with new users and items, so the approach combines models that share information, objectives that use implicit signals well, and optimizers built for sparsity. Model choices: matrix factorization learns low-dimensional user and item embeddings so that sparse rows borrow strength from similar users (ALS, which alternates closed-form least-squares updates and parallelizes well, or SGD-trained factorization); implicit-feedback methods (weighted ALS, BPR’s pairwise ranking loss) treat unobserved entries as weak negatives; factorization machines and two-tower neural models add side features (item category, text and image embeddings, user demographics) to handle cold start; and graph-based models propagate preferences through the interaction graph. Optimization choices: strong L2 regularization on embeddings, scaled by how often each user or item appears; negative sampling instead of scoring all items; sparse-aware optimizers such as AdaGrad, sparse Adam or FTRL, which give rarely seen embeddings larger effective learning rates; sparse matrix formats (CSR) and embedding tables with sparse gradients. Evaluation and serving: time-based splits, ranking metrics (recall@k, NDCG@k), popularity baselines, approximate nearest-neighbour retrieval for candidates, and exploration so new items get impressions.

Read full answer

Applies toNumPy 2.x, SciPy 1.13+

Toolkit

Challenge Technique
Few ratings per user Matrix factorization, regularization
Implicit feedback Weighted ALS, BPR, negative sampling
Cold start Side features (factorization machines, two-tower), content embeddings
Rare users/items Frequency-scaled regularization, AdaGrad/FTRL
Scale Sparse matrices, ALS parallelism, ANN retrieval
Evaluation Time split, recall@k, NDCG@k vs popularity

Code Example: ALS Matrix Factorization on a Sparse Matrix

import numpy as np
from scipy.sparse import csr_matrix

rng = np.random.default_rng(42)
n_users, n_items, k = 500, 300, 8
U_true, V_true = rng.normal(size=(n_users, k)), rng.normal(size=(n_items, k))
mask = rng.random((n_users, n_items)) < 0.03                       # 3% observed
users, items = np.nonzero(mask)
ratings = np.sum(U_true[users] * V_true[items], axis=1) + rng.normal(0, 0.5, len(users))
R = csr_matrix((ratings, (users, items)), shape=(n_users, n_items))

order = rng.permutation(len(users))
test_idx = order[: len(order) // 10]
train = csr_matrix((ratings[order[len(order) // 10:]],
                    (users[order[len(order) // 10:]], items[order[len(order) // 10:]])), shape=R.shape)
Rt = train.T.tocsr()

def als(R, Rt, k=8, lam=0.5, iters=15):
    U = rng.normal(0, 0.1, (R.shape[0], k))
    V = rng.normal(0, 0.1, (R.shape[1], k))
    for _ in range(iters):
        for M, A, B in [(R, U, V), (Rt, V, U)]:                     # update users, then items
            for row in range(M.shape[0]):
                start, end = M.indptr[row], M.indptr[row + 1]
                cols, vals = M.indices[start:end], M.data[start:end]
                if len(cols) == 0:
                    continue
                Bc = B[cols]
                reg = lam * len(cols)                                # frequency-scaled regularization
                A[row] = np.linalg.solve(Bc.T @ Bc + reg * np.eye(k), Bc.T @ vals)
    return U, V

U, V = als(train, Rt)
pred = np.sum(U[users[test_idx]] * V[items[test_idx]], axis=1)
rmse = np.sqrt(np.mean((pred - ratings[test_idx]) ** 2))
baseline = np.sqrt(np.mean((train.data.mean() - ratings[test_idx]) ** 2))
print(f"density {mask.mean():.3%}, test RMSE {rmse:.3f} vs global-mean baseline {baseline:.3f}")
# ALS clearly beats the baseline; try lam values with cross-validation, since 3% density is easy to overfit

Each ALS update solves a small k × k system per user or item using only that row’s observed entries, so the cost scales with the number of observed ratings, not with the full matrix.

Common Pitfalls

  • Treating missing entries as zeros in explicit-rating models.

  • Random splits that leak future interactions; use time-based evaluation.

  • Weak regularization, which lets embeddings of rare users fit noise.

  • Measuring only RMSE when the product needs top-k ranking quality.

Interview Tips

  • Name matrix factorization, implicit-feedback losses and side features for cold start.

  • Mention regularization, sparse-aware optimizers, negative sampling and ranking metrics.

44

How would you optimize a model for fast inference on mobile devices?

Mobile inference is limited by latency, memory, battery and model download size, so the plan is to pick an efficient architecture, compress it, and run it with an on-device runtime that uses the phone’s accelerators, measuring on real devices throughout. Architecture: start from mobile-oriented models (MobileNetV3, EfficientNet-Lite, MobileViT, small distilled Transformers such as DistilBERT or MobileBERT), reduce input resolution or sequence length where accuracy allows, and use neural architecture search results tuned for latency. Compression: post-training quantization to int8 (about 4x smaller and often 2–3x faster, with a small calibration set; quantization-aware training if accuracy drops), float16 or int4 weight quantization for large models, structured pruning of channels or heads (unstructured sparsity rarely speeds up phones), knowledge distillation from a large teacher into a small student, and low-rank factorization. Runtime and deployment: export to TensorFlow Lite / LiteRT, Core ML, ONNX Runtime Mobile or ExecuTorch; use delegates for the GPU, NPU, Apple Neural Engine or DSP; fuse operations (convolution + batch norm + activation); avoid unsupported ops that fall back to the CPU; batch work and warm up the model; cache results; and consider splitting work between device and server. Always benchmark latency percentiles, memory and accuracy on target phones, because desktop timings don’t transfer.

Read full answer

Applies toPyTorch 2.x (similar workflows exist for TensorFlow Lite and Core ML)

Techniques

Technique Typical gain Cost
Efficient architecture Large Retraining
int8 quantization ~4x smaller, 2–3x faster Small accuracy loss, calibration
Quantization-aware training Recovers accuracy Extra training
Structured pruning Faster, smaller Fine-tuning
Distillation Small model, near-teacher accuracy Teacher + training
Operator fusion, delegates Lower latency Export/runtime work
Lower resolution/sequence length Large Possible accuracy loss

Code Example: Dynamic Quantization and Size/Latency Check

import io
import time

import torch
from torch import nn

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(512, 1024), nn.ReLU(), nn.Linear(1024, 1024), nn.ReLU(),
                      nn.Linear(1024, 10)).eval()
quantized = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)

def size_mb(m):
    buffer = io.BytesIO()
    torch.save(m.state_dict(), buffer)
    return buffer.getbuffer().nbytes / 1e6

def latency_ms(m, runs=200):
    x = torch.randn(1, 512)
    with torch.inference_mode():
        for _ in range(20):
            m(x)                                          # warm-up
        start = time.perf_counter()
        for _ in range(runs):
            m(x)
    return (time.perf_counter() - start) / runs * 1000

x = torch.randn(64, 512)
with torch.inference_mode():
    agreement = (model(x).argmax(1) == quantized(x).argmax(1)).float().mean().item()
print(f"size {size_mb(model):.1f} MB -> {size_mb(quantized):.1f} MB")
print(f"CPU latency {latency_ms(model):.3f} ms -> {latency_ms(quantized):.3f} ms, top-1 agreement {agreement:.2%}")
# Size drops by roughly 4x; latency improves on most CPUs; measure on the target phone before deciding

example = torch.randn(1, 512)
exported = torch.export.export(model, (example,))       # starting point for ExecuTorch or other runtimes
print(type(exported).__name__)
# ExportedProgram

Common Pitfalls

  • Benchmarking on a laptop and assuming phone performance.

  • Unstructured pruning that shrinks files but not latency.

  • Operations unsupported by the mobile accelerator, silently falling back to the CPU.

  • Quantizing without checking accuracy per class or on edge cases.

Interview Tips

  • Cover architecture, compression, runtime/accelerators and on-device measurement.

  • Quantify typical gains of quantization and mention distillation.

45

Outline a strategy for optimizing models in a distributed environment

A distributed training strategy starts by finding the bottleneck, then picks the simplest form of parallelism that removes it, and keeps optimization behaviour equivalent to a well-tuned single-machine run. If the model fits on one GPU and the problem is training time, use synchronous data parallelism (PyTorch DistributedDataParallel or Horovod): every worker holds the model, processes its shard of the global batch, and gradients are averaged with all-reduce, overlapped with the backward pass. If the model or its optimizer state doesn’t fit, shard it: FSDP or ZeRO (DeepSpeed) partition parameters, gradients and optimizer state across workers; tensor parallelism splits large layers; pipeline parallelism splits groups of layers with micro-batches; very large models combine all three. Keep the optimization sound: scale the learning rate with the global batch (linear rule for SGD, often square-root for Adam) with warmup, use large-batch optimizers (LARS, LAMB) when batches become huge, sync batch-norm statistics or use layer norm, and seed and shard the data loader so every example is seen once per epoch. Make it efficient: mixed precision, gradient accumulation, activation checkpointing, fast interconnects and local data caches, communication compression or local SGD on slow networks. Make it robust: checkpoint regularly (sharded checkpoints), handle worker failures and elastic scaling (torchrun), and monitor throughput, GPU utilization, loss curves and gradient norms. For hyperparameter search, run trials in parallel with early stopping (ASHA, Hyperband via Ray Tune or Optuna).

Read full answer

Applies toPyTorch 2.x (torch.distributed, FSDP), DeepSpeed, Ray

Decision Guide

Bottleneck Strategy
Training too slow, model fits Data parallel (DDP)
Optimizer state/parameters don’t fit FSDP / ZeRO stage 1–3
Single layers too large Tensor parallelism
Many layers, memory per GPU Pipeline parallelism
Slow network Gradient compression, local SGD, larger accumulation
Many hyperparameter trials Parallel search with early stopping

Code Example: DDP Training Skeleton (Launch with torchrun)

import os

import torch
import torch.distributed as dist
from torch import nn
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler, TensorDataset

def main():
    dist.init_process_group(backend="nccl" if torch.cuda.is_available() else "gloo")
    rank, world = dist.get_rank(), dist.get_world_size()
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    device = torch.device(f"cuda:{local_rank}" if torch.cuda.is_available() else "cpu")
    torch.manual_seed(42)

    data = TensorDataset(torch.randn(10_000, 32), torch.randn(10_000, 1))
    sampler = DistributedSampler(data, num_replicas=world, rank=rank, shuffle=True, seed=42)
    loader = DataLoader(data, batch_size=64, sampler=sampler)      # per-worker batch; global = 64 * world

    model = DDP(nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 1)).to(device),
                device_ids=[local_rank] if torch.cuda.is_available() else None)
    base_lr = 1e-3
    optimizer = torch.optim.AdamW(model.parameters(), lr=base_lr * world ** 0.5)   # square-root scaling for Adam
    loss_fn = nn.MSELoss()

    for epoch in range(3):
        sampler.set_epoch(epoch)                                    # different shuffle each epoch
        for xb, yb in loader:
            optimizer.zero_grad(set_to_none=True)
            loss = loss_fn(model(xb.to(device)), yb.to(device))
            loss.backward()                                         # gradients all-reduced here
            optimizer.step()
        if rank == 0:
            print(f"epoch {epoch}: loss {loss.item():.4f}")
            torch.save(model.module.state_dict(), "checkpoint.pt")
    dist.destroy_process_group()

if __name__ == "__main__":
    main()

# Launch on one machine with 4 processes:  torchrun --nproc_per_node=4 train_ddp.py

Common Pitfalls

  • Forgetting sampler.set_epoch, so every epoch uses the same order.

  • Keeping the single-GPU learning rate when the global batch grew several times.

  • Saving checkpoints from every rank, or not at all.

  • Data loading or network bandwidth, not GPUs, limiting throughput; profile before scaling.

Interview Tips

  • Start from the bottleneck and choose data, sharded, tensor or pipeline parallelism.

  • Mention learning-rate scaling, warmup, checkpointing and monitoring.

46

Explain natural gradient descent and its relevance in optimization

Natural gradient descent (Amari, 1998) changes the notion of “step size”: instead of moving a fixed distance in parameter space, it moves a fixed distance in the space of probability distributions the model represents, measured by the KL divergence, which leads to the update θ ← θ − η·F(θ)⁻¹∇L(θ), where F is the Fisher information matrix, F = E[∇log p(y|x,θ)·∇log p(y|x,θ)ᵀ]. This matters because ordinary gradients depend on how the model is parameterized: rescaling or reparameterizing a parameter changes the gradient and the path of gradient descent even though the model’s predictions are the same, while the natural gradient is invariant to such smooth reparameterizations and follows the steepest descent direction of the model’s output distribution. In practice it converges in fewer, better-conditioned steps on problems with very different sensitivities across parameters. Its cost is the Fisher matrix, d × d for d parameters, so it is used directly only for small models; scalable approximations include K-FAC (Kronecker-factored blocks per layer), diagonal approximations (connected to Adam’s second-moment scaling, though not identical), Shampoo, and conjugate-gradient solves with Fisher-vector products. It is central in reinforcement learning: natural policy gradient and TRPO constrain each policy update by a KL-divergence bound, and PPO’s clipping is a cheaper approximation of the same idea. For log-likelihood losses the Fisher equals the expected Hessian, linking natural gradient to Gauss-Newton methods.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Comparison

Gradient descent Natural gradient
Step measured in Parameter space (Euclidean) Distribution space (KL)
Update −η∇L −ηF⁻¹∇L
Reparameterization invariant No Yes (locally)
Cost O(d) O(d³) exact; approximations scale
Approximations — K-FAC, diagonal Fisher, Shampoo
Uses Everywhere RL (TRPO, NPG), second-order DL optimizers, variational inference

Code Example: Fitting a Gaussian’s Mean and Scale

import numpy as np

rng = np.random.default_rng(42)
data = rng.normal(5.0, 0.2, 5000)                 # true mean 5, true sd 0.2

def grad_nll(mu, log_sigma):
    sigma2 = np.exp(2 * log_sigma)
    g_mu = -np.mean(data - mu) / sigma2
    g_ls = 1 - np.mean((data - mu) ** 2) / sigma2
    return np.array([g_mu, g_ls])

def fisher(mu, log_sigma):
    return np.diag([np.exp(-2 * log_sigma), 2.0])  # Fisher for (mu, log sigma) of a Gaussian

def run(natural, lr, steps=200):
    theta = np.array([0.0, 0.0])                    # start: mean 0, sd 1
    for step in range(steps):
        g = grad_nll(*theta)
        update = np.linalg.solve(fisher(*theta), g) if natural else g
        theta = theta - lr * update
        if abs(theta[0] - data.mean()) < 1e-3 and abs(np.exp(theta[1]) - data.std()) < 1e-3:   # reached the MLE
            return step + 1, theta
    return steps, theta

steps_gd, theta_gd = run(natural=False, lr=0.01)
steps_ng, theta_ng = run(natural=True, lr=0.5)
print(f"gradient descent: {steps_gd} steps, mean {theta_gd[0]:.3f}, sd {np.exp(theta_gd[1]):.3f}")
print(f"natural gradient: {steps_ng} steps, mean {theta_ng[0]:.3f}, sd {np.exp(theta_ng[1]):.3f}")
# Plain gradient descent needs a small rate (the curvature in mu grows as sigma shrinks) and typically uses
# far more steps; the natural gradient rescales the mean update by sigma^2 and converges in a few dozen

For the mean, the natural-gradient step is σ²·∇μ = −(mean(data) − μ), a step toward the sample mean that doesn’t depend on the current scale; ordinary gradient descent’s step is divided by σ², so it explodes when σ is small or crawls when σ is large.

Common Pitfalls

  • Computing and inverting the full Fisher matrix for large networks.

  • Confusing the empirical Fisher (from observed labels) with the true Fisher (expectation over the model’s predictions).

  • Forgetting damping (F + λI) when F is ill-conditioned.

Interview Tips

  • Write the F⁻¹∇L update and explain KL-based steps and invariance.

  • Mention K-FAC, TRPO/natural policy gradient and the Gauss-Newton connection.

47

What is simulated annealing and how is it used in machine learning?

Simulated annealing (Kirkpatrick, Gelatt and Vecchi, 1983) is a probabilistic global optimization method inspired by cooling metal: from the current solution it proposes a random nearby solution, always accepts it if it is better, and accepts a worse one with probability exp(−Δ/T), where Δ is how much worse it is and T is a “temperature” that starts high and is gradually lowered. At high temperature the search wanders freely and can climb out of local minima; as T falls it becomes increasingly greedy and settles into a good minimum, and with a slow enough (logarithmic) cooling schedule it converges to a global optimum in probability, though practical schedules (geometric cooling, T ← αT with α ≈ 0.95–0.999) trade that guarantee for speed. It needs no gradients and works on discrete, combinatorial and noisy problems, which is where it is used in machine learning: feature subset selection, hyperparameter search over mixed or discrete spaces, neural architecture or rule search, clustering and graph partitioning, scheduling and routing problems inside ML systems, and as a component of other methods (annealed importance sampling, temperature annealing in MCMC and in Gumbel-softmax relaxations, and “annealed” learning-rate schedules inspired by it). For smooth, high-dimensional problems such as training neural networks, gradient methods are far more efficient.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+

Components

Component Choice
State A candidate solution (feature mask, permutation, parameters)
Neighbour move Flip one feature, swap two items, small random perturbation
Energy Objective to minimize (e.g. negative CV score)
Acceptance Better: always; worse: probability exp(−Δ/T)
Schedule T₀ high, geometric cooling, stop at T_min or no improvement

Code Example: Feature Selection with Simulated Annealing

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=600, n_features=25, n_informative=5, n_redundant=5, random_state=42)
rng = np.random.default_rng(42)
cache = {}

def energy(mask):
    key = mask.tobytes()
    if key not in cache:
        if not mask.any():
            cache[key] = 1.0
        else:
            score = cross_val_score(LogisticRegression(max_iter=1000), X[:, mask], y, cv=3).mean()
            cache[key] = -score + 0.002 * mask.sum()          # accuracy with a small size penalty
    return cache[key]

def simulated_annealing(n_features, t0=0.05, cooling=0.97, steps=400):
    state = rng.random(n_features) < 0.5
    e = energy(state)
    best, best_e, t = state.copy(), e, t0
    for _ in range(steps):
        candidate = state.copy()
        candidate[rng.integers(n_features)] ^= True           # flip one feature in or out
        e_new = energy(candidate)
        if e_new < e or rng.random() < np.exp(-(e_new - e) / t):
            state, e = candidate, e_new
            if e < best_e:
                best, best_e = state.copy(), e
        t = max(t * cooling, 1e-4)
    return best, best_e

mask, best_energy = simulated_annealing(X.shape[1])
all_features = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=3).mean()
print(f"selected {mask.sum()} of {X.shape[1]} features, CV accuracy {-best_energy + 0.002 * mask.sum():.3f} "
      f"(all features: {all_features:.3f})")
# Typically a much smaller subset with accuracy close to or above using all 25 features

Because the selection uses cross-validation scores on the same data many times, the chosen subset’s score is optimistic; confirm it on a held-out test set.

Common Pitfalls

  • Cooling too fast (it behaves like greedy search and gets stuck) or too slowly (wasted evaluations).

  • A starting temperature unrelated to typical Δ values; set it so early worse moves are often accepted.

  • Using it on smooth, high-dimensional problems where gradients are available.

  • Overfitting the validation score during search.

Interview Tips

  • Describe the acceptance rule, temperature and cooling schedule.

  • Give ML uses (feature selection, discrete hyperparameters) and when gradients are better.

48

Explain minimax optimization and its role in adversarial networks

A minimax problem optimizes one set of variables to minimize an objective that another set tries to maximize, min_θ max_φ V(θ, φ); unlike ordinary minimization, the goal is a saddle point (an equilibrium) where neither player can improve by changing only its own variables. Generative adversarial networks are the best-known ML example: the generator G (parameters θ) and discriminator D (parameters φ) play min_G max_D E_x[log D(x)] + E_z[log(1 − D(G(z)))], where the discriminator learns to tell real from generated samples and the generator learns to fool it; at the theoretical equilibrium the generator’s distribution matches the data and D outputs 1/2 everywhere. Other minimax problems in ML are adversarial training for robustness (min over model weights of the max over input perturbations of the loss), domain-adversarial training, robust optimization over uncertain distributions, and Lagrangian duals of constrained problems. They are harder to optimize than minimization: simultaneous or alternating gradient descent-ascent can cycle around the equilibrium or diverge, even on the bilinear toy problem V(x, y) = xy, and in GANs this appears as oscillation, mode collapse and vanishing generator gradients. Fixes include the non-saturating generator loss, Wasserstein objectives with gradient penalties (WGAN-GP), spectral normalization, two time-scale learning rates (TTUR), optimistic or extragradient methods that anticipate the opponent’s move, and averaging iterates.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Minimax in ML

Problem Min player Max player
GAN Generator Discriminator
Adversarial training Model weights Input perturbation
Domain adaptation (DANN) Feature extractor Domain classifier
Distributionally robust optimization Model Worst-case data distribution
Constrained optimization (Lagrangian) Primal variables Lagrange multipliers

Code Example: Gradient Descent-Ascent vs Extragradient on V(x, y) = x·y

import numpy as np

def grad(x, y):
    return y, x                      # dV/dx = y (x minimizes), dV/dy = x (y maximizes)

def simultaneous_gda(steps=200, lr=0.1):
    x, y = 1.0, 1.0
    for _ in range(steps):
        gx, gy = grad(x, y)
        x, y = x - lr * gx, y + lr * gy
    return np.hypot(x, y)

def extragradient(steps=200, lr=0.1):
    x, y = 1.0, 1.0
    for _ in range(steps):
        gx, gy = grad(x, y)
        x_half, y_half = x - lr * gx, y + lr * gy           # look-ahead step
        gx, gy = grad(x_half, y_half)
        x, y = x - lr * gx, y + lr * gy                     # update with the look-ahead gradient
    return np.hypot(x, y)

print(f"start distance from the equilibrium (0, 0): {np.hypot(1, 1):.3f}")
print(f"simultaneous GDA after 200 steps: {simultaneous_gda():.3f}")   # spirals outward
print(f"extragradient after 200 steps:    {extragradient():.3f}")      # spirals inward
# start 1.414; GDA grows by sqrt(1 + 0.01) per step to about 3.8; extragradient shrinks toward 0

For simultaneous gradient descent-ascent, each step multiplies the distance from (0, 0) by √(1 + η²), so with η = 0.1 it grows by a factor of 1.01¹⁰⁰ ≈ e^0.995 ≈ 2.7 over 200 steps (from 1.414 to about 3.8). Extragradient’s look-ahead turns the rotation into a contraction, by √(1 − η² + η⁴) ≈ 0.995 per step here.

Common Pitfalls

  • Treating a GAN’s falling generator loss as progress; the game’s losses aren’t minimized in the usual sense.

  • Equal learning rates for both players when one consistently dominates.

  • Ignoring cycling behaviour and stopping on an arbitrary iterate instead of averaging or using a quality metric (FID).

Interview Tips

  • Define the saddle-point goal and write the GAN objective.

  • Explain why gradient descent-ascent cycles and name stabilizing techniques.

49

How does stochastic optimization relate to reinforcement learning?

Reinforcement learning is a stochastic optimization problem: the agent wants parameters θ of a policy (or value function) that maximize the expected return J(θ) = E_{τ∼π_θ}[Σₜ γᵗ rₜ], an expectation over trajectories whose randomness comes from the policy’s own actions, the environment’s transitions and rewards, and the start state, so neither J nor its gradient can be computed exactly and must be estimated from sampled experience. Policy gradient methods use the likelihood-ratio (REINFORCE) estimator ∇J = E[Σₜ ∇log π_θ(aₜ|sₜ)·Ĝₜ], which is unbiased but has high variance, and the field’s main techniques are stochastic-optimization tools for reducing that variance: baselines and advantage functions, actor-critic methods that learn a value function, generalized advantage estimation, larger or parallel batches of rollouts. Value-based methods such as Q-learning and TD learning are stochastic approximation algorithms (in the Robbins-Monro sense) that move estimates toward noisy bootstrapped targets, with replay buffers and target networks to reduce correlation and instability. Off-policy learning uses importance sampling ratios, trust-region and clipped updates (TRPO, PPO) control the step size in a noisy setting, and exploration (entropy bonuses, ε-greedy) shapes which samples the gradient estimates see. Unlike supervised learning, the data distribution depends on the parameters being optimized, which makes RL a non-stationary stochastic optimization problem. Black-box methods such as evolution strategies and the cross-entropy method treat the return as a noisy function and optimize it without backpropagation.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x

Connections

Stochastic optimization idea In RL
Noisy gradient estimates Policy gradients from sampled episodes
Variance reduction Baselines, advantages, critics, GAE
Stochastic approximation TD learning, Q-learning
Importance sampling Off-policy correction
Step-size control Learning-rate schedules, trust regions (TRPO), clipping (PPO)
Derivative-free optimization Evolution strategies, cross-entropy method
Non-stationary objective Data depends on the current policy

Code Example: REINFORCE With and Without a Baseline

import numpy as np

true_means = np.array([1.0, 1.5, 0.5])               # expected reward per action

def run(baseline, steps=2000, lr=0.05, seed=42):
    rng = np.random.default_rng(seed)
    theta, b = np.zeros(3), 0.0
    grad_samples = []
    for _ in range(steps):
        p = np.exp(theta - theta.max())
        p /= p.sum()
        a = rng.choice(3, p=p)
        r = rng.normal(true_means[a], 1.0) + 10.0     # large constant offset makes variance obvious
        grad_log = -p
        grad_log[a] += 1
        advantage = r - b if baseline else r
        g = grad_log * advantage                      # unbiased stochastic gradient estimate
        grad_samples.append(g)
        theta += lr * g / 10
        if baseline:
            b += 0.05 * (r - b)                       # running average of rewards
    p = np.exp(theta - theta.max())
    p /= p.sum()
    return p, np.var(np.array(grad_samples), axis=0).sum()

for use_baseline in [False, True]:
    policy, variance = run(use_baseline)
    print(f"baseline={use_baseline!s:5}: P(best action) {policy[1]:.3f}, gradient-estimate variance {variance:.2f}")
# Both estimators are unbiased, but subtracting the baseline cuts the variance sharply
# and the policy concentrates on the best action (index 1) more reliably

Common Pitfalls

  • Treating RL losses like supervised losses; their value doesn’t track progress reliably.

  • Too large policy updates on noisy estimates, which collapse performance (use trust regions or clipping).

  • Ignoring that learning changes the data distribution (non-stationarity).

Interview Tips

  • Frame RL as maximizing an expectation with sampled gradients.

  • Name REINFORCE, baselines/critics, stochastic approximation and PPO/TRPO step control.

50

What are recent developments in optimization for large-scale ML?

Optimization for large-scale ML, driven by training models with billions of parameters on trillions of tokens, has advanced in optimizers, schedules, precision and systems. Optimizers: AdamW remains the default, with decoupled weight decay; Lion (sign-based, found by program search) and Sophia (light-weight second-order clipping) reduce memory or steps for some workloads; Shampoo and its distributed variants and SOAP bring preconditioning (Kronecker-factored second-order information) to large networks and have won optimizer benchmarks such as AlgoPerf; Muon orthogonalizes updates for matrix-shaped parameters using Newton-Schulz iterations and has been adopted in several 2025 large-model training runs; memory-light methods (Adafactor, 8-bit optimizer states, GaLore’s low-rank gradient projection) cut optimizer memory; and schedule-free optimizers remove the need for a predefined decay schedule. Schedules and scaling: warmup with cosine or “warmup-stable-decay” (WSD) schedules that make it easy to continue training, compute-optimal scaling laws (Chinchilla) for choosing model size versus data, and maximal-update parameterization (μP) so hyperparameters tuned on small models transfer to large ones. Precision and stability: bfloat16 and FP8 training with careful scaling, QK-layer-norm and z-loss to prevent attention-logit blowups, and gradient clipping and loss-spike recovery. Systems: sharded data parallelism (ZeRO, FSDP2), tensor, pipeline, sequence and expert parallelism for mixture-of-experts models, activation checkpointing, communication overlap and compression, and fused kernels (FlashAttention). Fine-tuning has its own efficient methods (LoRA and QLoRA), and for alignment, preference-optimization objectives such as DPO replace some reinforcement-learning steps.

Read full answer

Applies togeneral concepts as of 2026; examples with PyTorch 2.x

Developments at a Glance

Area Examples
Optimizers AdamW, Lion, Sophia, Shampoo/SOAP, Muon, schedule-free AdamW
Memory-efficient optimizers Adafactor, 8-bit Adam, GaLore
Schedules and scaling Warmup-stable-decay, μP hyperparameter transfer, scaling laws
Precision and stability bf16/FP8, QK-norm, z-loss, spike handling
Distributed systems FSDP/ZeRO, tensor/pipeline/sequence/expert parallelism
Efficient fine-tuning LoRA, QLoRA
Alignment objectives DPO and related preference losses

Code Example: A Warmup-Stable-Decay Schedule and the Lion Update Rule

import numpy as np

def wsd_schedule(step, total, peak=3e-4, warmup=0.02, decay=0.2, floor=0.1):
    """Linear warmup, long constant phase, then linear decay to floor * peak."""
    w, d = int(total * warmup), int(total * decay)
    if step < w:
        return peak * (step + 1) / w
    if step < total - d:
        return peak
    frac = (step - (total - d)) / d
    return peak * (1 - frac * (1 - floor))

total = 10_000
print([f"{wsd_schedule(s, total):.2e}" for s in [0, 199, 5_000, 8_000, 9_999]])
# ['1.50e-06', '3.00e-04', '3.00e-04', '3.00e-04', '3.01e-05']

def lion_step(theta, grad, m, lr=1e-4, beta1=0.9, beta2=0.99, weight_decay=0.1):
    """Lion (Chen et al., 2023): the update is the sign of an interpolated momentum."""
    update = np.sign(beta1 * m + (1 - beta1) * grad)
    theta = theta - lr * (update + weight_decay * theta)
    m = beta2 * m + (1 - beta2) * grad
    return theta, m

theta, m = np.array([1.0, -2.0, 0.5]), np.zeros(3)
theta, m = lion_step(theta, np.array([0.3, -0.01, 0.0]), m)
print(theta.round(6))
# [ 0.99989 -1.99988  0.499995]   every non-zero coordinate moves by exactly lr (plus weight decay)

The constant phase of a WSD schedule lets you branch off and decay from any checkpoint, which is convenient when the final training length isn’t known in advance.

Common Pitfalls

  • Adopting a new optimizer without retuning its learning rate and weight decay (Lion typically needs a 3–10x smaller learning rate than AdamW).

  • Comparing optimizers at a single budget or on one task.

  • Ignoring numerical stability issues that appear only at scale.

Interview Tips

  • Group advances into optimizers, schedules/scaling, precision/stability and systems.

  • Explain one in detail (AdamW vs Lion, μP, or ZeRO/FSDP) and stay cautious about benchmark claims.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Optimization answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects