What is optimization in the context of machine learning?
In machine learning, optimization is the process of finding the model parameters θ that minimize an objective function, usually the average loss over the training data plus a regularization term, J(θ) = (1/n) Σ L(f(xᵢ; θ), yᵢ) + λR(θ); training a linear regression, a logistic regression, a gradient-boosted model or a neural network is, at its core, solving such a minimization problem. The problem can be convex (linear and logistic regression, SVMs), where any local minimum is the global one and methods converge reliably, or non-convex (neural networks), where we settle for good local minima and must deal with saddle points and flat regions. Solvers range from closed-form solutions (the normal equations) and second-order methods (Newton, L-BFGS) for small smooth problems, to first-order stochastic methods (SGD, momentum, Adam) for large data, and constrained methods (Lagrange multipliers, KKT conditions) for problems like SVMs. A second layer of optimization tunes hyperparameters (learning rate, regularization strength, tree depth) using validation performance, with grid, random or Bayesian search. The goal isn’t just a low training loss: it is good performance on new data, which is why regularization, early stopping and validation are part of the process.
Read full answerHide full answer
Applies togeneral concepts; examples with NumPy 2.x and SciPy 1.13+
Ingredients
| Part | Example |
|---|---|
| Parameters θ | Weights and biases |
| Loss L | Squared error, cross-entropy, hinge |
| Regularizer R | L2 (‖θ‖²), L1 (‖θ‖₁) |
| Constraints | θ ≥ 0, margin constraints in SVMs |
| Algorithm | Gradient descent, Adam, L-BFGS, coordinate descent |
| Hyperparameters | Learning rate, λ, batch size |
| Evaluation | Validation loss/metric |
Code Example: The Same Problem Solved Three Ways
import numpy as np
from scipy.optimize import minimize
rng = np.random.default_rng(42)
X = np.column_stack([np.ones(200), rng.normal(size=(200, 2))])
true_w = np.array([1.0, 2.0, -3.0])
y = X @ true_w + rng.normal(0, 0.5, 200)
lam = 0.1
def objective(w):
return np.mean((X @ w - y) ** 2) + lam * np.sum(w[1:] ** 2) # loss + L2 penalty (no penalty on bias)
def gradient(w):
g = 2 * X.T @ (X @ w - y) / len(y)
g[1:] += 2 * lam * w[1:]
return g
penalty = np.diag([0.0, lam, lam]) * len(y)
w_closed = np.linalg.solve(X.T @ X + penalty, X.T @ y) # closed form (normal equations)
w_gd = np.zeros(3)
for _ in range(500): # gradient descent
w_gd -= 0.1 * gradient(w_gd)
result = minimize(objective, np.zeros(3), jac=gradient, method="L-BFGS-B") # quasi-Newton
print(w_closed.round(4), w_gd.round(4), result.x.round(4), result.nit)
print(np.allclose(w_closed, w_gd, atol=1e-4), np.allclose(w_closed, result.x, atol=1e-4))
# True True: all three reach the same minimum of this convex objective
The closed form multiplies the penalty by n because the objective uses the mean squared error; with that scaling, all three methods agree to four decimals.
Common Pitfalls
-
Treating a low training loss as success without validation.
-
Using a method that doesn’t fit the problem size (Newton on millions of parameters).
-
Forgetting that features need scaling for gradient methods to converge well.
Interview Tips
-
Define the objective as loss plus regularization over parameters.
-
Mention convex vs non-convex problems, solver families and hyperparameter optimization.