How does a cost function work in machine learning?
A cost function (also called an objective function) measures how far a model’s predictions are from the true targets by aggregating a per-example loss over the dataset, often plus a regularization term. It turns training into an optimization problem: gradient-based algorithms adjust parameters to minimize this cost. The main trade-off is choosing a function that is smooth enough for stable optimization while reflecting which errors matter most, such as sensitivity to large outliers versus robustness.
Read full answerHide full answer
Applies tomachine learning concepts (examples in Python 3.12)
Loss, Cost, and Differentiability
-
Loss: The error for a single example, for example
loss_i = (y_i - y_hat_i)^2, wherey_iis the true target andy_hat_iis the model prediction for examplei. -
Cost / objective: The quantity optimized during training, typically the average loss over
nexamples plus an optional regularization term:J(w) = (1/n) * sum(loss_i) + regularization(w), wherewrepresents model parameters andnis the number of examples. -
How optimization works: During each training step, the model makes predictions, the cost is computed, and parameters are updated in the direction that reduces the cost. For gradient-based methods (gradient descent, Adam), the update is
w := w - lr * dJ/dw, wherelris the learning rate anddJ/dwis the gradient of the cost with respect tow. -
Why differentiable: If the cost is not differentiable (or provides uninformative gradients), gradient-based updates cannot be computed reliably, which slows or prevents learning.
Common Cost Functions
| Cost function | Formula (plain notation) | When to use | Notes | ||
|---|---|---|---|---|---|
| MSE (Mean Squared Error) | MSE = (1/n) * sum((y_i - y_hat_i)^2) |
Regression with roughly Gaussian errors | Penalizes large errors more because of squaring; sensitive to outliers. | ||
| MAE (Mean Absolute Error) | ‘MAE = (1/n) * sum( | y_i - y_hat_i | )‘ | Regression with outliers | Robust to outliers; gradient magnitude is constant away from zero. |
| Huber loss | Quadratic when ‘ | y_i - y_hat_i | <= delta, linear otherwise (delta` is a threshold) |
Regression needing a balance | Behaves like MSE near zero and MAE for large errors. |
| Binary cross-entropy (log loss) | BCE = -(1/n) * sum(y_i*log(p_i) + (1-y_i)*log(1-p_i)), where p_i is the predicted probability of class 1 |
Binary classification (often with sigmoid) | Logistic regression uses cross-entropy instead of MSE because, with a sigmoid link, the objective is convex in the weights and provides stronger, more useful gradients when predictions are confidently wrong. | ||
| Categorical cross-entropy | CCE = -(1/n) * sum(sum(y_{i,c} * log(p_{i,c}))) over classes c, with p_{i,c} from softmax |
Multi-class, one-label classification | y_{i,c} is 1 for the true class and 0 otherwise. |
||
| Hinge loss | Hinge = (1/n) * sum(max(0, 1 - y_i * s_i)), where y_i is in {-1, +1} and s_i is the decision score |
Support Vector Machines | Encourages a margin; zero loss once an example is correctly classified beyond the margin. |
| KL divergence | KL(P || Q) = sum(P(x) * log(P(x) / Q(x))) over outcomes x | Comparing distributions, distillation, variational models | Measures how distribution Q diverges from target distribution P. |
Regularized Cost Functions
-
L2 (Ridge / weight decay): Adds
lambda * sum(w_j^2)over weightsw_j. It shrinks weights smoothly and improves stability when features are correlated. -
L1 (Lasso): Adds
lambda * sum(|w_j|). It can drive some weights exactly to zero, producing sparse models useful for feature selection. -
lambda: The regularization strength; larger values penalize complexity more and can reduce overfitting but may underfit.
Common Pitfalls and Interview Tips
-
Numerical instability:
log(0)produces infinities in cross-entropy. Predictions should be clipped away from 0 and 1, or a stable logits-based loss such astorch.nn.BCEWithLogitsLossin PyTorch 2.x should be used. -
Training loss vs. evaluation metric: You may train on log loss but report F1, ROC-AUC, or PR-AUC for imbalanced classes. The metric should match the business decision, not just the optimizer.
-
Wrong cost for the problem: MSE on probabilities can optimize poorly for classification; MAE or Huber is often better than MSE when outliers dominate regression.
-
Preprocessing leakage: Fit scalers/encoders on training data only, typically inside an
sklearn.pipeline.Pipeline. -
Interview follow-ups: “Why is cross-entropy convex for logistic regression but MSE is not with a sigmoid?”, “How does L1 create sparsity?”, and “When would you choose Huber over MSE or MAE?”