What is gradient descent in machine learning?
Gradient descent is an iterative optimization method that updates a model’s parameters in the direction that reduces its loss function. It solves the core optimization problem behind training many machine-learning models, trading slower convergence for simplicity and memory efficiency. The central trade-off is between update stability and training speed.
Read full answerHide full answer
Applies tomachine learning concepts (all versions)
Update Rule
For a parameter vector w and cost function J(w), gradient descent uses:
w_(t+1) = w_t - learning_rate * gradient(J(w_t))
-
w_t: parameter values at iterationt. -
J(w_t): cost or loss at those values. -
gradient(J(w_t)): vector of partial derivatives indicating how the cost changes as each parameter changes. -
learning_rate: positive scalar controlling the update size. -
w_(t+1): updated parameter values.
The gradient points toward the steepest local increase in cost, so subtracting it moves parameters toward a lower cost. For linear regression with mean squared error:
MSE = (1/n) * sum((y - y_hat)^2)
Here, n is the number of observations, y is the target, and y_hat is the prediction. The sum adds the squared prediction errors across all observations.
Gradient Descent Variants
| Variant | Cost per Update | Gradient Noise | Convergence | Memory |
|---|---|---|---|---|
| Batch | O(n*d) |
None; exact gradient | Smooth and stable | Uses all n examples per update |
| Stochastic | O(d) |
High | Fast per update but erratic | Uses one example per update |
| Mini-batch | O(b*d) |
Moderate | Common compromise | Uses b examples per update |
Here, n is the dataset size, d is the feature count, and b is the mini-batch size. One full-batch iteration costs O(n*d); a stochastic epoch also processes roughly n*d work overall.
Common Optimizers
| Optimizer | Intuition | Typical Use |
|---|---|---|
| Momentum | Accumulates previous gradients to damp oscillation and accelerate consistent movement | Non-steady gradients |
| Nesterov momentum | Evaluates the momentum-adjusted direction before applying the update | Often converges faster than basic momentum |
| AdaGrad | Accumulates squared gradients and gives frequently updated parameters smaller steps | Sparse features, but learning can decay excessively |
| RMSprop | Uses an exponential moving average of squared gradients | Recurrent networks and non-stationary objectives |
| Adam | Combines momentum with adaptive per-parameter step sizes and bias correction | Strong general-purpose default |
| AdamW | Applies weight decay directly rather than adding its gradient to the adaptive update | Weight decay in modern deep-learning models |
AdamW’s decoupled weight decay avoids adaptive scaling of the regularization term. Neither momentum nor adaptive methods eliminate the need to tune the learning rate.
Learning Rate and Convergence
-
Too large a learning rate: updates overshoot, causing oscillation or divergence.
-
Too small a learning rate: training is stable but slow and may appear stalled.
-
Step decay: reduces the rate after selected epochs.
-
Cosine decay: gradually lowers the rate along a smooth cosine curve.
-
Warmup: starts with a small rate and increases it during early training, which can stabilize large-batch or transformer training.
-
Learning-rate range test: increase the rate exponentially over a short run and select a point before the loss becomes unstable.
-
Line search: estimates a suitable step along the gradient direction for each update.
For convex objectives, gradient descent can converge toward a global minimum when the objective is strictly convex. For non-convex objectives, it may stop at a local minimum or saddle point. Feature scaling often improves conditioning and convergence speed, especially with gradient-based optimizers.
Deep networks can also suffer from vanishing gradients, where updates become extremely small, or exploding gradients, where updates become excessively large. Gradient clipping limits exploding updates, but it does not solve vanishing gradients.
Common Pitfalls
-
Data leakage: Fit scalers and other learned preprocessing only on training folds; use a scikit-learn
Pipelinefor production workflows. -
Poor conditioning: Standardize features when scales differ greatly, while checking whether the model truly requires it.
-
Wrong metric: For regression, use RMSE, MAE, or
R^2; classification accuracy can be misleading for imbalanced targets. -
Validation overfitting: Repeatedly tuning against one validation set leaks information; use
KFoldfor regression tuning. -
Missing values: Impute them before optimization, fitting the imputer on training data only.
-
Gradient checking: Compare analytically computed gradients with finite-difference approximations on a small example.
-
Ignoring resources: GPU acceleration helps with large matrix operations, but CPU execution may be more efficient for small datasets because of transfer overhead.
-
Not checking convergence: Plot the loss and monitor parameter norms to verify that optimization is progressing.