What is linear regression in predictive modeling?
Linear regression predicts a continuous target from one or more features by learning a linear relationship between them. It usually estimates coefficients using ordinary least squares (OLS), making it fast and highly interpretable. Its main trade-off is simplicity and interpretability versus sensitivity to outliers, correlated features, and violated assumptions.
Read full answerHide full answer
Applies toscikit-learn 1.5+ and statsmodels 0.14+ (Python 3.12)
Model and OLS Fitting
The model is:
y = w0 + w1*x1 + ... + wn*xn + error
-
y: observed target value. -
w0: intercept. -
wj: coefficient for featurexj. -
error: unobserved random disturbance in the data-generating process. -
n: number of features.
OLS chooses the coefficients that minimize the sum of squared residuals:
SSE = sum((y_i - y_hat_i)^2)
Here, i indexes observations, y_i is the observed target, and y_hat_i is its prediction. The residual is the observed value minus the predicted value; squaring penalizes large errors more heavily.
With an augmented design matrix X containing a column of ones for the intercept, target vector y, and coefficient vector w, the closed-form normal equation is:
w_hat = (X^T X)^-1 X^T y
The inverse exists only when X has full column rank, meaning there is no perfect multicollinearity. If the design matrix is rank-deficient, the coefficients are not uniquely identifiable and a pseudoinverse can provide one minimum-norm solution. For very large datasets, iterative methods such as gradient descent can be more practical because they avoid explicitly computing an inverse.
Predictive Modeling Workflow
-
Define the continuous target and useful predictors.
-
Split the data into training and test sets.
-
Fit the model on training data only.
-
Predict the target for unseen test observations.
-
Evaluate the predictions with regression metrics.
Common metrics include:
-
MAE: average absolute prediction error; easier to interpret in target units.
-
RMSE: penalizes large errors more strongly.
-
R²: proportion of target variance explained by the model.
Use cross-validation, such as KFold, when selecting models or hyperparameters. For time-dependent data, use TimeSeriesSplit and preserve chronological order to prevent leakage.
Assumptions and Diagnostics
OLS inference generally assumes:
-
Linearity: the relationship is adequately represented by the model.
-
Independent errors: residuals are not systematically related across observations.
-
Homoscedasticity: residual variance remains approximately constant.
-
Normally distributed errors: required primarily for conventional p-values and confidence intervals.
-
No perfect multicollinearity: one predictor is not an exact linear combination of others.
Residual-versus-fitted plots can reveal nonlinearity or changing variance. A Q–Q plot helps assess residual normality. Variance Inflation Factor (VIF) identifies multicollinearity; high values indicate that individual coefficients may be unstable.
Coefficients, Inference, and Model Extensions
A coefficient represents the expected change in y for a one-unit increase in one feature while holding all other features constant. It does not automatically imply causation.
For inferential questions, statsmodels provides OLS p-values and confidence intervals. A p-value tests whether a coefficient is zero under the model assumptions, while a confidence interval describes the estimated coefficient’s uncertainty. Adjusted R² penalizes additional predictors and is useful when comparing models with different numbers of features.
| Method | How it works | Main strength | Main weakness | Typical use |
|---|---|---|---|---|
| OLS | Minimizes squared residuals | Interpretable coefficients | Sensitive to outliers and collinearity | Baseline regression |
| Ridge | Adds an L2 penalty | Stabilizes correlated features | Does not normally set coefficients exactly to zero | Multicollinear data |
| Lasso | Adds an L1 penalty | Can set coefficients to zero | May select unstable features | Feature selection |
| ElasticNet | Combines L1 and L2 penalties | More flexible sparsity | Additional tuning | Correlated predictors |
| Huber regression | Reduces large residual influence | Robust to outliers | Less statistically efficient | Noisy data |
PolynomialFeatures can model curvature by adding squared and interaction terms, but increases overfitting risk. Dense OLS typically costs approximately O(n*p² + p³) time, while iterative solvers cost roughly O(epochs*n*p). Standard scikit-learn regression runs on CPU; large-scale gradient descent can use GPUs through frameworks such as PyTorch.
Common Pitfalls
-
Extrapolation: predictions outside the observed feature range may be unreliable.
-
Outliers and leverage: a few unusual observations can disproportionately change OLS coefficients.
-
Correlated predictors: coefficients may become unstable even when overall predictions remain accurate.
-
Causation errors: a fitted association does not prove that changing a feature causes the target to change.
-
Data leakage: fit preprocessing, imputation, scaling, and feature selection only on training data using an sklearn
Pipeline. -
Missing values: handle them inside the training pipeline rather than before splitting.
-
Overfitting: compare training and validation performance and avoid repeatedly tuning against the test set.
-
Misreading inference: p-values depend on OLS assumptions and do not measure effect size, practical importance, or causality.