What is logistic regression versus linear regression?
Logistic regression is a linear classification model that estimates the probability of a binary outcome and converts that probability into a class label using a threshold. It differs from linear regression, which predicts continuous values and minimizes squared error. Logistic regression is interpretable and well suited to linear decision boundaries, but it can struggle with nonlinear relationships, class imbalance, poorly separated classes, or features that are not scaled appropriately.
Read full answerHide full answer
Applies tomachine learning concepts; scikit-learn 1.7+ (Python 3.12)
How Logistic Regression Works
For each observation, logistic regression first calculates a linear score:
z = w · x + b
Here, x is the feature vector, w contains the model weights, · means dot product, and b is the intercept. The score is passed through the sigmoid function:
sigmoid(z) = 1 / (1 + e^(-z))
The resulting value p lies between 0 and 1 and represents the estimated probability that y = 1:
P(y = 1 | x) = p
A threshold then converts the probability into a class. A common default is 0.5, meaning that p >= 0.5 produces the positive class. Changing this threshold is useful when false positives and false negatives have different costs.
Because the model combines features linearly before applying the sigmoid, its decision boundary in feature space is linear.
Training and Loss
Logistic regression is usually trained by maximizing the likelihood of the observed labels. In practice, this is equivalent to minimizing log loss, also called binary cross-entropy:
log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))
In this formula:
-
nis the number of observations. -
yis the true binary label, either 0 or 1. -
pis the model’s predicted probability of class 1. -
logis the natural logarithm. -
The summation adds the loss from every observation.
For valid predictions, the loss strongly penalizes confident but incorrect predictions. Unlike squared-error linear regression, logistic loss accounts explicitly for probabilistic outputs. For data that are not perfectly separable, the unregularized logistic-regression objective is convex and has a global minimum. Under perfect separation, however, a finite minimizer may not exist, which is one reason regularization is important.
Interpreting Coefficients
A coefficient w_j represents the estimated change in log-odds when feature j increases by one unit, holding other features constant:
log(p / (1 - p)) = w · x + b
Exponentiating the coefficient gives the corresponding odds ratio:
odds_ratio = exp(w_j)
For example, if exp(w_j) = 2, a one-unit increase in that feature doubles the odds of belonging to the positive class. This interpretation applies directly to unregularized coefficients; regularization can shrink coefficients toward zero, and standardization changes the scale on which their effects are measured.
Logistic Regression vs. Linear Regression
| Aspect | Logistic regression | Linear regression |
|---|---|---|
| Purpose | Classification | Regression |
| Output | Probability from 0 to 1 | Unrestricted continuous value |
| Decision rule | Threshold on predicted probability | Direct numerical prediction |
| Typical loss | Log loss or binary cross-entropy | Mean squared error |
| Training objective | Likelihood maximization | Squared-error minimization |
| Decision boundary | Linear in the original feature space | Not primarily a classification model |
| Interpretation | Coefficients represent log-odds changes | Coefficients represent unit changes in the prediction |
| Main limitation | Limited nonlinear expressiveness | Sensitive to outliers and often sensitive to noise |
A linear regression model can produce values below 0 or above 1, making it inappropriate when the target is a probability. Logistic regression naturally produces bounded probabilities, although those probabilities are not automatically perfectly calibrated.
Multiclass and Regularization
For more than two classes, logistic regression can use:
-
One-vs-rest: Fit one binary classifier per class and select the class with the highest score.
-
Multinomial logistic regression: Fit one multiclass model using the softmax function, which generalizes the sigmoid across mutually exclusive classes.
In scikit-learn 1.5–1.6, the multi_class parameter was deprecated; it was removed in 1.7 and must not be set. The lbfgs solver uses a multinomial formulation for multiclass targets.
Regularization helps prevent overfitting:
-
penalty="l2": Shrinks coefficients toward zero and is the default. -
penalty="l1": Can produce sparse models by driving some coefficients to zero; supported byliblinearandsaga. -
penalty="elasticnet": Combines L1 and L2 penalties and requiressaga.
The C parameter is the inverse of regularization strength. Small C means stronger regularization, while large C means weaker regularization. Features should generally be standardized before fitting a regularized logistic regression model so that regularization is applied comparably across coefficients.
Common Pitfalls
-
Ignoring class imbalance: Accuracy can be misleading when one class dominates. Consider
class_weight="balanced"and evaluate precision, recall, F1, ROC-AUC, and PR-AUC. -
Failing to scale features: Standardization is especially important for strongly regularized or gradient-based models.
-
Assuming nonlinear patterns: A linear decision boundary may underfit nonlinear relationships unless meaningful nonlinear features are engineered.
-
Interpreting probabilities as perfectly calibrated: Compare predicted probabilities with observed event rates when calibration matters.
-
Overfitting: Keep regularization, use cross-validation, and avoid tuning repeatedly against the test set.
-
Perfect separation: If a feature perfectly predicts the outcome, coefficients can become extremely large. Stronger regularization and appropriate validation help control this problem.
-
Data leakage: Fit preprocessing only on training folds by using a scikit-learn
Pipeline.
Interviewers commonly ask about the sigmoid function, log loss, coefficient interpretation, odds ratios, the meaning of C, the difference between one-vs-rest and multinomial models, and how logistic regression handles imbalanced data.