Skip to content
Machine Learning & Data Science

Logistic Regression interview questions and answers

50 questions, 29 with code. About 5 h 25 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is logistic regression versus linear regression?

Logistic regression is a linear classification model that estimates the probability of a binary outcome and converts that probability into a class label using a threshold. It differs from linear regression, which predicts continuous values and minimizes squared error. Logistic regression is interpretable and well suited to linear decision boundaries, but it can struggle with nonlinear relationships, class imbalance, poorly separated classes, or features that are not scaled appropriately.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.7+ (Python 3.12)

How Logistic Regression Works

For each observation, logistic regression first calculates a linear score:

z = w · x + b

Here, x is the feature vector, w contains the model weights, · means dot product, and b is the intercept. The score is passed through the sigmoid function:

sigmoid(z) = 1 / (1 + e^(-z))

The resulting value p lies between 0 and 1 and represents the estimated probability that y = 1:

P(y = 1 | x) = p

A threshold then converts the probability into a class. A common default is 0.5, meaning that p >= 0.5 produces the positive class. Changing this threshold is useful when false positives and false negatives have different costs.

Because the model combines features linearly before applying the sigmoid, its decision boundary in feature space is linear.

Training and Loss

Logistic regression is usually trained by maximizing the likelihood of the observed labels. In practice, this is equivalent to minimizing log loss, also called binary cross-entropy:

log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

In this formula:

  • n is the number of observations.

  • y is the true binary label, either 0 or 1.

  • p is the model’s predicted probability of class 1.

  • log is the natural logarithm.

  • The summation adds the loss from every observation.

For valid predictions, the loss strongly penalizes confident but incorrect predictions. Unlike squared-error linear regression, logistic loss accounts explicitly for probabilistic outputs. For data that are not perfectly separable, the unregularized logistic-regression objective is convex and has a global minimum. Under perfect separation, however, a finite minimizer may not exist, which is one reason regularization is important.

Interpreting Coefficients

A coefficient w_j represents the estimated change in log-odds when feature j increases by one unit, holding other features constant:

log(p / (1 - p)) = w · x + b

Exponentiating the coefficient gives the corresponding odds ratio:

odds_ratio = exp(w_j)

For example, if exp(w_j) = 2, a one-unit increase in that feature doubles the odds of belonging to the positive class. This interpretation applies directly to unregularized coefficients; regularization can shrink coefficients toward zero, and standardization changes the scale on which their effects are measured.

Logistic Regression vs. Linear Regression

Aspect Logistic regression Linear regression
Purpose Classification Regression
Output Probability from 0 to 1 Unrestricted continuous value
Decision rule Threshold on predicted probability Direct numerical prediction
Typical loss Log loss or binary cross-entropy Mean squared error
Training objective Likelihood maximization Squared-error minimization
Decision boundary Linear in the original feature space Not primarily a classification model
Interpretation Coefficients represent log-odds changes Coefficients represent unit changes in the prediction
Main limitation Limited nonlinear expressiveness Sensitive to outliers and often sensitive to noise

A linear regression model can produce values below 0 or above 1, making it inappropriate when the target is a probability. Logistic regression naturally produces bounded probabilities, although those probabilities are not automatically perfectly calibrated.

Multiclass and Regularization

For more than two classes, logistic regression can use:

  • One-vs-rest: Fit one binary classifier per class and select the class with the highest score.

  • Multinomial logistic regression: Fit one multiclass model using the softmax function, which generalizes the sigmoid across mutually exclusive classes.

In scikit-learn 1.5–1.6, the multi_class parameter was deprecated; it was removed in 1.7 and must not be set. The lbfgs solver uses a multinomial formulation for multiclass targets.

Regularization helps prevent overfitting:

  • penalty="l2": Shrinks coefficients toward zero and is the default.

  • penalty="l1": Can produce sparse models by driving some coefficients to zero; supported by liblinear and saga.

  • penalty="elasticnet": Combines L1 and L2 penalties and requires saga.

The C parameter is the inverse of regularization strength. Small C means stronger regularization, while large C means weaker regularization. Features should generally be standardized before fitting a regularized logistic regression model so that regularization is applied comparably across coefficients.

Common Pitfalls

  • Ignoring class imbalance: Accuracy can be misleading when one class dominates. Consider class_weight="balanced" and evaluate precision, recall, F1, ROC-AUC, and PR-AUC.

  • Failing to scale features: Standardization is especially important for strongly regularized or gradient-based models.

  • Assuming nonlinear patterns: A linear decision boundary may underfit nonlinear relationships unless meaningful nonlinear features are engineered.

  • Interpreting probabilities as perfectly calibrated: Compare predicted probabilities with observed event rates when calibration matters.

  • Overfitting: Keep regularization, use cross-validation, and avoid tuning repeatedly against the test set.

  • Perfect separation: If a feature perfectly predicts the outcome, coefficients can become extremely large. Stronger regularization and appropriate validation help control this problem.

  • Data leakage: Fit preprocessing only on training folds by using a scikit-learn Pipeline.

Interviewers commonly ask about the sigmoid function, log loss, coefficient interpretation, odds ratios, the meaning of C, the difference between one-vs-rest and multinomial models, and how logistic regression handles imbalanced data.

2

What is the logit function in logistic regression?

The logit function is the natural logarithm of the odds, logit(p) = log(p / (1 - p)), and maps a probability strictly between 0 and 1 to any real number. Logistic regression applies its inverse, the sigmoid, to a linear score to obtain P(y = 1 | x), so the logit expresses the model’s score in log-odds. The main trade-off is an interpretable model with a linear decision boundary, but limited ability to represent nonlinear relationships without feature engineering.

Read full answer

Applies tologistic-regression concepts in all versions; scikit-learn 1.7+ (Python 3.12) for current solver and regularization details

Logit and Sigmoid

Let p be the probability that y = 1. The logit is:

logit(p) = log(p / (1 - p))

  • p is the predicted probability of the positive class.

  • 1 - p is the probability of the negative class.

  • p / (1 - p) is the odds of the positive class.

  • log is the natural logarithm.

The function is defined only for 0 < p < 1; its value is undefined at p = 0 and p = 1.

Logistic regression first calculates a linear score:

z = w · x + b

  • x is the vector of input features.

  • w contains the feature weights.

  • b is the intercept.

  • w · x is the dot product of the weights and features.

The sigmoid, or inverse logit, converts that score into a probability:

sigmoid(z) = 1 / (1 + e^(-z))

Here, e is the base of the natural exponential. A larger z produces a probability closer to 1, while a smaller z produces one closer to 0. The relationship is inverse:

logit(p) = z when p = sigmoid(z)

Thus, a positive score favors class 1, a negative score favors class 0, and a score of zero corresponds to probability 0.5.

Training Objective

Logistic regression estimates w and b by maximizing the likelihood of the observed labels. This is equivalent to minimizing binary cross-entropy, also called log loss:

LogLoss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

  • n is the number of training examples.

  • y is the observed binary label, either 0 or 1.

  • p is the predicted probability of class 1.

  • The sum runs over the training examples.

  • The first term penalizes low probabilities for positive examples.

  • The second term penalizes low probabilities for negative examples.

The logistic-regression log-loss objective is convex, so an appropriate optimizer can find a global optimum rather than becoming trapped in a local minimum. Adding the standard L1 or L2 regularization penalty also preserves convexity.

Coefficients and Decision Boundary

A coefficient β_j, one component of w, represents the change in log-odds associated with a one-unit increase in feature j, holding the other features constant. Its odds ratio is:

Odds ratio = exp(β_j)

For example, if exp(β_j) = 2, a one-unit increase doubles the estimated odds of the positive class. With standardized features, the coefficient describes the change associated with a one-standard-deviation increase.

Because the sigmoid is monotonic, thresholding its output produces a linear decision boundary in the original feature space. The boundary is defined by w · x + b = 0; changing the classification threshold moves it parallel to that boundary.

Multiclass and Regularization

Strategy How it works Main trade-off
One-vs-rest Fits one binary classifier for each class and combines their scores Straightforward, but classes are modeled separately
Multinomial Uses a softmax formulation to model mutually exclusive classes jointly Produces a joint probability model and requires a multinomial-capable solver

Current scikit-learn automatically uses the multinomial formulation for multiclass targets with the default lbfgs solver. The multi_class parameter was deprecated in scikit-learn 1.5-1.6 and removed in 1.7, so current code should omit it.

  • L2 regularization: The default penalty; discourages large coefficients and can reduce overfitting.

  • L1 regularization: Can set coefficients exactly to zero and is supported by the saga and liblinear solvers.

  • Elastic Net: Combines L1 and L2 penalties and is supported by saga.

  • C: The inverse of regularization strength. A larger C applies a weaker penalty, while a smaller C applies a stronger one.

Features should generally be standardized before fitting logistic regression, especially when using regularized optimization.

Common Pitfalls

  • Perfect separation: With very weak regularization, coefficients can become extremely large. A sensible C helps control them.

  • Class imbalance: Accuracy can be misleading when one class is rare. Consider precision, recall, F1, PR-AUC, and class_weight="balanced".

  • Nonlinear relationships: Logistic regression cannot directly learn arbitrary nonlinear boundaries; feature engineering, interactions, or a more flexible model may be needed.

  • Probability calibration: Predicted probabilities may become poorly calibrated under distribution shift, so evaluate calibration when probability quality matters.

  • Missing values: Impute missing values inside a pipeline fitted only on the training data.

  • Interview follow-ups: Be ready to distinguish log-odds from odds ratios, explain why the loss is convex, describe how C affects regularization, and clarify why a linear decision boundary does not imply linear probabilities.

3

How does logistic regression perform classification tasks?

Logistic regression is a linear classification model that estimates class probabilities and converts them into category predictions using a threshold. It is especially useful for binary tasks because its coefficients are interpretable, although it can represent only linear decision boundaries unless features are transformed. Its main trade-off is this simple, explainable structure versus the limited flexibility needed for nonlinear relationships.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For each observation, the model calculates a linear score:

z = w · x + b

  • x is the vector of input features.

  • w contains one weight per feature.

  • · denotes the dot product.

  • b is the intercept.

The score is passed through the sigmoid function:

sigmoid(z) = 1 / (1 + e^(-z))

Here, e is Euler’s number. The output, P(y = 1 | x), estimates the probability that the observation belongs to the positive class. In scikit-learn’s binary predict(), class 1 is selected when this probability is greater than 0.5; a different threshold can be chosen when false positives and false negatives have different costs.

Training and Coefficients

Logistic regression maximizes the likelihood of the observed labels. In practice, it minimizes binary cross-entropy, also called log loss:

log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

  • n is the number of observations.

  • y is the true binary label, either 0 or 1.

  • p is the predicted probability of class 1.

  • log is the natural logarithm.

This regularized objective is convex, so an optimizer such as L-BFGS can converge to its global optimum within numerical tolerance.

Each coefficient changes the feature’s log-odds for a one-unit increase, holding other features constant:

odds_ratio = e^(coefficient)

Thus, a coefficient of 0.7 multiplies the positive-class odds by approximately e^0.7, or 2.01.

Regularization controls overfitting:

  • C is the inverse of regularization strength; larger C means weaker regularization.

  • The default penalty="l2" shrinks coefficients toward zero.

  • penalty="l1" is supported by the liblinear and saga solvers and can produce sparse coefficients.

  • penalty="elasticnet" is supported by saga and combines L1 and L2 penalties.

  • Features should generally be scaled before regularized optimization.

Binary and Multiclass Classification

  • Binary classification: predict_proba() returns one probability column per learned class, ordered by model.classes_; predict() applies the decision threshold.

  • One-vs-rest multiclass: fits one binary classifier for each class and combines their decisions.

  • Multinomial multiclass: jointly fits one model whose class probabilities are normalized with the softmax function.

  • Recent scikit-learn versions use multinomial loss by default with the lbfgs solver for multiclass problems. The multi_class parameter is deprecated, so new code should not set it.

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import numpy as np

# Load a bundled binary-classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Reserve an untouched final test set.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

# Generate out-of-fold probabilities for threshold tuning.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]

thresholds = np.linspace(0.1, 0.9, 161)
validation_f1 = [
    f1_score(y_train, oof_probabilities > threshold, zero_division=0)
    for threshold in thresholds
]
best_threshold = thresholds[np.argmax(validation_f1)]

# Fit the final model on all training data.
model.fit(X_train, y_train)

test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities > best_threshold).astype(int)

precision_curve, recall_curve, _ = precision_recall_curve(
    y_test, test_probabilities
)

print(f"Selected threshold: {best_threshold:.2f}")
print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Test precision: {precision_score(y_test, test_predictions):.3f}")
print(f"Test recall: {recall_score(y_test, test_predictions):.3f}")
print(f"Test F1: {f1_score(y_test, test_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"Test PR-AUC: {auc(recall_curve, precision_curve):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions))
print("First three predicted probabilities:")
print(test_probabilities[:3])

Advantages and Limitations

Aspect Logistic regression
Decision boundary Linear in the original feature space
Interpretability Coefficients describe log-odds changes
Training Convex; roughly O(n*d) per optimizer pass
Prediction O(d) for a single observation
Feature scaling Recommended for regularized models
Nonlinear relationships Requires engineered features or transformations
Class imbalance Supports class_weight="balanced"
Large datasets Often scalable, although dense feature matrices can consume substantial memory

Common Pitfalls

  • Data leakage: Fit StandardScaler, imputers, and encoders inside a pipeline on training folds only.

  • Perfect separation: Coefficients can become very large; regularization and separation checks help.

  • Imbalanced classes: Accuracy can mislead. Inspect precision, recall, F1, ROC-AUC, and PR-AUC, especially when the positive class is rare.

  • Missing values: LogisticRegression does not accept NaN values directly; use an imputation step in the pipeline.

  • Overfitting: Use regularization and cross-validation, and do not repeatedly tune against the test set.

  • Poor calibration: High ROC-AUC does not guarantee accurate probabilities; evaluate calibration when probabilities drive decisions.

  • Nonlinear patterns: Add polynomial or interaction features, or consider a more flexible model.

4

Why does logistic regression use the sigmoid function for binary classification?

The sigmoid function converts a linear score into a probability between 0 and 1, giving binary logistic regression a natural probabilistic interpretation. It lets the model estimate P(y = 1 | x) while keeping the decision boundary linear in the original features. Its main trade-off is interpretability and efficient convex training versus an inability to represent nonlinear relationships without feature engineering.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5–1.7 (Python 3.12)

How the Sigmoid Works

The sigmoid function is:

σ(z) = 1 / (1 + e^(-z))

Here, z is a real-valued linear score, and e is the base of the natural logarithm. The function maps:

  • Large positive z to a probability close to 1

  • z = 0 to 0.5

  • Large negative z to a probability close to 0

In logistic regression, the linear score is:

z = w · x + b

In this formula, w contains the model’s coefficient weights, x is the input feature vector, · denotes a dot product, and b is the intercept. The sigmoid transforms that score into:

p = P(y = 1 | x) = σ(w · x + b)

This is the logit link used with the Bernoulli distribution: assuming the log-odds are linear gives p / (1 - p) = e^z, which solves to the sigmoid formula.

Why It Is Important

The sigmoid provides the probabilistic interpretation required for binary classification:

  • Probability prediction: The output estimates the probability of class 1 and can be used by probability-based metrics such as ROC-AUC and PR-AUC.

  • Flexible thresholding: A threshold such as 0.5 converts the probability into a class prediction. Changing the threshold affects the precision-recall trade-off without retraining the model.

  • Bounded and smooth output: Extreme scores produce probabilities near 0 or 1, while small changes around z = 0 have the greatest effect on the probability.

The sigmoid is nonlinear, but a fixed-probability threshold still produces a linear decision boundary:

w · x + b = 0

For example, assigning class 1 when p >= 0.5 is equivalent to assigning it when the linear score is nonnegative.

Training Objective

Logistic regression usually estimates its parameters by maximizing the likelihood of the observed binary labels. This is equivalent to minimizing binary cross-entropy, also called log loss:

L = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

In this formula:

  • n is the number of training examples.

  • y is the true label, either 0 or 1.

  • p is the predicted probability of class 1.

  • log is the natural logarithm.

  • The sum calculates the loss over all examples.

This objective is convex in the model coefficients, so it has no nonglobal local minima. Suitable optimizers seek a global optimum; under unregularized perfect separation, however, the coefficients can grow without bound while the loss approaches its infimum.

Coefficient Interpretation

Because:

log(p / (1 - p)) = w · x + b

the linear score is the log-odds of class 1. Holding other features constant, each coefficient represents the change in log-odds associated with a one-unit increase in that feature.

Exponentiating a coefficient gives the corresponding odds ratio:

odds ratio = e^(coefficient)

For example, a coefficient of 0.7 means a one-unit increase multiplies the estimated odds of class 1 by approximately e^0.7, or 2.01. This interpretation is valid per unit of the feature’s original scale; standardization is useful when comparing coefficient magnitudes and when regularization should penalize features on comparable scales.

Advantages and Limitations

Advantages Limitations
Produces class-probability estimates Does not automatically guarantee calibrated probabilities
Coefficients are easy to interpret Assumes a linear relationship between features and log-odds
Fast to train and predict Cannot model nonlinear boundaries without transformations or interaction terms
Supports regularization and class weighting Can produce unstable coefficients under perfect separation
Scales well to large datasets May underperform when relationships are highly nonlinear

Common Pitfalls

  • Ignoring nonlinear relationships: If the true boundary is curved, add suitable polynomial, interaction, or transformed features rather than assuming the sigmoid itself creates a nonlinear boundary.

  • Perfect separation: Completely separable data can drive coefficients toward very large values. Strong regularization or a more suitable model may be appropriate.

  • Class imbalance: Accuracy can be misleading when one class dominates. Evaluate precision, recall, F1, ROC-AUC, and PR-AUC, and consider scikit-learn’s class_weight="balanced".

  • Threshold confusion: The probability threshold is not fixed by the model. Select it using validation data when the costs of false positives and false negatives differ.

  • Poor calibration: A good ROC-AUC does not guarantee well-calibrated probabilities. Evaluate calibration when predicted probabilities will drive decisions.

  • Multiclass distinction: Binary logistic regression uses a sigmoid. In scikit-learn 1.5–1.7, multiclass LogisticRegression uses multinomial loss by default with solver="lbfgs", and the multi_class parameter is deprecated. One-vs-rest remains available through OneVsRestClassifier; it fits a binary sigmoid model for each class rather than using one shared softmax model.

5

How should logistic regression probability outputs be interpreted?

Logistic regression outputs estimate the conditional probability that an observation belongs to the positive class, given its feature values. The resulting score is useful because it can be interpreted both as a probability and as log-odds, but it represents model-based association rather than a causal effect or a guarantee about an individual outcome. Its main trade-off is an interpretable, linear decision boundary versus limited ability to represent nonlinear relationships without feature engineering.

Read full answer

Applies tologistic regression concepts (all versions); scikit-learn 1.5+ for the `class_weight="balanced"` option (Python 3.12)

How the Probability Is Produced

For observation (i), logistic regression first calculates a linear score:

z_i = β₀ + β₁x₁ᵢ + β₂x₂ᵢ + … + βₖxₖᵢ

  • (z_i): linear score for observation (i)

  • (β₀): intercept, or baseline log-odds

  • (β_j): coefficient associated with feature (j)

  • (x_{ji}): value of feature (j) for observation (i)

  • (k): number of features

The sigmoid then converts that score into a probability:

p_i = 1 / (1 + e^(-z_i))

  • (p_i): estimated probability of the positive class for observation (i)

  • (e): Euler’s mathematical constant

  • (z_i): the linear score defined above

Training chooses coefficients that maximize the likelihood of the observed binary labels, or equivalently minimize binary cross-entropy, subject to any selected regularization.

Meaning of the Probability

The output can be understood as an estimate of:

P(Y = 1 | X = x)

This means “the probability that the outcome is 1 given the observed features.” If the same feature pattern appeared many times in a representative population and the model were correctly specified, the average frequency of outcome 1 would approach the predicted probability.

This interpretation has important qualifications:

  • Population-dependent: The probability refers to the population and conditions represented by the training data.

  • Model-dependent: Predictions depend on the features, coefficient estimates, preprocessing, and assumptions of the model.

  • Not causal: P(Y = 1 | X = x) is not the same as P(Y = 1 | do(X = x)); the model’s prediction for a changed feature value is not evidence of a causal effect.

  • Not a certainty measure: A prediction of 0.90 does not mean the model is 90% confident that its overall analysis is correct.

  • Not uncertainty about the estimate: The probability value is a point prediction; it does not by itself include a confidence interval.

Probability Versus Log-Odds and Odds

The sigmoid is equivalent to the logit relationship:

log(p / (1 - p)) = z

The odds are p / (1 - p), and their natural logarithm is the linear score. Consequently, each coefficient represents a change in log-odds:

  • A coefficient of 0.7 means a one-unit increase in that feature, holding other features fixed, increases log-odds by 0.7.

  • The corresponding odds multiplier is exp(0.7), approximately 2.01.

  • A positive coefficient increases the estimated probability; a negative coefficient decreases it.

  • A coefficient does not imply that the probability changes by the same fixed amount for every starting probability.

These interpretations are conditional on the model and should not be treated as causal effects.

Class Decisions and Thresholds

A classifier usually converts the probability into a class using a threshold:

  • p >= 0.5: predict the positive class.

  • p < 0.5: predict the negative class.

The 0.5 threshold is conventional, not inherent to logistic regression. It should be selected using held-out validation data when false-positive and false-negative costs differ or operating conditions require a different positive-class priority. A change in target prevalence may also change the appropriate operating threshold.

Poor calibration is a separate issue: changing a threshold changes class decisions but does not repair the predicted probabilities themselves. Probability calibration should instead be evaluated and, if necessary, corrected using appropriate held-out data and calibration methods.

For multiclass logistic regression, softmax produces class probabilities that sum to one. In a one-vs-rest formulation, each class is modeled through a separate binary sigmoid; these binary probabilities should not automatically be interpreted as a jointly normalized multiclass distribution.

Calibration and Practical Limitations

A probability is calibrated when observations predicted with probability p have an observed event frequency close to p. For example, among cases assigned 0.80, roughly 80% should have outcome 1 in a representative sample.

Common issues include:

  • Perfect separation: Coefficients can become extremely large, producing probabilities near 0 or 1.

  • Class imbalance: A model may predict the majority class while assigning deceptively low minority-class probabilities; class_weight="balanced" can help the fitted objective but may alter calibration.

  • Misspecification: Missing nonlinearities or interactions can make scores poorly calibrated even when classification accuracy appears acceptable.

  • Distribution shift: Calibration can deteriorate when deployment data differs from training data.

  • Extrapolation: Linear scores can generate implausible probabilities when feature values fall outside the training range.

Probability calibration should therefore be checked on held-out data, using calibration curves and metrics such as Brier score or log loss.

Interview Tips

Clearly distinguish among a predicted class, its estimated probability, the calibration of that probability, and the causal effect of changing a feature. A probability can be useful for ranking observations even when it is not perfectly calibrated, but it should not be presented as certainty or as evidence of causation.

6

What assumptions does logistic regression require?

Logistic regression assumes that the logarithm of the outcome odds is a linear function of the predictors, allowing it to estimate probabilities for binary classification. Its main trade-off is an interpretable model with a linear decision boundary versus the risk of misspecification when the true relationship is nonlinear or observations are dependent.

Read full answer

Applies tologistic-regression theory (all versions) and scikit-learn 1.5+ (Python 3.12)

Model and Link Function

For observation i, logistic regression calculates a linear score:

z_i = w · x_i + b

It then converts that score into the predicted probability of the positive class:

p_i = 1 / (1 + e^(-z_i))

Here:

  • x_i is the vector of predictor values for observation i.

  • w contains the fitted coefficients, one per predictor.

  • b is the intercept.

  • · means the dot product of the coefficient and predictor vectors.

  • z_i is the linear score, also called the log-odds.

  • e is Euler’s number.

  • p_i is the estimated probability that y_i = 1.

Equivalently:

log(p_i / (1 - p_i)) = w · x_i + b

This is the logit link. The linearity assumption therefore applies to the log-odds, not directly to the probability. The predicted class is normally 1 when p_i exceeds a chosen threshold such as 0.5.

Holding the other predictors fixed, a one-unit increase in predictor j changes the log-odds by w_j and multiplies the odds by exp(w_j).

Core Statistical Assumptions

  • Binary outcome: Each observation is modeled as a Bernoulli trial with two possible outcomes. For example, a customer either purchases or does not purchase.

  • Independent observations: Outcomes must be independent conditional on the predictors. Violations can occur with repeated measurements, clustered customers, time-series observations, or family records.

  • Correct link-function specification: The included terms should capture the linear relationship between predictors and log-odds. Polynomial terms, interactions, splines, or transformations may be needed for nonlinear relationships.

  • Relevant predictors are included: Omitted variables can bias coefficients, especially when related to both predictors and outcome. This matters particularly when coefficients are interpreted causally.

  • No perfect separation: If a linear combination perfectly separates the classes, unregularized maximum-likelihood coefficients may become extremely large or fail to have finite estimates.

  • Sufficient overlap and sample size: Rare outcomes, severe imbalance, and small samples can destabilize estimates. Effective sample size for rare events depends on the number of positive outcomes, not only the number of rows.

  • Representative, reliable measurements: Incorrect labels, measurement error, and training data that do not represent deployment data can undermine the model regardless of whether the mathematical assumptions hold.

Logistic regression does not require predictor values to be normally distributed. It also does not require raw residuals to be normally distributed or to have constant variance in the same way ordinary least squares requires.

Training and Regularization

Logistic regression is normally trained by maximizing likelihood, equivalently by minimizing binary cross-entropy, also called log loss. For the standard formulation, this objective is convex, so it does not suffer from the multiple local minima found in many neural-network models.

In scikit-learn:

  • penalty="l2" is the default and shrinks coefficients toward zero.

  • penalty="l1" is supported by solvers such as saga and liblinear and can produce sparse coefficients.

  • penalty="elasticnet" is supported by saga and combines L1 and L2 shrinkage.

  • C is the inverse of regularization strength: larger C means weaker regularization, while smaller C means stronger regularization.

Features should generally be standardized before fitting a regularized logistic regression. With an unregularized model and an intercept, standardization does not change fitted probabilities, but it can improve numerical optimization and makes coefficient magnitudes easier to compare.

Multiclass Classification

For more than two classes, common strategies are:

Approach How It Works Key Consideration
One-vs-rest Fits one binary classifier per class Binary probabilities may not be mutually coherent, although scikit-learn normalizes them for predict_proba
Multinomial Fits one model using multiple linear scores and softmax Produces jointly modeled class probabilities that sum to one

The softmax formula is p_k = e^(z_k) / sum_j e^(z_j), where z_k is class k’s linear score, p_k is its probability, and j ranges over all classes.

Recent scikit-learn versions use multinomial loss by default for multiclass targets with solvers that support it, including lbfgs. The multi_class parameter was deprecated in scikit-learn 1.5 and is scheduled for removal, so leave it unset; use OneVsRestClassifier when you explicitly want one-vs-rest.

Common Pitfalls

  • Ignoring nonlinearity: A linear log-odds boundary may underfit interactions and curved relationships.

  • Using accuracy alone: On imbalanced data, a majority-class model can appear accurate while detecting almost no minority cases. Consider precision, recall, F1, ROC-AUC, and PR-AUC.

  • Using class_weight="balanced" indiscriminately: It can emphasize minority cases during training, but it does not replace suitable metrics, adequate overlap, or probability calibration.

  • Treating repeated observations as independent: Use grouped folds, mixed models, or generalized estimating equations when observations are clustered.

  • Leaking preprocessing information: Fit scaling, imputation, encoding, and feature selection only on training folds.

  • Overinterpreting coefficients: A coefficient describes an adjusted log-odds change, not automatically a causal effect.

  • Ignoring calibration: Good discrimination does not guarantee that predicted probabilities reflect real-world frequencies.

  • Missing perfect separation: Regularization helps, but extreme separation may still require stronger shrinkage or revised features.

Common interview follow-ups include: “How would you check the linearity assumption?”, “What changes if observations are grouped?”, “Why is accuracy misleading for imbalanced classes?”, and “How would you detect and address perfect separation?”

7

How does regularization make logistic regression select features?

Logistic regression performs embedded feature selection when trained with L1 regularization, because the L1 penalty can set some coefficients exactly to zero. Those features then contribute nothing to the fitted model and can be removed without a separate feature-selection step. The main trade-off is bias-variance: stronger regularization can reduce variance and overfitting, but too much can discard useful predictors and increase bias.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For binary classification, logistic regression calculates a linear score:

z_i = w · x_i + b

It then converts that score into a probability:

p_i = P(y = 1 | x_i) = 1 / (1 + e^(-z_i))

Here, i identifies an observation, x_i is its feature vector, w contains the feature coefficients, b is the intercept, and e is the base of the natural logarithm. The sigmoid maps any score to a probability between zero and one.

L1-regularized logistic regression conceptually minimizes:

L1 objective = log loss + λ * sum(j=1 to d) |w_j|

The binary log loss is:

log loss = -(1/n) * sum(i=1 to n) [y_i log(p_i) + (1-y_i) log(1-p_i)]

Here, n is the number of observations, d is the number of features, y_i is the binary target for observation i, p_i is its predicted probability of class 1, λ is the regularization strength, and w_j is the coefficient for feature j. The intercept b is not included in the penalty.

The absolute-value penalty has a sharp corner at zero, so the optimizer can place coefficients exactly at zero, up to numerical tolerance. An L2 penalty instead adds λ * sum(j=1 to d) w_j²; it shrinks coefficients smoothly but generally does not make them exactly zero.

Regularization Choices

Penalty How it works Main benefit Main weakness Typical use
penalty="l1" Adds an absolute-value penalty Produces sparse coefficients and removes some features Can discard useful predictors and be unstable with correlated features High-dimensional data where a sparse model is desirable
penalty="l2" Adds a squared-coefficient penalty Shrinks coefficients smoothly and is usually stable Does not directly eliminate features A strong general-purpose default
penalty="elasticnet" Combines L1 and L2 penalties Balances sparsity and coefficient stability Requires tuning of C and l1_ratio Data with correlated predictors and a need for sparsity

In scikit-learn, C is the inverse of regularization strength: a smaller C means stronger regularization. L1 is supported by the liblinear and saga solvers, while elasticnet requires saga. Feature scaling is important because otherwise the penalty implicitly treats features with large numeric ranges as more costly to retain.

A full-batch gradient evaluation generally costs O(n * d), although total runtime also depends on the solver and number of iterations.

Interpretation and Alternatives

  • In a standardized model, a coefficient describes the change in log-odds for a one-standard-deviation increase in that feature.

  • The exponential of a coefficient is its odds ratio. For example, exp(0.8) is approximately 2.22, so the odds are multiplied by approximately 2.22, or increase by about 122%.

  • Regularization does not perform statistical-significance testing. A feature may receive a zero coefficient because its contribution is redundant or because the penalty is too strong.

  • L1 selection can be unstable when predictors are highly correlated because the model may choose either member of a correlated group.

  • External methods such as recursive feature elimination can also select features, but their preprocessing and evaluation must be performed within each cross-validation fold to prevent leakage.

Multiclass and Alternatives

For multiclass classification, one-vs-rest fits one binary problem per class, whereas multinomial logistic regression fits a separate linear score for each class and turns the scores into probabilities with softmax. In scikit-learn, the lbfgs solver uses multinomial loss by default for multiclass targets, while liblinear uses one-vs-rest. The old multi_class parameter was deprecated in scikit-learn 1.5 and should not be relied on; leave it unset and select the behavior through the solver.

Because logistic regression has a linear decision boundary, feature engineering or interaction terms may be more appropriate than selection when important relationships are nonlinear.

Common Pitfalls

  • Skipping scaling: Fit StandardScaler inside a Pipeline so the L1 penalty compares coefficients on comparable scales.

  • Data leakage: Fit preprocessing and the model only on training data or training folds.

  • Excessive regularization: A very small C can remove genuinely predictive features. Compare selected-feature counts and cross-validated performance across several values of C.

  • Correlated predictors: Use stability across resamples or folds and consider elastic-net regularization.

  • Class imbalance: Consider class_weight="balanced" and evaluate precision, recall, F1, PR-AUC, and ROC-AUC; accuracy alone can be misleading.

  • Perfect separation: Separated data can produce very large coefficients and unstable estimates; stronger regularization or additional data may help.

  • Poor calibration: A model can rank cases well while producing unreliable probabilities, so evaluate calibration when probabilities drive decisions.

8

What are odds and odds ratios in logistic regression?

Odds express how likely a positive outcome is relative to a negative outcome, while an odds ratio (OR) compares two sets of odds. These quantities let logistic regression quantify how predictors change outcome likelihood on a multiplicative scale. The main trade-off is that odds ratios are statistically useful but can be less intuitive than probabilities or risk ratios, especially when the positive outcome is common.

Read full answer

Applies tologistic regression concepts; scikit-learn 1.7+ and statsmodels 0.14+ (Python 3.12)

Core Formulas

For a binary outcome, let:

  • y ∈ {0, 1} be the class label, where 1 denotes the positive class.

  • p = P(y = 1 | x) be the positive-class probability given predictors x; P denotes probability.

  • x = (x_1, ..., x_d) be the vector of d predictor values.

  • w = (w_1, ..., w_d) be the coefficient vector.

  • b be the intercept.

The odds of the positive outcome are:

odds = p / (1 - p)

An odds value of 2 means the positive outcome is twice as likely as the negative outcome. It does not mean its probability is twice as large.

Logistic regression calculates a linear score:

z = w · x + b

Here, · denotes the vector dot product, and z is the score, or predicted log-odds. The sigmoid converts it to probability:

p = 1 / (1 + e^(-z))

In this expression, e is the exponential base and the exponent is -z. Taking the logarithm of the odds recovers the linear relationship:

log(p / (1 - p)) = z = w · x + b

For two conditions A and B, the odds ratio is:

OR = odds_A / odds_B = [p_A / (1 - p_A)] / [p_B / (1 - p_B)]

Here, p_A and p_B are the positive-outcome probabilities under conditions A and B.

Interpreting Coefficients as Odds Ratios

For predictor x_j, coefficient w_j is the change in log-odds for a one-unit increase in x_j, holding other predictors constant; j indexes the predictor. Its corresponding odds ratio is:

OR_j = e^(w_j)

For example, if w_j = 0.7, then:

OR_j = e^0.7 ≈ 2.01

A one-unit increase therefore multiplies the positive-outcome odds by approximately 2.01. It does not necessarily double the probability.

For a binary categorical predictor, the odds ratio compares the encoded category with its reference category. For a continuous predictor, it is per unit of measurement, so that unit matters. If predictors are standardized, the odds ratio applies to a one-standard-deviation increase.

Worked Example

Suppose a churn model has a coefficient of 0.4 for annual contract length:

OR = e^0.4 ≈ 1.49

Holding other variables constant, each additional year multiplies the churn odds by approximately 1.49, or increases them by about 49%.

If the original churn probability is 0.20, its odds are:

0.20 / (1 - 0.20) = 0.25

Multiplying by 1.49 gives 0.3725. Converting back to probability:

p = 0.3725 / (1 + 0.3725) ≈ 0.271

Thus, the probability rises from 20% to about 27.1%; it does not rise by 49% in absolute probability terms.

Odds Ratio Versus Other Measures

Measure Interpretation Common mistake
Probability Chance that the outcome occurs Treating an absolute change as a relative change
Odds Positive outcome relative to the negative outcome Mistaking odds for probability
Odds ratio Multiplicative comparison of two odds Interpreting it as a probability ratio
Risk ratio Ratio of two outcome probabilities Treating it as an odds ratio

Odds ratios and risk ratios are closest when the outcome is rare. For common outcomes, an odds ratio can make the multiplicative effect on odds look larger than the corresponding effect on probability, so predicted probabilities should also be inspected.

Assumptions and Limitations

  • Conditional comparison: A predictor’s odds ratio compares values while other predictors remain unchanged.

  • Functional form: Logistic regression assumes a linear relationship between predictors and log-odds, not necessarily between predictors and probability.

  • Association, not causation: A fitted odds ratio does not establish that changing the predictor causes the outcome.

  • Separation risk: Perfect separation can produce extreme coefficients and misleadingly large odds ratios in unregularized models.

  • Scaling affects interpretation: Rescaling a predictor changes the meaning of “one unit” and therefore the numerical odds ratio.

  • Multiclass models: One-vs-rest fits one binary problem per class, whereas multinomial/softmax jointly models mutually exclusive classes. In scikit-learn 1.7+, the default lbfgs solver uses multinomial for multiclass; the former multi_class option was removed in 1.7 after being deprecated in 1.5. Exponentiating one multinomial coefficient is not generally a binary odds ratio; use an appropriate class-vs-class coefficient and intercept contrast.

  • Inference: Confidence intervals for odds ratios are obtained by exponentiating coefficient confidence intervals. For binary logistic regression, statsmodels.Logit provides coefficient tests, p-values, and confidence intervals.

Common Pitfalls

  • Confusing odds with probability: Doubling the odds does not double the probability.

  • Reporting only exponentiated coefficients: Include the corresponding confidence interval when presenting inferential estimates.

  • Ignoring effect size: For common outcomes, report predicted probabilities alongside the odds ratio.

  • Omitting the reference category: For categorical variables, state the baseline category.

  • Assuming causality: A significant predictor may reflect omitted variables or study-design bias.

  • Ignoring nonlinear log-odds: Use transformations or interactions when the relationship with log-odds, rather than probability itself, is nonlinear.

9

How do you interpret coefficients in a logistic regression model?

A logistic regression coefficient represents the change in the log-odds of the positive class for a one-unit increase in a feature, holding all other features constant. Logistic regression solves binary classification with an interpretable linear decision boundary, but it cannot naturally capture nonlinear relationships without feature engineering. Coefficients are on the log-odds scale, so exponentiating them gives odds ratios for practical interpretation.

Read full answer

Applies tologistic regression concepts; scikit-learn 1.5-1.6 (`multi_class` deprecated) and 1.7+ (`multi_class` removed), Python 3.12

How Coefficients Work

Logistic regression first calculates a linear score:

z = w · x + b

  • x is the vector of input features.

  • w contains one coefficient per feature.

  • b is the intercept.

  • · means dot product.

The score is converted into the probability of class 1 using the sigmoid:

p = 1 / (1 + e^(-z))

Its corresponding log-odds relationship is:

log(p / (1 - p)) = w · x + b

Here, p / (1 - p) is the odds of belonging to class 1, and log is the natural logarithm.

The model is trained by maximizing the likelihood of the observed labels, equivalently by minimizing binary log loss:

log loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the number of observations, y is an observed binary label, and p is its predicted probability. The log-loss objective is convex in the coefficients and intercept.

Interpreting Coefficients

  • Positive coefficient: Increasing the feature raises the predicted probability of class 1, holding all other features fixed.

  • Negative coefficient: Increasing the feature lowers the predicted probability of class 1.

  • Odds ratio: For feature j, exp(w_j) multiplies the odds of class 1 when x_j increases by one unit while other features remain fixed.

  • Example: If w_j = 0.7, the odds are multiplied by exp(0.7) ≈ 2.01 for a one-unit increase.

  • Intercept: The log-odds when every feature equals zero. This may not be meaningful or realistic, especially when features are standardized.

  • Magnitude comparison: Coefficients are most comparable when features use comparable scales. Standardization generally makes such comparisons more meaningful.

Multiclass Coefficients

Approach Coefficient interpretation
One-vs-rest Each coefficient describes the log-odds of one class against all remaining classes.
Multinomial softmax Each coefficient contributes to a class’s linear score; pairwise log-odds are obtained from differences between class coefficients.

In scikit-learn 1.5-1.6, multi_class="auto" selects multinomial logistic regression with the lbfgs solver, but the multi_class parameter is deprecated. It was removed in scikit-learn 1.7, where lbfgs continues to use the multinomial formulation for multiclass classification.

Practical Considerations

  • Regularization: The default penalty is "l2", which shrinks coefficients toward zero and generally makes fitted odds ratios less extreme. C is the inverse of regularization strength, so larger C means weaker regularization.

  • Penalty choices: The "l1" penalty requires the liblinear or saga solver, while "elasticnet" requires saga and also uses l1_ratio to control the mix of L1 and L2 penalties.

  • Scaling: Standardize features before fitting a regularized model, particularly when comparing coefficient magnitudes.

  • Inference: scikit-learn does not provide coefficient p-values or confidence intervals. Use an unregularized statsmodels.Logit model when inferential estimates are required.

  • Prediction: predict_proba returns class probabilities. For binary classification, predict selects the class with the largest probability, equivalent to a 0.5 threshold for the probability of class 1.

  • Causality: A coefficient describes an association in the fitted model; it does not establish that changing the feature would cause the outcome to change.

Common Pitfalls

  • Class imbalance: Accuracy can be misleading when classes have different frequencies. Inspect precision, recall, F1, PR-AUC, and ROC-AUC; class_weight="balanced" can increase the importance of minority-class errors during fitting.

  • Perfect separation: Unpenalized logistic regression can produce extremely large coefficients under separation. Penalized fitting can stabilize estimation, while statsmodels.Logit reports separation rather than regularizing the model.

  • Nonlinear relationships: If important relationships are nonlinear, add suitable transformed or interaction features rather than assuming the raw linear predictor is sufficient.

  • Calibration: A coefficient may be well estimated while predicted probabilities are poorly calibrated; inspect calibration plots and consider recalibration when probability quality matters.

  • Ignoring scale: Comparing raw coefficients from differently scaled features can be misleading.

  • Data leakage: Fit scalers, encoders, imputers, and feature selectors only on the training data or training folds.

10

How does maximum likelihood estimation apply to logistic regression?

Maximum likelihood estimation (MLE) chooses the logistic-regression parameters that assign the greatest likelihood to the observed labels. It turns logistic regression into an optimization problem over the coefficients and intercept. The main trade-off is an interpretable model with a linear decision boundary versus the need for feature engineering to represent nonlinear relationships.

Read full answer

Applies tomachine learning concepts (all versions); scikit-learn 1.5-1.7 for implementation notes

Maximum-Likelihood Objective

For binary labels, logistic regression computes a linear score:

z_i = w · x_i + b

  • i is the index of an observation.

  • x_i is the feature vector for observation i.

  • w contains the feature coefficients.

  • b is the intercept.

  • · denotes the dot product.

The sigmoid converts that score into the probability of the positive class:

p_i = sigmoid(z_i) = 1 / (1 + e^(-z_i))

Here, e is the base of the natural logarithm. Assuming the labels are independent conditional on their features, the likelihood is:

L(w, b) = product over i of [p_i^y_i * (1 - p_i)^(1 - y_i)]

  • L(w, b) is the likelihood of the observed labels under the parameters.

  • y_i is 1 for a positive observation and 0 for a negative observation.

  • The product combines the probability contribution from every observation.

MLE selects the parameters that maximize L(w, b). Because the logarithm is monotonic, maximizing the likelihood is equivalent to maximizing:

log-likelihood = sum over i of [y_i * log(p_i) + (1 - y_i) * log(1 - p_i))

Most software instead minimizes the negative log-likelihood:

NLL = sum over i of [-y_i * log(p_i) - (1 - y_i) * log(1 - p_i))

NLL is the negative log-likelihood, and log is the natural logarithm. This objective is also called binary cross-entropy or log loss.

Optimization and Decision Boundary

  • Convex objective: Log loss is convex in the coefficients and intercept, so an optimizer can find a global optimum when a finite optimum exists.

  • Gradient-based fitting: Solvers such as L-BFGS repeatedly update the parameters using gradients of the loss.

  • Linear boundary: A threshold of 0.5 is equivalent to z_i = 0, so the decision boundary is w · x_i + b = 0.

  • Parameter versus probability: MLE estimates w and b; the fitted sigmoid converts those estimates into class probabilities. A threshold then converts probabilities into class labels.

Coefficients and Regularization

A fitted coefficient β_j changes the log-odds by β_j when feature j increases by one unit while other features remain fixed. Therefore, exp(β_j) is the estimated multiplicative change in the odds for that one-unit increase.

Pure MLE does not regularize the model. For the default L2 penalty, scikit-learn minimizes an objective equivalent to:

NLL + (1 / (2C)) * ||w||_2^2

  • ||w||_2^2 is the sum of squared coefficients; the intercept b is not penalized.

  • C is the inverse of regularization strength, with a default value of 1.0.

  • Larger C means weaker regularization and behavior closer to unregularized MLE.

  • Smaller C means stronger shrinkage.

The default penalty is "l2". "l1" is supported with the liblinear or saga solver, while "elasticnet" requires saga. Scale features before fitting a regularized model so the penalty treats features consistently.

Multiclass Extension

Formulation How it works Strength Weakness Typical scale
One-vs-rest Fits one binary classifier for each class against the remaining classes Simple and independently parallelizable Does not model class relationships jointly K binary models for K classes
Multinomial Fits one joint softmax model Produces coherent joint class probabilities More coupled optimization One model with K class probabilities

For multinomial logistic regression, the probability of class k is:

p_k(x) = exp(z_k) / sum over j of exp(z_j)

Here, z_k is the score for class k, p_k(x) is its predicted probability, and j indexes all candidate classes.

In scikit-learn 1.5-1.7, the lbfgs solver uses multinomial softmax by default for multiclass targets. The multi_class option was deprecated in 1.5-1.6 and removed in 1.7, so new code should not set it; use OneVsRestClassifier(LogisticRegression()) when explicit one-vs-rest behavior is required.

Common Pitfalls

  • Perfect separation: A completely separable dataset can make the unregularized MLE coefficients diverge, so convexity does not guarantee a finite solution.

  • Data leakage: Fit preprocessing only on training data, preferably inside a Pipeline, and keep the final test set untouched until evaluation.

  • Class imbalance: Accuracy can mislead when classes have different frequencies. Inspect precision, recall, F1, ROC-AUC, and PR-AUC; consider class_weight="balanced".

  • Nonlinear relationships: Logistic regression cannot learn nonlinear boundaries directly. Add nonlinear features or interactions, or use a more flexible model.

  • Overfitting and threshold selection: Use regularization and cross-validation with StratifiedKFold. Select any probability threshold on validation data rather than the final test set.

  • Calibration: Model-based probabilities can be miscalibrated, especially under distribution shift or strong regularization. Check calibration on held-out data when reliable probabilities are required.

11

How should categorical features be encoded for logistic regression?

Categorical features in logistic regression are usually converted into numeric columns, primarily through one-hot encoding for nominal variables and ordinal encoding only when categories have a genuine order. The encoded features produce a linear score, z = w · x + b, which is passed through the sigmoid function to estimate P(y = 1 | x). One-hot encoding preserves unordered categories and is interpretable, but it can increase dimensionality; ordinal encoding is compact but may impose an artificial order.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Encoding Strategies

Category type Recommended approach Advantages Limitations
Unordered nominal One-hot encoding Preserves distinct categories without implying order Creates additional columns and may need regularization
Ordered categorical Ordinal encoding Compact and preserves the intended order Invalid when the numeric order is arbitrary
Binary categorical Map to 0 and 1 Simple and efficient Only appropriate for two categories
High-cardinality categorical Rare-category grouping or leakage-safe target encoding Controls feature width Requires careful cross-fitting to prevent leakage

For nominal variables, one-hot encoding is normally the safest default. A binary feature can also use one-hot encoding, although a single 0/1 column is more compact.

Model and Coefficient Interpretation

After preprocessing, logistic regression calculates:

z = w · x + b

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, x is the encoded feature vector, w contains one coefficient per feature, b is the intercept, z is the linear score, and the sigmoid converts that score into a probability between 0 and 1.

The model is trained by minimizing binary cross-entropy:

loss = -(1/n) * sum[y_i * log(p_i) + (1-y_i) * log(1-p_i)]

Here, n is the number of samples, y_i is the actual binary label, p_i is the predicted probability, and log is the natural logarithm. This objective is convex, so optimization can find a global minimum when it converges.

Each coefficient represents a change in log-odds for a one-unit increase in that feature, holding other features constant. Its exponential, exp(coefficient), is the corresponding odds multiplier. For standardized numeric features, “one unit” means one training-set standard deviation; for one-hot features using a dropped baseline, it means a change relative to that baseline.

For multiclass targets, the same encoding strategies apply. One-vs-rest fits one binary model per class, whereas multinomial logistic regression uses one softmax model across all classes. With the lbfgs solver, scikit-learn uses multinomial by default; the explicit multi_class parameter was deprecated in 1.5, so new code should rely on the solver default.

Code Example

This example uses a pipeline so preprocessing is fitted only on each training fold. Cross-validated out-of-fold predictions are used to tune the decision threshold, while the test set remains untouched.

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    balanced_accuracy_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)

# Create a small synthetic binary-classification dataset.
n = 1200
y = rng.binomial(1, 0.40, n)

education_probs = np.tile([0.25, 0.35, 0.40], (n, 1))
education_probs[y == 1] = [0.10, 0.35, 0.55]
education_labels = np.array(["high_school", "bachelor", "graduate"])
education = education_labels[np.argmax(education_probs, axis=1)]

X = pd.DataFrame({
    "income": rng.normal(50, 15, n) + 8 * y,
    "age": rng.normal(45, 12, n),
    "education": education,
    "region": rng.choice(["north", "south", "east", "west"], n),
})

# Hold out the test set before fitting preprocessing or the model.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

numeric_features = ["income", "age"]
categorical_features = ["education", "region"]

preprocessor = ColumnTransformer([
    ("numeric", StandardScaler(), numeric_features),
    (
        "categorical",
        OneHotEncoder(
            drop="first",
            handle_unknown="ignore",
            sparse_output=False,
        ),
        categorical_features,
    ),
])

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    (
        "classifier",
        LogisticRegression(max_iter=1000, penalty="l2"),
    ),
])

# Each fold fits preprocessing only on that fold's training data.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    pipeline,
    X_dev,
    y_dev,
    cv=cv,
    method="predict_proba",
)[:, 1]

# Select the threshold using out-of-fold predictions only.
thresholds = np.linspace(0.10, 0.90, 161)
f1_scores = [
    f1_score(y_dev, oof_probabilities >= threshold, zero_division=0)
    for threshold in thresholds
]
best_index = int(np.argmax(f1_scores))
tuned_threshold = thresholds[best_index]

# Fit the final pipeline on all development data.
pipeline.fit(X_dev, y_dev)

test_probabilities = pipeline.predict_proba(X_test)[:, 1]
test_predictions = test_probabilities >= tuned_threshold

precision_curve, recall_curve, _ = precision_recall_curve(
    y_test, test_probabilities
)
pr_auc = auc(recall_curve, precision_curve)

print(f"Tuned threshold: {tuned_threshold:.2f}")
print(f"Accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(
    "Balanced accuracy: "
    f"{balanced_accuracy_score(y_test, test_predictions):.3f}"
)
print(
    "Precision: "
    f"{precision_score(y_test, test_predictions, zero_division=0):.3f}"
)
print(
    "Recall: "
    f"{recall_score(y_test, test_predictions, zero_division=0):.3f}"
)
print(
    "F1: "
    f"{f1_score(y_test, test_predictions, zero_division=0):.3f}"
)
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"PR-AUC: {pr_auc:.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("Example predict_proba output:")
print(pipeline.predict_proba(X_test.iloc[:3]))

predict_proba returns one probability per class, ordered by pipeline.classes_, which is [0, 1] here. The second column is therefore the probability of the positive class.

Evaluation and Regularization

ROC-AUC measures ranking quality across all thresholds, so it is unaffected by the selected threshold. PR-AUC is useful when positive cases are relatively rare. Accuracy can be misleading with imbalanced classes; in that situation, also inspect precision, recall, F1, PR-AUC, and the confusion matrix.

Scikit-learn’s default l2 regularization helps control large coefficients and overfitting. Its C parameter is the inverse of regularization strength: increasing C reduces regularization, while decreasing C strengthens it. Feature scaling is recommended before fitting logistic regression, particularly when using regularization or gradient-based optimization.

Best Practices

  • Fit scaling, encoding, imputation, and feature selection only on training data.

  • Use Pipeline and ColumnTransformer to make leakage harder to introduce.

  • Keep a final test set separate from model and threshold selection.

  • Use stratified folds for classification so each fold preserves class proportions.

  • Set a fixed random seed for reproducibility.

  • Consider class_weight="balanced" for severe class imbalance, then evaluate with imbalance-aware metrics.

  • Check probability calibration if predicted probabilities drive business or medical decisions.

Common Pitfalls

  • Using ordinal encoding for unordered categories: This creates a false ordering and can substantially distort coefficients.

  • Encoding before splitting: Fitting encoders on the full dataset leaks test information.

  • Globally target encoding: This can expose validation labels; calculate encodings within training folds instead.

  • Ignoring rare or unseen categories: Use handle_unknown="ignore" or explicit rare-category grouping.

  • Perfect separation: Coefficients can become very large; regularization and checking for data issues are important.

  • Linear assumptions: Logistic regression cannot capture nonlinear interactions without feature engineering or more flexible models.

12

How does logistic regression handle multiclass classification?

Multiclass logistic regression assigns one of several classes by learning a linear score for each class and normalizing the scores into probabilities. Multinomial logistic regression uses softmax, while one-vs-rest trains one binary classifier per class. The main trade-off is an interpretable model with linear decision boundaries versus limited ability to represent nonlinear relationships without feature engineering.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Binary logistic regression computes a linear score and applies the sigmoid:

z = w · x + b

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, x is the feature vector, w is the weight vector, b is the intercept, z is the weighted sum, and e is Euler’s number.

For K classes, logistic regression computes one score per class:

z_k = w_k · x + b_k

Softmax then produces:

P(y = k | x) = e^(z_k) / sum(e^(z_j) for j = 1, ..., K))

Here, K is the number of classes, w_k and b_k are class k’s weights and intercept, z_k is its score, and j ranges over all classes. Prediction selects the class with the largest probability. Comparing two softmax probabilities produces a linear boundary, so the overall decision boundary is linear in the original features.

Training maximizes likelihood by minimizing cross-entropy:

L = -(1/n) * sum(log(P(y_i = k_i | x_i)))

Here, L is the loss, n is the number of observations, i indexes observations, x_i contains an observation’s features, k_i is its true class, and log is the natural logarithm. This objective is convex.

Multiclass Strategies

Approach How it works Strength Limitation Typical use
Multinomial softmax Learns one weight vector per class and normalizes all scores together Coherent class probabilities Requires solver support for multinomial optimization Preferred for genuine multiclass classification
One-vs-rest Trains one binary classifier for each class versus all others Simple; supports liblinear Independently fitted probabilities can conflict Legacy workflows or when liblinear is required

In scikit-learn 1.5-1.6, multi_class was deprecated and was removed in 1.7. With solver="lbfgs", multiclass fitting uses the multinomial formulation by default; solver="liblinear" uses one-vs-rest.

Coefficient Interpretation

  • Binary model: A coefficient changes the log-odds of y = 1 by that amount for a one-unit feature increase, holding other features constant. exp(coefficient) is the corresponding odds ratio.

  • Multinomial model: Softmax has no inherent reference class. If class r is selected as the reference, the log-odds of class k versus r change by coefficient[k, j] - coefficient[r, j] for a one-unit increase in feature j. The exponential of that difference is the corresponding odds ratio.

  • One-vs-rest model: Each binary classifier’s coefficient changes the log-odds of its class versus all remaining classes.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, average_precision_score, confusion_matrix, f1_score,
    precision_score, recall_score, roc_auc_score
)
from sklearn.model_selection import (
    StratifiedKFold, cross_val_predict, train_test_split
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, label_binarize

# Multiclass example using Iris.
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, solver="lbfgs")),
])
model.fit(X_train, y_train)

predictions = model.predict(X_test)
probabilities = model.predict_proba(X_test)
classes = model.named_steps["classifier"].classes_
binary_test = label_binarize(y_test, classes=classes)

print("Multiclass accuracy:", accuracy_score(y_test, predictions))
print("Multiclass macro ROC-AUC:", roc_auc_score(
    binary_test, probabilities, multi_class="ovr", average="macro"
))
print("Multiclass confusion matrix:\n", confusion_matrix(y_test, predictions))
print("First three probability rows:\n", probabilities[:3])

# Binary example: tune the threshold using out-of-fold training predictions.
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

binary_model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    binary_model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]

thresholds = np.linspace(0.20, 0.80, 61)
f1_values = [
    f1_score(y_train, oof_probabilities >= threshold, zero_division=0)
    for threshold in thresholds
]
best_threshold = thresholds[np.argmax(f1_values)]

binary_model.fit(X_train, y_train)
test_probabilities = binary_model.predict_proba(X_test)[:, 1]
binary_predictions = (test_probabilities >= best_threshold).astype(int)

print("Tuned threshold:", best_threshold)
print("Binary accuracy:", accuracy_score(y_test, binary_predictions))
print("Binary ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("Binary PR-AUC (average precision):",
      average_precision_score(y_test, test_probabilities))
print("Binary precision:", precision_score(y_test, binary_predictions))
print("Binary recall:", recall_score(y_test, binary_predictions))
print("Binary F1:", f1_score(y_test, binary_predictions))
print("Binary confusion matrix:\n", confusion_matrix(y_test, binary_predictions))

Common Pitfalls

  • Data leakage: Fit scaling and imputation only on training data; use a Pipeline so cross-validation does not leak preprocessing information.

  • Class imbalance: Accuracy can mislead. Consider class_weight="balanced" and evaluate precision, recall, F1, PR-AUC, and ROC-AUC.

  • Nonlinear relationships: Logistic regression cannot easily represent curved boundaries or interactions without feature engineering.

  • Perfect separation: It can produce very large coefficients; regularization and appropriate feature selection can help.

  • Regularization: C is the inverse of regularization strength, so smaller C means stronger regularization. l2 is the default; l1 supports liblinear or saga, while elasticnet requires saga. Scale features first.

  • Probability calibration: Predicted probabilities may be poorly calibrated, especially under imbalance or distribution shift; use held-out calibration data when reliable probabilities matter.

13

What happens with correlated predictors in logistic regression?

Multicollinearity occurs when predictor variables are strongly correlated, making it difficult to isolate each predictor’s individual effect. In logistic regression, it need not reduce predictive performance, but it can make coefficient estimates unstable, widen coefficient confidence intervals, and weaken significance tests. The main trade-off is that predictions may remain useful while individual coefficient interpretations become unreliable.

Read full answer

Applies tomachine learning concepts (Python 3.12)

Core Effect

Binary logistic regression calculates the probability of the positive class using:

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, z = b + w₁x₁ + w₂x₂ + ... + wₚxₚ, where x contains the predictor values, each wⱼ is the coefficient for predictor xⱼ, and b is the intercept. The decision boundary is z = 0, which is linear in the original predictors unless nonlinear terms are engineered.

The model is fitted by maximizing the Bernoulli likelihood, equivalently minimizing binary cross-entropy or log loss. This objective is convex in the coefficients for binary logistic regression. Multicollinearity can make optimization ill-conditioned and coefficient estimates sensitive, but it does not make the objective non-convex.

Coefficient Interpretation

The log-odds are defined as:

log-odds(y = 1 | x) = log(P(y = 1 | x) / P(y = 0 | x))

Each coefficient represents the change in log-odds for a one-unit increase in its predictor while the other predictors are held constant.

  • Positive coefficient: The predictor increases the estimated log-odds of the positive class.

  • Negative coefficient: The predictor decreases the estimated log-odds of the positive class.

  • Odds ratio: exp(wⱼ) is the estimated multiplicative change in odds for a one-unit increase in predictor xⱼ.

Under substantial multicollinearity, these interpretations become misleading because holding one correlated variable constant may represent an unrealistic scenario. The corresponding coefficient estimate is then based on an unstable counterfactual comparison that may be poorly supported by the observed data.

Statistical Consequences

  • Higher variance: Coefficient estimates can change considerably across samples or training folds.

  • Large standard errors: Confidence intervals for individual coefficients become wider, reducing confidence in each predictor’s estimated effect.

  • Reduced significance-test power: A predictor may fail to reach statistical significance even when its effect is practically important.

  • Sign instability: A coefficient may switch signs between samples even when predictive performance is similar.

  • Extreme coefficients: Near-perfect multicollinearity can produce very large coefficients in opposite directions.

  • Identifiability problems: With exact collinearity, the unregularized likelihood has no unique coefficient vector because redundant predictors allow the same fitted probabilities. A suitable penalty can produce a unique penalized optimum, but it cannot recover separately identifiable causal effects; numerical convergence can also remain problematic.

Multicollinearity generally increases estimation variance rather than inherently biasing the model’s overall relationship with the outcome.

Impact on Predictions

Multicollinearity does not automatically reduce accuracy, AUC, or other predictive metrics. If correlated variables carry similar information, the model can still make stable predictions, especially within the range of the training data.

Problems are more likely when:

  • Prediction is required for new combinations of correlated features.

  • The model is used to infer individual feature effects.

  • Coefficients are used to make policy or business decisions.

  • Extrapolation occurs beyond observed feature relationships.

  • Exact collinearity prevents a unique unregularized solution or causes numerical instability.

A model with poor individual coefficient inference can therefore still have good predictive performance. Conversely, removing correlated variables may improve interpretability without materially changing predictions.

Practical Remedies

  • Remove redundant variables: Prefer variables with clear business meaning and independent information.

  • Combine variables: A single summary measure, such as a ratio or index, may capture the shared concept.

  • Use regularization: L2 regularization stabilizes coefficients, although it does not make their separate causal interpretations reliable.

  • Standardize when appropriate: Scaling does not remove collinearity, but it improves numerical conditioning and makes a scale-sensitive penalty more comparable across features.

  • Use domain knowledge: Select variables based on the intended prediction setting rather than only automated significance tests.

  • Check conditioning: Use variance inflation factors or inspect correlations among numerical predictors, while recognizing that VIF is only a diagnostic.

  • Check dummy-variable dependencies: Avoid including redundant indicator variables, such as every category of a categorical feature together with an intercept.

  • Check stability: Compare coefficients across resamples or cross-validation folds rather than trusting a single fitted model.

Common Pitfalls

  • Assuming multicollinearity must reduce classification accuracy.

  • Removing predictors solely because their p-values are large.

  • Interpreting large coefficients in opposite directions as meaningful separate effects.

  • Standardizing correlated variables and assuming the problem is solved; scaling changes coefficient units but not correlation.

  • Evaluating only on training data, which can conceal instability across samples.

  • Ignoring exact collinearity, which can create non-unique unregularized solutions or convergence difficulties.

Interviewers commonly ask whether multicollinearity affects predictions or inference, how it changes coefficient variance, and how regularization or feature selection can address it.

14

How do L1 and L2 regularization work in logistic regression?

Regularization adds a penalty to logistic regression’s loss function to constrain coefficient magnitudes and reduce overfitting. L2 regularization shrinks coefficients toward zero, while L1 regularization can drive some coefficients exactly to zero, producing a sparse model. The main trade-off is between controlling model complexity and potentially underfitting the data.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12) and logistic-regression concepts

Objective Function

For each example i, logistic regression calculates a linear score and converts it into a probability:

z_i = w · x_i + b

p_i = 1 / (1 + e^(-z_i))

Here, x_i is the feature vector for example i, w contains the feature weights, b is the intercept, and p_i is the predicted probability that the example belongs to the positive class. Because p_i = 0.5 when z_i = 0, the decision boundary is linear in the original feature space.

Training maximizes the Bernoulli likelihood, which is equivalent to minimizing binary cross-entropy:

BCE = -(1/n) * sum_i[y_i * log(p_i) + (1 - y_i) * log(1 - p_i)]

Here, n is the number of examples, y_i is the binary target for example i, p_i is its predicted probability, and log denotes the natural logarithm.

Regularized training conceptually minimizes:

Objective = BCE + lambda * Penalty(w)

  • lambda: regularization strength; larger values impose a stronger penalty.

  • Penalty(w): either sum_j(abs(w_j)) for L1 or sum_j(w_j^2) for L2.

  • j: indexes the features.

  • w_j: the fitted coefficient for feature j.

The log-loss and both regularization penalties are convex, so the regularized optimization problem remains convex.

L1 and L2 Penalties

Penalty Effect on coefficients Typical use
L2 Shrinks coefficients continuously toward zero without normally making them exactly zero Stable, dense models and correlated features
L1 Can set some coefficients exactly to zero Feature selection and sparse models
Elastic Net Combines L1 sparsity with L2 shrinkage Correlated features with many predictors

L2 is often called ridge regularization, even in logistic regression. It reduces coefficient variance and sensitivity to noisy observations. L1 is often called lasso regularization; its sharp corner at zero allows the optimizer to select a subset of features. With highly correlated predictors, L1 may select one predictor arbitrarily and feature selection can therefore be unstable.

In scikit-learn, the intercept is not penalized by default, so the L1 or L2 penalty applies to feature coefficients rather than b.

Coefficient Interpretation

In both regularized and unregularized logistic regression, w_j represents the change in log-odds for a one-unit increase in feature j, holding the other features constant. Its corresponding fitted odds multiplier is exp(w_j).

Regularization changes the fitted coefficients by shrinking them toward zero and introducing bias in exchange for lower variance and improved generalization. Consequently, a zero L1 coefficient means that feature is excluded from the fitted linear predictor.

scikit-learn Behavior

  • LogisticRegression(penalty="l2") uses L2 regularization and is the default.

  • LogisticRegression(penalty="l1") requires a compatible solver such as liblinear or saga.

  • LogisticRegression(penalty="elasticnet", solver="saga") combines L1 and L2.

  • C is the inverse of regularization strength: larger C means weaker regularization, while smaller C means stronger regularization.

  • Feature scaling matters because the penalty applies directly to coefficient magnitude. Scaling ensures that regularization compares coefficients rather than being distorted by measurement units.

  • For multiclass classification, recent scikit-learn versions use multinomial loss with the lbfgs solver by default; the multi_class parameter is deprecated and should not be set.

Common Pitfalls

  • Data leakage: Fit StandardScaler only on training data and use a Pipeline so cross-validation fits preprocessing within each training fold.

  • Unscaled features: Regularization penalizes coefficients, so inconsistent units affect which features are shrunk most.

  • Perfect separation: Logistic regression can have extremely large or divergent coefficients. L2 usually controls this, but regularization does not ensure calibration.

  • Class imbalance: Consider class_weight="balanced" and evaluate with precision, recall, F1, PR-AUC, and ROC-AUC rather than accuracy alone.

  • Poor threshold selection: Tune classification thresholds on validation or out-of-fold predictions, not on the final test set.

  • Nonlinear relationships: Logistic regression still has a linear decision boundary, so nonlinear patterns may require feature engineering or a more flexible model.

  • Excessive regularization: A very small C may cause underfitting; tune C with cross-validation.

15

How do you assess logistic regression goodness-of-fit?

Goodness-of-fit for logistic regression is assessed by checking probability accuracy, discrimination, and classification errors, preferably using data not used for training. Log loss, deviance, and calibration evaluate predicted probabilities, while ROC-AUC and confusion matrices evaluate ranking and errors; the main trade-off is that strong discrimination does not guarantee calibrated probabilities.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Model and Fit Measures

For binary logistic regression, the linear score z = w · x + b is passed through the sigmoid, p = 1 / (1 + e^(-z)), to produce P(y = 1 | x). The decision boundary z = 0 is linear. Training maximizes likelihood, equivalently minimizing binary log loss, which is a convex optimization problem.

  • Coefficient interpretation: A coefficient is the change in log-odds for a one-unit predictor increase, holding other predictors fixed; exp(coefficient) is the corresponding odds ratio.

  • Multiclass: One-vs-rest fits separate binary models, whereas multinomial logistic regression uses softmax. Recent scikit-learn versions use multinomial by default with lbfgs; the multi_class parameter is deprecated.

  • Regularization: C is the inverse of regularization strength, so smaller C means stronger regularization. The default penalty is l2; l1 requires liblinear or saga, and elasticnet requires saga. Scale features before fitting a regularized model.

Log loss is:

log loss = -(1/n) * sum[y_i * log(p_i) + (1 - y_i) * log(1 - p_i)]

  • n: number of observations.

  • y_i: observed binary label.

  • p_i: predicted probability of class 1.

  • Lower log loss means better probabilistic predictions.

Deviance is -2 times the maximized log-likelihood. Comparing model deviance with the null deviance of an intercept-only model gives McFadden’s pseudo-R²:

pseudo-R^2 = 1 - (model deviance / null deviance)

Both deviances should use the same observations. This is descriptive, not a conventional proportion of variance explained. For nested models, a likelihood-ratio test, AIC, or BIC can assess whether added predictors justify their complexity.

Discrimination and Classification

A confusion matrix counts four outcomes:

  • True positive (TP): an actual positive predicted as positive.

  • False positive (FP): an actual negative predicted as positive.

  • False negative (FN): an actual positive predicted as negative.

  • True negative (TN): an actual negative predicted as negative.

Report metrics aligned with the costs of each error:

  • ROC-AUC: Ranking quality across thresholds; it can obscure poor minority-class performance.

  • Precision: Proportion of predicted positives that are correct.

  • Recall: Proportion of actual positives detected.

  • F1: Harmonic mean of precision and recall.

  • PR-AUC: More informative than ROC-AUC when the positive class is rare.

  • Accuracy: Overall fraction correct, but potentially misleading under class imbalance.

Choose a decision threshold using validation or out-of-fold predictions. The usual 0.5 threshold is not necessarily appropriate for the operational cost of false positives and false negatives.

Calibration

Calibration asks whether predicted probabilities match observed frequencies: among predictions near 70%, about 70% should be positive. Use a reliability diagram or calibration curve, and use the Brier score as a summary measure of probability accuracy. Avoid relying exclusively on the Hosmer–Lemeshow test because its result depends on grouping and has important large-sample limitations.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    brier_score_loss,
    confusion_matrix,
    f1_score,
    log_loss,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# Reserve an untouched test set.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

# Generate out-of-fold probabilities for threshold selection.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]

# F1 is illustrative here; use operational costs to choose the real metric.
thresholds = np.linspace(0.05, 0.95, 181)
validation_f1 = [
    f1_score(y_train, oof_probabilities >= threshold, zero_division=0)
    for threshold in thresholds
]
best_threshold = float(thresholds[np.argmax(validation_f1)])

# Fit preprocessing and the model on the complete training set.
model.fit(X_train, y_train)
test_probabilities = model.predict_proba(X_test)[:, 1]
tuned_predictions = test_probabilities >= best_threshold

precision_curve, recall_curve = precision_recall_curve(
    y_test, test_probabilities
)
pr_auc = auc(recall_curve[::-1], precision_curve[::-1])

print(f"Selected threshold: {best_threshold:.2f}")
print(f"Test accuracy: {accuracy_score(y_test, tuned_predictions):.3f}")
print(f"Test precision: {precision_score(y_test, tuned_predictions):.3f}")
print(f"Test recall: {recall_score(y_test, tuned_predictions):.3f}")
print(f"Test F1: {f1_score(y_test, tuned_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"Test PR-AUC: {pr_auc:.3f}")
print(f"Test log loss: {log_loss(y_test, test_probabilities):.3f}")
print(f"Test Brier score: {brier_score_loss(y_test, test_probabilities):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, tuned_predictions, labels=[0, 1]))

Common Pitfalls

  • Data leakage: Fit scaling and the model only on training data.

  • Imbalance: Report minority-class metrics and PR-AUC; class_weight="balanced" can help training but changes the effective weighting and may warrant recalibration.

  • Test-set tuning: Select features, hyperparameters, and thresholds without using the final test set.

  • Perfect separation: Coefficients can become extreme; use appropriate regularization and inspect coefficient magnitudes.

  • Functional form: Logistic regression assumes log-odds are linear in the transformed predictors, so nonlinear relationships may require feature engineering.

  • Confusing discrimination with calibration: Good ranking does not prove that probabilities are accurate.

Interview Tips

Expect follow-up questions about threshold selection, diagnosing poor calibration, detecting overfitting, and choosing logistic regression over a more flexible classifier. Emphasize leakage prevention, out-of-fold tuning, and metrics aligned with the relative costs of false positives and false negatives.

16

What are pseudo R-squared measures in logistic regression?

Pseudo-R² measures summarize how much a fitted logistic regression improves likelihood-based fit relative to a baseline model; they do not measure the proportion of outcome variance explained. Their main trade-off is that they are useful relative fit summaries but depend on the sample, baseline, and likelihood convention and do not establish predictive performance or calibration.

Read full answer

Applies tologistic regression concepts (all versions); scikit-learn 1.5+ for multiclass behavior

Model Context

Logistic regression estimates a binary outcome probability using:

P(y = 1 | x) = 1 / (1 + e^(-z)), where z = w · x + b

Here, y is the binary outcome, x contains the input features, w contains the fitted coefficients, b is the intercept, z is the linear score, e is the natural exponential base, and · denotes a dot product. Unregularized training maximizes the likelihood, equivalently minimizing binary cross-entropy or log loss; this objective is convex.

Holding the other features fixed, a coefficient w_i is the change in log-odds for a one-unit increase in feature i; exp(w_i) is the corresponding odds ratio.

For multiclass logistic regression, one-vs-rest fits one binary problem per class, while multinomial logistic regression uses softmax. With scikit-learn 1.5+, the lbfgs solver uses multinomial/softmax by default for multiclass; multi_class was deprecated in 1.5–1.6 and removed in 1.7.

In scikit-learn, C is the inverse of regularization strength, so a smaller C means a stronger penalty. The default penalty is "l2"; "l1" requires liblinear or saga, and "elasticnet" requires saga. Scale features before fitting regularized models.

Common Pseudo-R² Measures

Pseudo-R² measures compare the fitted model’s likelihood with a null model and, for some measures, a saturated model.

Measure Formula Interpretation
McFadden’s R² R² = 1 - ell_M / ell_0 Likelihood improvement over the null model
Cox–Snell R² R² = 1 - exp[(2/n)(ell_0 - ell_M)] Baseline-relative likelihood fit
Nagelkerke’s R² R² = R²_CS / {1 - exp[(2/n)(ell_0 - ell_sat)]} Cox–Snell adjusted toward a 0-to-1 range
Efron’s R² Related likelihood-based adjustment variants Alternative null- and saturated-model adjustments

Here, ell_M is the fitted model’s log-likelihood, ell_0 is the null model’s log-likelihood, ell_sat is the saturated model’s log-likelihood, and n is the number of observations. The null model usually contains only an intercept and predicts the overall event rate, while the saturated model reproduces every observed outcome. Exact implementations can differ, so state the formula and software convention when quoting values.

Interpreting the Values

  • A larger value generally means better likelihood fit relative to the same baseline and sample.

  • A value near zero indicates little improvement over the null model.

  • A negative value indicates that the fitted model has worse likelihood than the corresponding null model.

  • Under the standard formula, Nagelkerke’s measure is zero for the null model and one for the saturated model.

  • A value such as 0.60 does not mean that 60% of the variance in y is explained.

  • There is no universal cutoff for deciding that a logistic regression is “good.”

Reliability and Predictive Evaluation

Pseudo-R² is an in-sample fit summary rather than a significance test or a measure of unseen-data performance.

  • Sample dependence: Values can vary across samples, sample sizes, and class compositions; additional observations alone do not make pseudo-R² an evidence measure.

  • Baseline dependence: Values are comparable only when models use the same observations, null model, and likelihood convention.

  • Implementation differences: Some software reports binary log-likelihoods with different constant-term conventions, which can change pseudo-R² values.

  • Class weighting: Weighted likelihoods and baseline models must be handled consistently; otherwise comparisons can be misleading.

  • Perfect separation: Coefficients may become extremely large, and maximum-likelihood estimation may fail to converge.

  • Nested models: For nested unpenalized models fitted to the same observations, a likelihood-ratio test is generally more informative than comparing pseudo-R² values.

Predictive evaluation should use held-out data and appropriate metrics: log loss or Brier score for probability accuracy, calibration plots for calibration, ROC-AUC for ranking, and PR-AUC for rare positive classes. Precision, recall, F1, and confusion matrices describe classification trade-offs at a selected threshold. Accuracy can mislead with imbalanced data because a majority-class-only model can score highly while missing most positive cases.

Common Pitfalls

  • Calling pseudo-R² an explained-variance measure.

  • Comparing values from different samples, baselines, or software conventions.

  • Treating a larger pseudo-R² as proof of better real-world performance.

  • Using pseudo-R² instead of a likelihood-ratio test for valid nested-model comparisons.

  • Ignoring class imbalance; consider class_weight="balanced", PR-AUC, precision, and recall.

  • Assuming linear predictors capture nonlinear relationships without feature engineering.

  • Failing to scale features before regularized logistic regression.

  • Tuning the decision threshold on test data or ignoring probability calibration.

18

How do you evaluate logistic regression performance?

Logistic regression is evaluated on held-out data by checking both its probability quality and the classification decisions made at a chosen threshold. The main trade-off is that ROC-AUC measures ranking across thresholds, whereas precision, recall, F1, and accuracy depend on the operating threshold and the relative costs of false positives and false negatives.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Logistic Regression Is Evaluated

For observation i, logistic regression computes a linear score and converts it to a probability:

z_i = w · x_i + b

p_i = 1 / (1 + e^(-z_i))

  • z_i is the linear score for observation i.

  • w contains the feature weights, and x_i is the observation’s feature vector.

  • b is the intercept.

  • p_i is the predicted probability of class 1.

Training minimizes binary cross-entropy, also called log loss:

log_loss = -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i))

Here, n is the number of observations, y_i is the true binary label, p_i is its predicted probability, and log is the natural logarithm. This objective is convex, so optimization does not encounter the non-convex local minima possible in some other models. Because the sigmoid is monotonic, logistic regression produces a linear decision boundary unless nonlinear features are added.

Metrics and Their Meaning

Metric What it measures Important consideration
Accuracy Fraction of correct predictions Can be misleading with imbalanced classes
Precision Fraction of predicted positives that are correct Useful when false positives are costly
Recall Fraction of actual positives detected Useful when false negatives are costly
F1 score Harmonic mean of precision and recall Balances the two errors
ROC-AUC Ranking quality across thresholds Can appear optimistic under severe imbalance
PR-AUC Precision-recall trade-off across thresholds Often more informative for imbalanced classes
Log loss Quality of predicted probabilities Strongly penalizes confident errors
Brier score Mean squared probability error Measures probability calibration

A confusion matrix should always specify class order because that determines the meaning of its cells. Compare classification metrics with a majority-class baseline, because an apparently useful accuracy may merely reflect class imbalance.

Threshold Selection and Coefficients

predict_proba returns class probabilities. Although 0.5 is a common starting threshold, choose it from validation data according to the application:

  • Maximize recall when missing a positive is expensive.

  • Maximize precision when acting on a false positive is expensive.

  • Maximize F1 when both errors matter and no direct costs are available.

Select the threshold using validation predictions, then evaluate the chosen threshold once on untouched test data. A coefficient is the change in log-odds for a one-unit feature increase, holding other features constant; exp(coefficient) is the corresponding odds ratio. With standardized features, one unit means one training-set standard deviation.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, average_precision_score, brier_score_loss,
    confusion_matrix, f1_score, log_loss, precision_score,
    recall_score, roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold, cross_validate, train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# Separate training, threshold-validation, and final test data.
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, random_state=42, stratify=y_temp
)

# The pipeline fits the scaler only on each training fold.
model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    model, X_train, y_train, cv=cv,
    scoring={"roc_auc": "roc_auc", "pr_auc": "average_precision"},
    n_jobs=-1,
)
print(f"CV ROC-AUC: {cv_results['test_roc_auc'].mean():.3f}")
print(f"CV PR-AUC: {cv_results['test_pr_auc'].mean():.3f}")

model.fit(X_train, y_train)

# Select an F1-maximizing threshold using validation data only.
val_probabilities = model.predict_proba(X_val)[:, 1]
thresholds = np.linspace(0.10, 0.90, 161)
f1_scores = [
    f1_score(
        y_val,
        (val_probabilities >= threshold).astype(int),
        zero_division=0,
    )
    for threshold in thresholds
]
best_threshold = thresholds[int(np.argmax(f1_scores))]

# Evaluate once on untouched test data.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

results = {
    "accuracy": accuracy_score(y_test, test_predictions),
    "precision": precision_score(y_test, test_predictions, zero_division=0),
    "recall": recall_score(y_test, test_predictions, zero_division=0),
    "f1": f1_score(y_test, test_predictions, zero_division=0),
    "roc_auc": roc_auc_score(y_test, test_probabilities),
    "pr_auc": average_precision_score(y_test, test_probabilities),
    "log_loss": log_loss(y_test, test_probabilities, labels=[0, 1]),
    "brier_score": brier_score_loss(y_test, test_probabilities),
}
print(f"Selected threshold: {best_threshold:.3f}")
print("First five test probabilities:", test_probabilities[:5])
for name, value in results.items():
    print(f"{name}: {value:.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))

Common Pitfalls

  • Data leakage: Fit scalers, imputers, and encoders only on training folds; pipelines prevent this.

  • Using accuracy alone: Report class-sensitive metrics and a majority-class baseline.

  • Tuning on the test set: Select model parameters and thresholds using training and validation data.

  • Confusing ranking with calibration: Excellent ROC-AUC does not guarantee well-calibrated probabilities; log loss, Brier score, and calibration plots can help assess this.

  • Class imbalance: class_weight="balanced" may be appropriate for some problems, but it changes the effective prior and can distort probabilities, so recalibrate when needed.

  • Perfect separation: It can produce very large coefficients; inspect them and consider stronger regularization.

  • Nonlinear relationships: Add interactions, basis expansions, or other engineered features when needed.

  • Multiclass evaluation: Recent scikit-learn versions use multinomial loss by default with the lbfgs solver; the multi_class parameter is deprecated.

19

What is a confusion matrix, and how is it interpreted?

A confusion matrix is a table that compares predicted classes with the true classes, showing which predictions were correct and which types of errors occurred. It helps diagnose model behavior when a single aggregate score, such as accuracy, is insufficient. The main trade-off is that it provides detailed error information but requires labeled data and depends on the chosen decision threshold.

Read full answer

Applies tomachine learning concepts

Confusion Matrix Structure

For binary classification, assume class 1 is the positive class:

Actual Predicted Predicted 0 Predicted 1
Actual 0 TN FP
Actual 1 FN TP
  • True Positive (TP): A positive instance was correctly predicted as positive.

  • True Negative (TN): A negative instance was correctly predicted as negative.

  • False Positive (FP): A negative instance was incorrectly predicted as positive. This is commonly called a Type I error.

  • False Negative (FN): A positive instance was incorrectly predicted as negative. This is commonly called a Type II error.

The meaning of “positive” depends on the domain. In medical screening, for example, the positive class might represent the presence of a disease.

Derived Performance Measures

The confusion matrix supports several metrics:

  • Accuracy = (TP + TN) / (TP + TN + FP + FN)
    The proportion of all predictions that are correct. It can be misleading when classes are imbalanced.

  • Precision = TP / (TP + FP)
    The proportion of positive predictions that are actually positive. High precision means few false alarms.

  • Recall, or sensitivity = TP / (TP + FN)
    The proportion of actual positives that the model detects. High recall means few missed positives.

  • Specificity = TN / (TN + FP)
    The proportion of actual negatives that the model correctly identifies.

  • F1 score = 2 * (precision * recall) / (precision + recall)
    A harmonic mean of precision and recall, useful when both false positives and false negatives matter.

A threshold of 0.5 is common, but it is not universally optimal. The best threshold depends on the relative cost of false positives and false negatives.

For multiclass classification, the matrix becomes C × C, where rows represent actual classes and columns represent predicted classes. Off-diagonal values show class confusions, while row totals show the number of actual examples in each class.

Common Pitfalls

  • Using accuracy on imbalanced data: A model can achieve high accuracy by always predicting the majority class. Also report precision, recall, F1, and—where appropriate—PR-AUC and ROC-AUC.

  • Tuning the threshold on test data: This leaks test information into model selection. Tune on validation or cross-validation predictions, then evaluate on untouched test data.

  • Ignoring class costs: FP and FN errors may have very different consequences. Choose or tune thresholds according to domain requirements.

  • Treating calibration as ranking: A model can have good ROC-AUC while its predicted probabilities are poorly calibrated. Use calibration methods when probability reliability matters.

  • Confusing logistic regression with a nonlinear classifier: Logistic regression has a linear decision boundary. Nonlinear relationships may require feature engineering or a different model.

  • Multiclass version differences: In scikit-learn 1.5–1.7, the multi_class option is deprecated and is scheduled for removal in 1.8, so it should not be passed in new code. The default lbfgs solver uses multinomial loss for multiclass problems; one-vs-rest and multinomial logistic regression are distinct multiclass formulations.

Common follow-up questions include: Which error is more costly in this application? Would you use a fixed threshold or tune it? How would class imbalance affect the results? What metric should the business use? How would you validate the threshold without leaking test data?

20

How do ROC curves and AUC evaluate logistic regression?

The ROC curve shows how a classifier’s sensitivity and false-positive rate change across decision thresholds, while the AUC summarizes its ability to rank positive examples above negative examples. This is useful for comparing logistic regression models independently of a particular threshold, but AUC does not select an operating threshold, measure probability calibration, or show whether useful precision is available at that threshold.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How ROC and AUC Work

Logistic regression first calculates a linear score:

z = w·x + b

Here, w contains the learned feature weights, x is the feature vector, b is the intercept, and · means dot product. The score is converted into a positive-class probability by the sigmoid function:

p = 1 / (1 + e^(-z))

A threshold converts this probability into a class prediction. The usual threshold is 0.5, although it should be selected according to the costs of false positives and false negatives.

At each threshold, the ROC curve records:

  • True-positive rate: TPR = TP / (TP + FN), the proportion of actual positives correctly detected.

  • False-positive rate: FPR = FP / (FP + TN), the proportion of actual negatives incorrectly classified as positive.

Here, TP, TN, FP, and FN are true positives, true negatives, false positives, and false negatives. The ROC curve plots FPR on the horizontal axis and TPR on the vertical axis.

AUC is the area under that curve:

AUC = ∫ TPR(t) dFPR(t)

Here, t is the decision threshold, and the integral measures the total area as FPR increases. An AUC of 1.0 represents perfect ranking, while 0.5 represents random ranking. Equivalently, AUC estimates the probability that a randomly selected positive receives a higher score than a randomly selected negative, counting a tie as half credit.

Interpreting Logistic Regression Predictions

Because the sigmoid is strictly increasing, predict_proba and decision_function produce the same ranking and ROC curve for this binary model. predict_proba returns the estimated positive-class probability.

Each coefficient changes the log-odds for a one-unit increase in its model input, holding other inputs constant; exp(coefficient) is the corresponding odds ratio. In the scaled pipeline below, one model-input unit is one standard deviation of the original feature.

A linear score produces a linear decision boundary in the original feature space. Unregularized logistic regression is trained by maximizing likelihood, equivalently minimizing binary cross-entropy or log loss. Scikit-learn’s default L2 version instead minimizes a penalized log-loss objective. Both objectives are convex, although the model cannot represent nonlinear relationships without feature engineering.

Practical Trade-offs

  • AUC: Provides one threshold-independent ranking metric for comparing models.

  • ROC curve: Reveals the sensitivity–false-positive-rate trade-off across thresholds and helps select an operating point.

  • Accuracy: Can be misleading on imbalanced data because a model favoring the majority class may appear accurate.

  • PR-AUC: Often more informative when the positive class is rare and false positives matter.

  • Calibration: AUC does not determine whether predicted probabilities reflect real-world frequencies.

  • Regularization: C is inverse regularization strength, so smaller C means stronger regularization. The default penalty is "l2"; "l1" requires "liblinear" or "saga", while "elasticnet" requires "saga". Features should be scaled because the penalty acts on coefficient magnitude.

  • Multiclass: Scikit-learn’s lbfgs solver uses multinomial logistic regression by default. The multi_class parameter was deprecated in versions 1.5–1.6 and removed in 1.7; current releases can use OneVsRestClassifier(LogisticRegression(...)) for explicit one-vs-rest behavior.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def build_model():
    return Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
    ])


# Load a bundled binary classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the test set untouched during training and threshold selection.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.20, stratify=y_train, random_state=42
)

# Fit preprocessing and the model only on the fitting data.
model = build_model()
model.fit(X_fit, y_fit)

# Select a threshold on validation probabilities using Youden's J.
validation_probabilities = model.predict_proba(X_val)[:, 1]
fpr, tpr, thresholds = roc_curve(y_val, validation_probabilities)
finite = np.isfinite(thresholds)
youden_j = tpr[finite] - fpr[finite]
best_threshold = float(thresholds[finite][np.argmax(youden_j)])

# Refit on training and validation data after threshold selection.
final_model = build_model()
X_train_val = np.vstack((X_fit, X_val))
y_train_val = np.concatenate((y_fit, y_val))
final_model.fit(X_train_val, y_train_val)

test_probabilities = final_model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

# Cross-validation estimates ranking performance on the training partition.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_auc = cross_val_score(
    build_model(), X_train, y_train, cv=cv, scoring="roc_auc"
)

print(f"Selected threshold: {best_threshold:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions))
print(f"Cross-validated ROC-AUC: {cv_auc.mean():.3f} +/- {cv_auc.std():.3f}")
print("Example predict_proba output:")
print(np.round(final_model.predict_proba(X_test[:3]), 3))

Common Pitfalls

  • Data leakage: Split the data before fitting the scaler or model. The Pipeline ensures scaling is learned separately within each training fold.

  • Ignoring imbalance: Accuracy can conceal poor minority-class performance. Inspect precision, recall, F1, PR-AUC, and confusion matrices; consider class_weight="balanced".

  • Using unscaled features: Scaling is important when regularization or gradient-based optimization is used. It preserves the family of linear boundaries, but with regularization it can change the fitted coefficients and resulting boundary.

  • Perfect separation: Logistic regression may produce extremely large coefficients. Lowering C strengthens regularization, while revised features may also address the cause.

  • Nonlinear relationships: A linear log-odds assumption may require polynomial features, interactions, or a more flexible model.

  • Treating AUC as calibration: A model can rank examples well while its predicted probabilities remain poorly calibrated.

  • Tuning on the test set: Select thresholds and hyperparameters using validation data or cross-validation, then use the test set only for final evaluation.

21

What classification metrics assess logistic regression models?

Common classification metrics for logistic regression include accuracy, precision, recall, F1 score, balanced accuracy, ROC-AUC, PR-AUC, and log loss. The best choice depends on class balance, decision thresholds, and whether false positives or false negatives are more costly; the main trade-off is that one aggregate score can conceal important error patterns.

Read full answer

Applies tomachine learning concepts and scikit-learn 1.5+ (Python 3.12)

Model Outputs and Evaluation

Binary logistic regression computes a linear score z = w · x + b and converts it into a probability with the sigmoid function p = 1 / (1 + e^(-z)), where x is the feature vector, w contains the coefficients, b is the intercept, and p is the estimated probability of class 1. The corresponding decision boundary is linear. Training maximizes the likelihood of the observed labels, equivalently minimizing binary cross-entropy, or log loss, which is a convex optimization problem.

Each coefficient changes the predicted log-odds for a one-unit increase in that feature; exp(coefficient) is the corresponding odds ratio. These probability and ranking outputs can be evaluated with threshold-dependent metrics, threshold-independent curve metrics, and probability-quality metrics.

Core Classification Metrics

Let TP mean true positives, TN true negatives, FP false positives, and FN false negatives.

Metric What It Measures Formula Main Use
Accuracy Fraction of all predictions that are correct Accuracy = (TP + TN) / (TP + TN + FP + FN) Balanced classes with similar error costs
Precision Fraction of predicted positives that are actually positive Precision = TP / (TP + FP) False positives are expensive
Recall Fraction of actual positives that the model detects Recall = TP / (TP + FN) Missing a positive case is expensive
F1 score Harmonic mean of precision and recall F1 = 2 * (Precision * Recall) / (Precision + Recall) Balancing precision and recall
Balanced accuracy Average recall across both classes Balanced accuracy = (Recall for class 0 + Recall for class 1) / 2 Unequal class frequencies
ROC-AUC Ranking performance across all thresholds Area under the TPR-versus-FPR curve Comparing threshold-independent ranking quality
PR-AUC Precision-recall trade-off across thresholds Area under the precision-versus-recall curve Strongly imbalanced classification
Log loss Quality of predicted probabilities Log loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p)) Evaluating probability estimates

In the log-loss formula, n is the number of examples, y is the true binary label, and p is the predicted probability of class 1. Lower log loss indicates better probability predictions.

Confusion Matrix and Class Imbalance

A confusion matrix shows the four prediction outcomes: TP, positive cases predicted as positive; TN, negative cases predicted as negative; FP, negative cases incorrectly predicted as positive; and FN, positive cases incorrectly predicted as negative.

Accuracy can be misleading when one class dominates. If 95% of cases are negative, always predicting negative achieves 95% accuracy while detecting no positives. In that situation, recall, balanced accuracy, F1, and PR-AUC usually provide a better assessment.

ROC-AUC uses TPR = TP / (TP + FN) and FPR = FP / (TP? + TN) conceptually, although FPR = FP / (FP + TN) is the correct formula: false positives divided by all actual negatives. A ROC-AUC of 0.5 corresponds to random ranking, while 1.0 represents perfect ranking. PR-AUC is especially useful under imbalance; its random-ranking baseline is the positive-class prevalence, which may be far below 0.5.

Multiclass Evaluation

For multiclass logistic regression, scikit-learn 1.5+ uses multinomial, or softmax, loss by default with the lbfgs solver. The multi_class parameter is deprecated, so it should not be set. One-vs-rest training remains available through OneVsRestClassifier.

Precision, recall, and F1 require an averaging strategy for multiclass predictions:

  • Macro averaging: Gives each class equal weight and reveals poor minority-class performance.

  • Weighted averaging: Weights each class by its support, so large classes dominate.

  • Micro averaging: Aggregates counts across classes; for single-label multiclass classification, micro precision, recall, and F1 equal accuracy.

ROC-AUC and PR-AUC also require a multiclass averaging choice, commonly macro or weighted one-vs-rest averaging. Multiclass log loss uses categorical cross-entropy and penalizes confidently incorrect class probabilities.

Choosing a Decision Threshold

Logistic regression produces probabilities such as 0.20 or 0.80. A binary classifier commonly uses 0.50 as its default decision threshold, but changing that threshold changes the precision-recall balance without retraining the model:

  • A higher threshold generally produces fewer positive predictions, increasing precision and potentially reducing recall.

  • A lower threshold generally produces more positive predictions, increasing recall and potentially reducing precision.

  • Select the threshold using validation or cross-validation data and the costs of false positives and false negatives.

Do not repeatedly inspect the final test set while selecting a threshold, because that leaks test information into model selection.

Common Pitfalls

  • Reporting only accuracy: It can conceal poor minority-class performance.

  • Treating ROC-AUC as calibration: Good ranking does not guarantee accurate probabilities; assess calibration separately.

  • Ignoring the confusion matrix: Aggregate scores can hide which errors the model makes.

  • Selecting a threshold on test data: This invalidates the final test estimate.

  • Ignoring class imbalance: Consider class_weight="balanced" when appropriate and inspect precision, recall, F1, ROC-AUC, and PR-AUC.

  • Failing to specify averaging: Multiclass precision, recall, and F1 depend on whether averaging is macro, weighted, or micro.

  • Failing to report the positive class: Precision and recall depend on which label is treated as positive.

22

How can you handle imbalanced classes in logistic regression?

Class weighting, probability-threshold tuning, and careful resampling are common ways to handle imbalanced classes in logistic regression. Class weighting and resampling influence training, while threshold tuning changes the operating point after training; the main trade-off is usually improved minority-class recall at the cost of more false positives and potentially lower overall accuracy.

Read full answer

Applies toscikit-learn 1.5–1.7 (Python 3.12)

How Logistic Regression Handles Imbalance

For binary classification, logistic regression computes a linear score:

z = w · x + b

It then converts that score into a probability:

p(y = 1 | x) = 1 / (1 + e^(-z))

Here, w is the coefficient vector, x is the feature vector, b is the intercept, · is the dot product, e is Euler’s number, and y is the binary class label. The model is trained by minimizing binary cross-entropy, also called log loss. With L2 regularization, this optimization problem is convex.

The coefficients represent changes in log-odds: increasing a feature by one unit changes the log-odds by that feature’s coefficient. Therefore, exp(coefficient) is the corresponding odds ratio for a one-unit increase.

Because the decision boundary is linear, logistic regression may require feature engineering for nonlinear relationships.

Main Strategies

  • Use class_weight="balanced": This gives minority-class examples larger loss weights, so errors on those examples have more influence. It uses all original observations while encouraging the model to learn the minority class.

  • Tune the probability threshold: The default threshold is 0.5, but another threshold may provide a better precision-recall trade-off. Select it using validation or cross-validated predictions, not the test set.

  • Use suitable metrics: Report precision, recall, F1, ROC-AUC, and especially PR-AUC for severely imbalanced data. Accuracy can be misleading when the majority class dominates.

  • Resample carefully: Random undersampling, oversampling, or SMOTE may help, but resampling must occur inside each training fold to prevent leakage.

  • Consider calibration: Class weighting can change the probability scale and lead to poor calibration. Threshold tuning does not modify predicted probabilities or affect calibration; it only determines which predicted probabilities count as positive.

For multiclass logistic regression, scikit-learn 1.5–1.6 supports one-vs-rest and multinomial softmax formulations, with multi_class deprecated. In scikit-learn 1.7, that parameter was removed: lbfgs uses multinomial loss, while liblinear uses one-vs-rest.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (
    train_test_split,
    StratifiedKFold,
    cross_val_score,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
    average_precision_score,
)

# Load a bundled binary classification dataset.
data = load_breast_cancer()
X = data.data

# Encode malignant cases as class 1 so threshold tuning targets
# the minority class. The original target uses 0 for malignant.
y = (data.target == 0).astype(int)

# Keep a final test set untouched during model and threshold selection.
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

# Use a separate validation set for threshold tuning.
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    stratify=y_train_val,
    random_state=42,
)

# Scaling and logistic regression are fitted only on training data.
model = Pipeline([
    ("scale", StandardScaler()),
    ("logistic", LogisticRegression(
        solver="liblinear",
        penalty="l2",
        C=1.0,
        class_weight="balanced",
        max_iter=1000,
        random_state=42,
    )),
])

# Stratified cross-validation preserves class proportions in each fold.
cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

cv_auc = cross_val_score(
    model,
    X_train,
    y_train,
    cv=cv,
    scoring="roc_auc",
    n_jobs=-1,
)

model.fit(X_train, y_train)

# The second column is the probability of class 1.
val_prob = model.predict_proba(X_val)[:, 1]
test_prob = model.predict_proba(X_test)[:, 1]

# Tune the threshold on validation data only.
thresholds = np.linspace(0.10, 0.90, 161)

best_threshold = max(
    thresholds,
    key=lambda threshold: (
        f1_score(
            y_val,
            (val_prob >= threshold).astype(int),
            zero_division=0,
        ),
        -abs(threshold - 0.5),
    ),
)

# Evaluate the selected threshold on the untouched test set.
test_pred = (test_prob >= best_threshold).astype(int)

print(f"Cross-validated ROC-AUC: {cv_auc.mean():.3f} "
      f"+/- {cv_auc.std():.3f}")
print(f"Selected threshold: {best_threshold:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_pred):.3f}")
print(f"Test precision: {precision_score(y_test, test_pred):.3f}")
print(f"Test recall: {recall_score(y_test, test_pred):.3f}")
print(f"Test F1: {f1_score(y_test, test_pred):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_prob):.3f}")
print(f"Test PR-AUC (average precision): "
      f"{average_precision_score(y_test, test_prob):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_pred))

Regularization and Feature Scaling

  • C is the inverse of regularization strength: a larger C means weaker regularization, while a smaller C means stronger regularization.

  • The default l2 penalty is appropriate for most multifeature problems and helps control large coefficients.

  • l1 can produce sparse models and is supported with liblinear or saga.

  • elasticnet combines L1 and L2 penalties and requires saga.

  • Scaling features is recommended before regularized logistic regression because coefficients are affected by feature scale.

  • Perfect separation can produce very large coefficients; regularization helps, but separated data should still be diagnosed explicitly.

Common Pitfalls

  • Applying SMOTE or undersampling before splitting the data causes data leakage.

  • Reporting only accuracy can make a model appear effective while minority-class recall is poor.

  • Tuning the threshold on the test set produces an optimistically biased result.

  • Assuming class_weight="balanced" guarantees good probabilities; calibration may still be needed.

  • Ignoring nonlinear feature relationships leads to underfitting even when imbalance is handled correctly.

  • For interview discussions, mention the trade-off between minority recall and false positives, and explain why PR-AUC is often more informative than accuracy for severe imbalance.

23

How do you select a threshold for logistic regression predictions?

A classification threshold converts a logistic-regression probability into a class prediction: predict 1 when P(y=1|x) >= t, otherwise predict 0. Although 0.5 is the conventional default, the best threshold depends mainly on the relative costs and consequences of false positives and false negatives.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Model and Threshold

Binary logistic regression computes a linear score z = w · x + b and converts it to a probability with p = 1 / (1 + e^(-z)). Here, w contains the fitted weights, x contains the input features, and b is the intercept. The sigmoid maps any score to a probability, and the resulting binary decision boundary is linear.

Training maximizes the likelihood of the observed labels, equivalently minimizing binary log loss; this objective is convex. Each coefficient represents the change in log-odds for a one-unit increase in that feature, holding other features fixed, so the exponential of a coefficient is the corresponding odds ratio.

For multiclass classification, one-vs-rest trains binary classifiers, while multinomial logistic regression uses softmax probabilities. In scikit-learn 1.5–1.6, multi_class is deprecated and lbfgs uses multinomial by default; the parameter was removed in 1.7.

The regularization parameter C is the inverse of regularization strength. The default penalty is l2; l1 requires liblinear or saga, and elasticnet requires saga. Scale features before regularization so coefficients and penalties are comparable.

Threshold Selection Methods

Method Objective Main limitation
Default 0.5 Use the conventional cutoff Does not reflect asymmetric error costs
Youden’s J Maximize TPR - FPR Ignores prevalence and operational costs
F1 or F-beta Optimize the harmonic mean of precision and recall Ignores true negatives; F-beta weights precision or recall
Expected cost Minimize expected false-positive and false-negative cost Requires reliable cost estimates
Quantile threshold Control the predicted-positive rate May select a rate with poor probability quality
Precision-recall curve Inspect performance across thresholds Does not automatically select one operating point

For expected-cost selection, minimize C_FP * FP + C_FN * FN. Here, C_FP is the cost of one false positive, FP is the number of false positives, C_FN is the cost of one false negative, and FN is the number of false negatives.

Practical Workflow

  1. Define the relevant error costs or target metric.

  2. Generate out-of-fold probabilities with StratifiedKFold; a pipeline ensures preprocessing is fitted only on each training fold.

  3. Select the threshold using the chosen objective.

  4. Check threshold stability across folds and assess probability calibration.

  5. Refit on the development data and evaluate once on an untouched test set.

  6. For rare positive classes, report precision, recall, F1, PR-AUC, and the confusion matrix because accuracy can be misleading.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Make "malignant" the positive class, encoded as 1.
data = load_breast_cancer()
X = data.data
y = (data.target == 0).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

# Tune the threshold using out-of-fold predictions only.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

# Youden's J threshold.
fpr, tpr, roc_thresholds = roc_curve(y_train, oof_probabilities)
youden_threshold = roc_thresholds[np.argmax(tpr - fpr)]

# F1 threshold from the precision-recall curve.
precision, recall, pr_thresholds = precision_recall_curve(
    y_train, oof_probabilities
)
denominator = precision[:-1] + recall[:-1]
f1_scores = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
f1_threshold = pr_thresholds[np.argmax(f1_scores)]

# Refit on the development data and evaluate on the untouched test set.
model.fit(X_train, y_train)
test_probabilities = model.predict_proba(X_test)[:, 1]

for name, threshold in [
    ("default", 0.5),
    ("Youden", youden_threshold),
    ("F1", f1_threshold),
]:
    predictions = (test_probabilities >= threshold).astype(int)
    print(
        name,
        "threshold=", round(float(threshold), 4),
        "accuracy=", round(accuracy_score(y_test, predictions), 4),
        "roc_auc=", round(roc_auc_score(y_test, test_probabilities), 4),
        "confusion_matrix=",
        confusion_matrix(y_test, predictions, labels=[0, 1]).tolist(),
    )

print("Example probabilities:", test_probabilities[:5])

Common Pitfalls

  • Choosing on test data: This contaminates the final evaluation; tune with validation or out-of-fold predictions.

  • Ignoring imbalance: Consider class_weight="balanced" and emphasize precision, recall, F1, and PR-AUC.

  • Assuming linearity suffices: Non-linear relationships may require feature engineering or a more flexible model.

  • Perfect separation: It can produce very large coefficients and unstable probabilities.

  • Confusing ranking with calibration: A strong ROC-AUC does not guarantee well-calibrated probabilities or an appropriate threshold.

Common follow-up questions include interpreting coefficients as log-odds, choosing C, detecting perfect separation, and improving probability calibration.

24

How does the Hosmer-Lemeshow test assess logistic regression fit?

The Hosmer-Lemeshow test is a goodness-of-fit test for binary logistic regression that checks whether predicted event probabilities are broadly consistent with observed outcomes. It is commonly used to detect systematic calibration error in a logistic model. Its main trade-off is that it is simple and widely recognized, but the result depends on how observations are grouped and it can have limited power when events are sparse.

Read full answer

Applies tomachine learning and statistical concepts (all versions)

How It Works

A binary logistic regression assigns each observation a linear score z_i = w · x_i + b and converts it into a probability using the sigmoid, p_i = 1 / (1 + e^(-z_i)) = P(y_i = 1 | x_i). Here, x_i is the observation’s feature vector, y_i is its binary outcome, w contains the coefficients, and b is the intercept. The model has a linear decision boundary and is trained by maximum likelihood, equivalently by minimizing binary cross-entropy, which is convex. A one-unit increase in feature k, holding other features fixed, changes the log-odds by w_k; e^(w_k) is the corresponding odds ratio.

The Hosmer-Lemeshow test compares observed event counts with the counts expected from the fitted probabilities:

  1. Calculate each observation’s predicted probability p_i.

  2. Sort observations by predicted probability and divide them into approximately equal-sized groups, often deciles.

  3. For each group j, calculate:

  4. O_1j: number of observed positive outcomes.

  5. E_1j: expected number of positive outcomes, equal to the sum of predicted probabilities in that group.

  6. O_0j: number of observed negative outcomes.

  7. E_0j: expected number of negative outcomes, equal to the sum of 1 - p_i.

  8. Compute the statistic:

X² = sum over groups j of [(O_1j - E_1j)² / E_1j + (O_0j - E_0j)² / E_0j]

Here, i indexes an observation, j indexes a group, and X² is the test statistic. Larger differences between observed and expected counts produce a larger statistic.

The classic same-data formulation compares the statistic with a chi-square distribution using g - 2 degrees of freedom, where g is the number of groups. With ten groups, this gives eight degrees of freedom. External-validation versions and modified variants can use different degrees-of-freedom conventions, so the documentation for the specific procedure should be followed.

Interpreting the Result

  • Null hypothesis: The observed event rates are consistent with the probabilities predicted by the logistic model.

  • Small p-value: Reject the null hypothesis at the chosen significance level, usually 0.05. The model shows evidence of lack of fit or poor calibration.

  • Large p-value: Fail to reject the null hypothesis. This does not prove that the model is correct; it only means the test did not detect a significant mismatch.

The test can reveal problems such as omitted nonlinear effects, missing interactions, inappropriate functional forms, or systematically biased predictions. However, it does not identify which observations or variables caused the problem.

Practical Use

The test is most useful as a calibration diagnostic after fitting a logistic regression. For example, if a credit-risk model predicts low probabilities for most customers and high probabilities for a smaller group, the test checks whether observed default rates in those probability groups match the model’s predictions.

For a dependable assessment:

  • Prefer predictions from a validation set or out-of-fold cross-validation predictions. Testing calibration on heavily overfitted training predictions may hide poor generalization.

  • Investigate significant results by examining calibration plots, residuals, and potential nonlinear terms.

  • Refit the model after addressing identified problems and validate the revised model again.

  • Do not confuse calibration with discrimination: a model can correctly rank cases by risk while producing poorly calibrated probabilities.

Limitations and Alternatives

Diagnostic What It Measures Main Advantage Main Limitation
Hosmer-Lemeshow test Global mismatch between observed outcomes and predicted probabilities Familiar chi-square test with a p-value Sensitive to grouping and sparse counts
Calibration plot Observed event rate across predicted-probability ranges Shows where and how miscalibration occurs Requires enough observations in each range
Brier score Mean squared error between probabilities and binary outcomes Simple measure of probability accuracy Does not decompose calibration from other errors
Calibration intercept and slope Systematic and proportional probability bias More informative than a single global p-value Requires appropriate sample size and model fitting

Modern guidance often treats the Hosmer-Lemeshow test as a supplementary diagnostic rather than the sole criterion for logistic regression fit.

Common Pitfalls

  • Treating p >= 0.05 as proof that the model is well calibrated.

  • Applying the test to training-set predictions from an overfitted model.

  • Creating groups with very small expected positive or negative counts, which can invalidate the chi-square approximation; combine or re-bin such groups rather than dropping them.

  • Using arbitrary group counts without documenting how they were selected.

  • Interpreting a significant result as proof of a particular omitted variable or nonlinear relationship.

  • Selecting variables, transformations, or thresholds repeatedly against the same validation set, which causes validation overfitting.

  • Confusing the test with classification metrics such as accuracy, precision, recall, or ROC-AUC; the Hosmer-Lemeshow test specifically evaluates probability calibration.

25

How do you diagnose and prevent logistic regression overfitting?

Overfitting in logistic regression occurs when the model learns training-specific patterns that fail on unseen data. The primary remedy is regularization, which sacrifices some training fit to reduce variance. The main trade-off is lower variance and more stable generalization versus increased bias.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Logistic Regression Overfits

For binary classification, logistic regression computes a linear score:

z = w · x + b

It then converts the score into a probability:

p(y = 1 | x) = 1 / (1 + e^(-z))

  • x is the feature vector and w is the corresponding coefficient vector.

  • · denotes the vector dot product.

  • b is the intercept and z is the linear decision score.

  • e is Euler’s number, the base of the natural exponential.

  • p(y = 1 | x) is the estimated probability of the positive class.

Training minimizes binary cross-entropy, also called log loss:

log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the sample count, y is the true binary label, p is the predicted probability, and log is the natural logarithm. This objective is convex, but an unregularized model can still overfit when the sample is small or the feature count is high.

Diagnostic signals include:

  • A persistent gap between training and validation performance.

  • Unstable coefficients across folds, especially with correlated features.

  • Strong training performance but weaker held-out recall, F1, PR-AUC, or log loss.

  • Performance that deteriorates as unnecessary features are added.

Each coefficient changes the log-odds for a one-unit feature increase, holding other features fixed; exp(coefficient) is the corresponding odds ratio.

Diagnosis and Prevention

  1. Create honest splits. Reserve the final test set before model selection. Use StratifiedKFold for classification and put scaling and imputation inside a Pipeline so they are fitted only on training folds.

  2. Compare training, validation, and cross-validation results. Learning curves can distinguish high variance from high bias. Inspect whether stronger regularization narrows the generalization gap without causing unacceptable underfitting.

  3. Tune regularization. In scikit-learn, C is the inverse of regularization strength, so smaller C means stronger regularization.

Penalty Suitable solvers Effect
l2 lbfgs, liblinear, saga Shrinks coefficients; default baseline
l1 liblinear, saga Can set coefficients exactly to zero
elasticnet saga Combines L1 sparsity and L2 shrinkage

Scale features before fitting, especially with L1 or elastic-net penalties, because these penalties operate on coefficient magnitudes.

  1. Use suitable metrics. Report precision, recall, F1, ROC-AUC, PR-AUC, log loss, and a confusion matrix when classes are imbalanced. Accuracy can be misleading when the positive class is rare.

  2. Address other causes deliberately. Engineer validated nonlinear features, collect more representative data, or use class_weight="balanced" when minority performance matters. Threshold tuning changes the operating point; it does not itself prevent overfitting.

For multiclass classification, lbfgs uses multinomial softmax by default, while liblinear uses one-vs-rest. The multi_class parameter was deprecated in scikit-learn 1.5 and is scheduled for removal, so it should not be set.

For dense data, one full optimization pass is roughly O(n * d) and prediction is O(d) per row. Memory is dominated by the O(n * d) training matrix; the model itself stores only O(d) coefficients (O(d * K) for K classes), and lbfgs keeps a few extra vectors of length d.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, average_precision_score, confusion_matrix,
    f1_score, precision_recall_curve, roc_auc_score,
)
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Class 1 represents malignant tumors.
X, y = load_breast_cancer(return_X_y=True)
y = (y == 0).astype(np.int64)

# Reserve the final test set before model selection.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.2, stratify=y_train, random_state=42
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("logreg", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
search = GridSearchCV(
    pipeline,
    param_grid={"logreg__C": [0.01, 0.1, 1.0, 10.0]},
    scoring="average_precision",
    cv=cv,
)
search.fit(X_fit, y_fit)
model = search.best_estimator_

# Select an F1-oriented threshold using validation data only.
val_probabilities = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, val_probabilities)
denominator = precision[:-1] + recall[:-1]
f1_values = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator != 0,
)
threshold = float(thresholds[int(np.argmax(f1_values))])

# Evaluate once on the untouched test set.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= threshold).astype(np.int64)

print(f"Selected C: {search.best_params_['logreg__C']}")
print(f"Best mean CV PR-AUC: {search.best_score_:.3f}")
print(f"Training accuracy: {model.score(X_fit, y_fit):.3f}")
print(f"Validation accuracy: {model.score(X_val, y_val):.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Test F1: {f1_score(y_test, test_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"Test PR-AUC: {average_precision_score(y_test, test_probabilities):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("First five positive-class probabilities:", test_probabilities[:5])

Common Pitfalls

  • Data leakage: Fitting the scaler, imputer, or hyperparameter search on the entire dataset.

  • Wrong metric: Relying only on accuracy for severely imbalanced classes.

  • Overfitting model selection: Repeatedly tuning against the same validation results; use nested cross-validation or an untouched test set.

  • Perfect separation: This can produce very large coefficients; strengthen regularization and investigate the data.

  • Poor calibration: Strong ROC-AUC does not guarantee accurate probabilities; use CalibratedClassifierCV with cross-validation or held-out calibration data.

  • Reproducibility: Set fixed random seeds for splits and folds.

  • Missing values: Impute inside the training pipeline, not before splitting.

  • Hardware: Standard scikit-learn logistic regression is CPU-based; GPU acceleration is not a typical benefit.

26

How does feature engineering affect logistic regression?

Feature engineering changes the input representation used by logistic regression, allowing the model to represent relationships that are nonlinear in the original variables. Transformations such as polynomial terms, interactions, bins, and domain-specific ratios can improve predictive performance, but they also increase dimensionality, overfitting risk, interpretation complexity, and the risk of data leakage. The main trade-off is whether the added signal justifies the extra complexity.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Feature Engineering Changes the Model

For binary classification, logistic regression calculates a linear score:

z = w · x + b

It then converts that score into a probability:

p(y = 1 | x) = 1 / (1 + e^(-z))

  • x is the feature vector.

  • w contains one coefficient per feature.

  • b is the intercept.

  • · denotes the dot product.

  • z is the model’s linear score.

  • e is the base of the natural logarithm.

A threshold such as 0.5 converts the probability into a class label. The boundary w · x + b = 0 is linear in the model’s input features.

Feature engineering applies a transformation g(x) before prediction. The boundary can therefore be nonlinear in the original variables while remaining linear in the engineered variables. For example, adding x² lets logistic regression represent a curved boundary that ordinary linear terms cannot.

Training maximizes the likelihood of the observed labels, equivalently minimizing binary cross-entropy, also called log loss. This objective is convex, although overfitting can still occur when the feature space is unnecessarily large.

Each coefficient changes the predicted log-odds, holding the other features constant. For a one-unit increase in a feature, exp(coefficient) is the corresponding odds ratio. For an engineered feature, this interpretation applies per unit of the engineered value rather than necessarily per unit of an original variable.

Common Feature Engineering Choices

Transformation Potential benefit Important limitation
Polynomial terms Capture curves and thresholds Can create correlated features and overfit
Interaction terms Model combined effects Multiply the number of features
Binning or one-hot encoding Capture thresholds and categorical effects Lose precision or impose arbitrary bins
Domain ratios or logarithms Encode meaningful relationships Require domain knowledge and valid values
Scaling Improve optimization and regularization behavior Does not add predictive information by itself

Feature scaling is especially important with L1, L2, or elastic-net regularization because the penalty treats coefficients on different raw scales as comparable. In scikit-learn, C is the inverse of regularization strength: larger C means weaker regularization. The default penalty is l2; l1 is supported with liblinear or saga, while elasticnet requires saga.

Multiclass Logistic Regression

For multiclass classification:

  • One-vs-rest: fits one binary classifier per class and combines their results.

  • Multinomial: models mutually exclusive classes directly with softmax probabilities.

In scikit-learn 1.5–1.6, the multi_class parameter is deprecated, and lbfgs selects multinomial behavior by default. The parameter was removed in scikit-learn 1.7; lbfgs continues to use multinomial behavior, while liblinear uses one-vs-rest.

Feature engineering affects both formulations by changing the dimensionality and the hypotheses each classifier can represent.

Evaluation and Practical Guidance

  1. Establish a baseline: Compare engineered features with an unengineered logistic regression model.

  2. Use cross-validation: Compare models with StratifiedKFold, ensuring every transformation is fitted separately within each training fold.

  3. Prevent leakage: Do not fit scaling, binning, imputation, feature selection, or polynomial transformations on the full dataset before splitting.

  4. Select thresholds correctly: Use validation or out-of-fold predictions to choose a threshold, then evaluate it once on an untouched test set.

  5. Use suitable metrics: Report precision, recall, F1, ROC-AUC, and PR-AUC when classes are imbalanced. Accuracy can be misleading because a model can appear accurate by favoring the majority class.

  6. Handle missing values explicitly: Impute missing values within the training pipeline so that imputation rules do not leak information from validation or test data.

  7. Preserve reproducibility: Fix random seeds and record the selected transformations and hyperparameters.

For dense binary data, an optimization pass is approximately O(n × d), where n is the number of samples and d is the number of features. Polynomial expansions can increase d rapidly, so the computational cost may grow much faster than the original feature count suggests.

Common Pitfalls

  • Assuming engineering always helps: Additional features may introduce noise, redundant information, or overfitting; always compare against a baseline.

  • Data leakage: Fitting transformations before splitting can produce unrealistically strong validation or test results.

  • Perfect separation: Separated data can produce extremely large coefficients and unstable predictions; stronger regularization may help.

  • Misinterpreting coefficients: A coefficient for a polynomial or interaction term does not describe a simple one-unit change in an original feature.

  • Nonlinearity alone may be insufficient: If interactions or transformations still do not capture the relationship, consider a more flexible model.

  • Poor calibration: Strong discrimination does not guarantee accurate probabilities; assess calibration when predicted probabilities are used in decisions.

27

How do polynomial and interaction terms work in logistic regression?

Polynomial and interaction terms extend logistic regression so it can model curved relationships and effects that depend on multiple features. They preserve the model’s probabilistic classification structure, but increase overfitting risk and reduce interpretability. The main trade-off is flexibility versus regularization and complexity.

Read full answer

Applies tomachine learning concepts; examples in Python 3.12 with scikit-learn 1.5+

How It Works

For binary classification, logistic regression computes a linear score and converts it into a probability:

z = β0 + Σ_j βj*xj + Σ_j γj*xj² + Σ_(j<k) δjk*xj*xk

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, x_j is original feature j; j and k are feature indices; the sigma signs denote sums; β0 is the intercept; βj is the coefficient for feature xj; γj is the coefficient for its squared term; and δjk is the coefficient for the interaction xj*xk. The symbol P denotes probability, y is the binary label, and e is Euler’s number.

A polynomial expansion of degree r adds powers such as xj² through xj^r. Interaction terms add products such as xj*xk, allowing one feature’s effect to depend on another.

Although the expanded model is nonlinear in the original features, its decision boundary remains linear in the transformed feature space. The prediction threshold p >= 0.5 is equivalent to z >= 0.

The unregularized objective is log loss, also called binary cross-entropy:

log loss = -(1/n) * Σ [y*log(p) + (1-y)*log(1-p)]

Here, n is the number of observations, p is the predicted probability of class 1, and log is the natural logarithm. Minimizing this convex objective is equivalent to maximizing the likelihood of the observed labels.

When the Terms Are Useful

Term Purpose Example
Polynomial term Models curvature for one feature x² represents diminishing or accelerating effects
Interaction term Models effects that depend on another feature income*age represents a joint effect
Both Models nonlinear and conditional relationships distance² plus distance*traffic

Use terms supported by domain knowledge or validated empirically. For d original features, a complete quadratic expansion creates d + d(d+1)/2 transformed features when a constant bias column is excluded, so dimensionality grows quadratically.

Interpretation and Regularization

A coefficient represents a change in log-odds for a one-unit increase in its transformed feature, holding other terms fixed. Its odds ratio is exp(coefficient). For an interaction, the marginal effect of x depends on another variable; for x², the derivative is 2γx.

In scikit-learn:

  • C is the inverse of regularization strength: larger C means weaker regularization.

  • "l2" is the default penalty.

  • "l1" requires the liblinear or saga solver.

  • "elasticnet" requires the saga solver.

  • Scaling features before fitting is strongly recommended, especially with regularization.

  • In scikit-learn 1.5–1.6, multi_class was deprecated and removed in 1.7. Current versions use multinomial loss with the default lbfgs solver; use OneVsRestClassifier(LogisticRegression(...)) when explicit one-vs-rest behavior is required.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_predict
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    roc_auc_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
)

# Bundled binary classification dataset
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the test set untouched during model and threshold selection
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = make_pipeline(
    PolynomialFeatures(degree=2, include_bias=False),
    StandardScaler(),
    LogisticRegression(max_iter=1000, random_state=42),
)

# Generate out-of-fold probabilities for threshold tuning
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]

thresholds = np.linspace(0.05, 0.95, 181)
f1_scores = [
    f1_score(
        y_train,
        (oof_probabilities >= threshold).astype(int),
        zero_division=0,
    )
    for threshold in thresholds
]
best_threshold = float(thresholds[np.argmax(f1_scores)])

# Refit on all training data, then evaluate once on the test set
model.fit(X_train, y_train)
test_probabilities = model.predict_proba(X_test)[:, 1]
predictions = (test_probabilities >= best_threshold).astype(int)

print(f"Best validation threshold: {best_threshold:.2f}")
print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(
    f"PR-AUC (average precision): "
    f"{average_precision_score(y_test, test_probabilities):.3f}"
)
print("Confusion matrix:")
print(confusion_matrix(y_test, predictions, labels=[0, 1]))
print("First five positive-class probabilities:", test_probabilities[:5])

Evaluation Guidance

Use StratifiedKFold for classification so every fold preserves class proportions. Select feature terms and regularization using cross-validation, then keep a final test set untouched.

ROC-AUC measures ranking quality across thresholds, while PR-AUC is especially useful for imbalanced data. Accuracy can mislead when the minority class is rare. Threshold tuning should use validation or out-of-fold predictions, not the test set.

Common Pitfalls

  • Leakage: Fitting StandardScaler before the split leaks test statistics. PolynomialFeatures is deterministic, but keeping all transforms in the pipeline ensures each cross-validation fold learns preprocessing from its training fold only.

  • Overfitting: Adding every polynomial and interaction creates correlated, high-dimensional features; prefer targeted terms and regularization.

  • Perfect separation: Flexible terms can produce extremely large coefficients; regularization helps control their magnitude.

  • Class imbalance: Consider class_weight="balanced" and metrics such as PR-AUC, F1, recall, or precision.

  • Interpretation errors: Coefficients describe transformed terms, so exp(coefficient) is not automatically a one-unit effect in the original feature scale.

  • Calibration changes: Greater flexibility may improve discrimination while worsening probability calibration; validate reliability when predicted probabilities matter.

Interview follow-ups commonly ask why a particular interaction is justified, how the transformed feature count grows, why scaling is needed, how coefficients become odds ratios, and how separation or imbalance affects fitting.

28

How does ordinal logistic regression handle ordered outcomes?

Ordinal logistic regression extends binary logistic regression with cumulative logits to model outcomes whose categories have a meaningful order, such as low, medium, and high. It estimates the probability of falling at or below each threshold while sharing one coefficient vector across thresholds. Its main trade-off is parsimony and interpretability versus the restrictive proportional-odds assumption.

Read full answer

Applies tostatsmodels 0.14+ and scikit-learn 1.7+ (Python 3.12)

Model and Formulation

For an outcome Y in {0, 1, ..., K-1}, an ordinal logistic regression estimates K-1 cumulative probabilities:

P(Y <= k | x) = 1 / (1 + e^-(alpha_k - beta^T x))

for thresholds k = 0, ..., K-2, where:

  • P denotes probability.

  • Y is the observed ordinal category.

  • k identifies a threshold between adjacent categories.

  • e is the natural exponential.

  • alpha_k is the threshold-specific intercept.

  • beta contains the feature coefficients shared across thresholds.

  • x is the feature vector.

  • beta^T x is the linear predictor.

Individual category probabilities are differences between adjacent cumulative probabilities:

P(Y = k | x) = P(Y <= k | x) - P(Y <= k-1 | x)

Here, the cumulative probability is defined as zero below the first category and one above the last.

The thresholds and coefficients are estimated jointly by maximizing the ordered log-likelihood, equivalently minimizing the negative ordered log-likelihood or log loss. For the logit model, this objective is convex, although optimization can still be sensitive to separation or poorly scaled predictors.

Coefficient Interpretation

Re-expressing the model gives:

log[P(Y > k) / P(Y <= k)] = gamma_k + beta^T x

where gamma_k = -alpha_k is the threshold-specific intercept. For a one-unit increase in feature x_j, exp(beta_j) is the multiplicative change in the odds of being above a threshold. For example, exp(beta_j) = 2 means those odds double for each one-unit increase, holding the other variables constant.

Because the proportional-odds model shares the same coefficients at every threshold, the corresponding odds ratio is also shared across thresholds.

Key Assumptions and Extensions

  • Ordered categories: The categories have a meaningful order, but their numerical distances need not be equal. Treating low, medium, and high as equally spaced is not required.

  • Proportional odds: The same beta applies at every threshold. Consequently, category boundaries are parallel in feature space.

  • Independent observations: Standard likelihood estimation assumes observations do not influence one another.

  • Linear predictor: Predictors enter the log-odds through a linear function unless nonlinear terms are added.

Approach How it works Strength Limitation
Ordinal logit Shares coefficients across cumulative thresholds Compact and interpretable Requires proportional odds
Ordinal probit Uses a normal latent variable Models continuous latent severity Odds are not directly produced
Partial proportional odds Allows coefficients to vary by threshold More flexible Requires more parameters
Continuation ratio Models sequential category decisions Suitable for staged outcomes Makes a sequential assumption

Do not simply encode ordered categories as integers and use multinomial logistic regression, because that treats them as nominal and ignores their ordering. In scikit-learn 1.7+, LogisticRegression uses multinomial loss by default for multiclass problems with the lbfgs solver; the multi_class option was deprecated in 1.5 and removed in 1.7.

Practical Guidance

  • Split the data before fitting preprocessing or the model. For compatible scikit-learn estimators, use a Pipeline so transformations are learned only from each training fold.

  • statsmodels.OrderedModel is not a drop-in scikit-learn classifier: it requires the outcome during initialization and returns a probability matrix from predict. An adapter is needed if it is used inside scikit-learn cross-validation or a Pipeline.

  • Evaluate with accuracy, per-category precision and recall, and weighted kappa. Mean absolute error in category levels is appropriate only if differences between adjacent categories have a meaningful numerical interpretation.

  • For a binary thresholded task, also inspect F1, ROC-AUC, and PR-AUC, especially with severe class imbalance. Accuracy can conceal poor minority-class performance.

  • Compare the proportional-odds model with a partial proportional-odds alternative when the proportional-odds assumption is questionable.

  • For high-dimensional data, use an ordinal implementation that supports regularization. statsmodels.OrderedModel does not expose standard scikit-learn-style L1 or L2 penalty controls.

  • Check calibration when predicted probabilities will support decisions or risk estimates, and use cost-sensitive evaluation when errors at different distances have different consequences.

Common Pitfalls

  • Ignoring ordinal structure: Nominal logistic regression ignores meaningful category order.

  • Assuming equal intervals: A one-level increase from low to medium may differ from medium to high.

  • Violating proportional odds: Shared coefficients may fit poorly when predictor effects differ by threshold.

  • Using accuracy alone: It can conceal poor performance on rare categories.

  • Leaking preprocessing information: Fit scaling, encoding, and imputation on training data only.

  • Overconfident p-values: Statistical inference still depends on model assumptions, independent observations, adequate category sample sizes, and the proportional-odds assumption.

29

What is the role of quasi-likelihood in logistic regression?

Quasi-likelihood is an estimating function for a response’s conditional mean that resembles a log-likelihood but need not correspond to a normalized probability distribution. In ordinary binary logistic regression, the Bernoulli log-likelihood is already a canonical quasi-likelihood, so unpenalized fitting produces the same coefficients as maximum likelihood. Its main trade-off is flexibility under variance or dependence misspecification versus inference based on estimating equations, which can be less efficient when the full likelihood is correctly specified.

Read full answer

Applies tomachine learning concepts (all versions)

Logistic Regression Model

For observation i, logistic regression defines a linear predictor and converts it into a probability:

eta_i = beta_0 + beta^T * x_i

mu_i = 1 / (1 + exp(-eta_i))

  • y_i is the binary outcome, either 0 or 1.

  • x_i is the observation’s feature vector.

  • beta_0 is the intercept.

  • beta contains the coefficient vector.

  • beta^T * x_i is the linear combination of features and coefficients.

  • eta_i is the linear score, or log-odds.

  • mu_i is the predicted probability that y_i = 1.

The corresponding Bernoulli log-likelihood is:

log L(beta) = sum[y_i * log(mu_i) + (1 - y_i) * log(1 - mu_i)]

Each term is the log-probability of the observed outcome. Maximizing this expression is equivalent to minimizing binary cross-entropy, or log loss. The objective is convex, although perfect separation can cause coefficients to grow without reaching a finite maximum.

Quasi-Likelihood Formulation

For a specified variance function V(mu_i), the quasi-likelihood function q_i is defined through the relationship:

derivative of q_i with respect to mu_i = (y_i - mu_i) / V(mu_i)

Unlike a log-likelihood, q_i need not be normalized into a probability distribution. For the canonical Bernoulli logit model, V(mu_i) = mu_i * (1 - mu_i), and:

q_i(mu_i) = y_i * log(mu_i) + (1 - y_i) * log(1 - mu_i)

This is exactly the Bernoulli log-probability. The coefficient estimates satisfy the estimating equation:

sum_i [(y_i - mu_i) / (mu_i * (1 - mu_i))] * [1, x_i]^T = 0

The vector [1, x_i] includes the intercept and features. For a general quasi-likelihood, the denominator is replaced by V(mu_i). Thus, quasi-likelihood provides a framework for estimating the mean through an estimating equation rather than requiring a complete probability model.

Practical Roles

  • Overdispersed grouped data: Quasi-binomial models can represent extra-binomial variation, such as successes out of several trials. A single binary 0/1 observation cannot have conditional variance other than mu * (1 - mu) under the ordinary Bernoulli model.

  • Correlated observations: If the mean model is correct, independence-based estimates stay consistent under within-cluster correlation, but their model-based standard errors are wrong. A cluster-robust sandwich estimator or GEE with a working correlation structure fixes the inference; neither helps if the dependence comes from a misspecified mean, such as an omitted cluster-level confounder.

  • Variance misspecification: If the mean equation is preserved and only dispersion differs, the coefficients may match the maximum-likelihood estimates while model-based standard errors are incorrect. In a general quasi-likelihood, however, changing V(mu) changes the estimating equation and can change the coefficients.

  • Generalized linear modeling: Quasi-likelihood supports models in which a convenient complete likelihood is unavailable or the variance does not follow the standard distribution.

  • Penalized estimation: L1 or L2 regularization produces penalized-likelihood estimates rather than ordinary unpenalized maximum-likelihood estimates.

Interpretation and Limitations

The model remains interpretable because each coefficient changes the log-odds:

log[odds(y_i = 1)] = beta_0 + beta^T * x_i

For a one-unit increase in a feature, exp(beta_j) is the estimated multiplicative change in the odds of the positive class, holding other features constant.

Quasi-likelihood does not automatically correct overdispersion, nonlinear relationships, calibration problems, or dependence. It also does not make a linear decision boundary nonlinear. If the complete Bernoulli likelihood is correctly specified, likelihood-based inference is generally more efficient than robust quasi-likelihood inference.

Common Pitfalls

  • Treating quasi-likelihood as an approximation when standard logistic regression already uses the exact Bernoulli likelihood.

  • Assuming quasi-likelihood automatically fixes overdispersion; the variance function, estimating equation, and covariance estimator must all be considered.

  • Using a sandwich covariance estimator when clustered dependence may also have biased the point estimates.

  • Using the same data for scaling, feature selection, and model evaluation, which causes leakage.

  • Interpreting fitted probabilities as perfectly calibrated, especially under severe class imbalance.

  • Ignoring complete or quasi-complete separation, which can produce extremely large coefficients.

  • Calling regularized logistic regression an ordinary maximum-likelihood fit.

  • Reporting accuracy alone on imbalanced data; consider precision, recall, F1, ROC-AUC, and PR-AUC.

30

How is logistic regression interpreted with a nonlinear dependent-variable transformation?

A nonlinear dependent-variable transformation changes the scale on which the response is modeled, so its coefficients describe that transformed scale rather than automatically the original binary outcome. Standard binary logistic regression does not transform the observed y; it applies the logit as an inverse link to the conditional mean probability. The main trade-off is a potentially better-fitting or more interpretable scale versus additional back-transformation assumptions and less direct coefficient interpretation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Distinction

For observation i, standard logistic regression calculates:

  • z_i = w · x_i + b, a linear score, where x_i is the feature vector, w contains the feature weights, b is the intercept, and · denotes the dot product.

  • p_i = 1 / (1 + e^(-z_i)), the predicted probability that y_i = 1, where e is the base of the natural logarithm.

  • log(p_i / (1 - p_i)) = z_i, the log-odds of that probability.

The logit is an inverse link function applied to the conditional mean p, not normally a transformation of the observed binary outcome. Directly calculating log(y / (1 - y)) is invalid because it is undefined at y = 0 and y = 1.

A dependent-variable transformation must be interpreted carefully:

  • Response transformation: If q = g(Y) is created before modeling and becomes continuous, standard binary logistic regression is inappropriate. If q remains binary, coefficients describe the odds of the transformed event q = 1, which must correspond to the original event before interpreting them.

  • Nonlinear link: If E[g(mu | x)] = z, where mu(x) = E[Y | x], the fitted scale is g(mu). Recovering an original-scale effect requires back-transformation; for differentiable g, d mu / d x_j = w_j / g'(mu).

  • Fractional response: For a bounded response q, a fractional logit model can use logit(E[q | x]) = w · x + b. Its coefficients describe the log-odds of the expected proportion, rather than the odds of an individual binary outcome.

Coefficient Interpretation

In standard logistic regression, w_j is the change in log-odds associated with a one-unit increase in feature j, holding the other features constant. Therefore, exp(w_j) is the odds ratio for that increase; a coefficient of 0.8 multiplies the odds by approximately e^0.8 = 2.23. The intercept is the baseline log-odds when all features equal zero.

These interpretations apply to the scale being modeled. With standardized features, a coefficient represents a one-standard-deviation increase. For a transformed response or nonlinear link, exp(w_j) is not automatically an original-scale odds ratio; the transformation must first be reversed.

Training and Practical Scope

Logistic regression maximizes the Bernoulli likelihood, equivalently minimizing binary cross-entropy or log loss:

log loss = -(1/n) * sum_i [y_i log(p_i) + (1 - y_i) log(1 - p_i)]

Here, n is the number of observations, i indexes them, y_i is the binary label, p_i is its predicted probability, and log denotes the natural logarithm. The unregularized log-loss objective is convex in the coefficients.

In scikit-learn, C is the inverse regularization strength, so larger C means weaker regularization. The default l2 penalty is supported broadly; l1 requires liblinear or saga, and elasticnet requires saga. Features should be scaled before regularized fitting so that their coefficients are comparable. For multiclass data, the lbfgs solver uses the multinomial softmax formulation by default; multi_class was deprecated in scikit-learn 1.5 and removed in 1.7, so it should not be set in current code.

Common Pitfalls

  • Applying the logit to binary y: This produces infinities at zero and one and is not standard logistic regression.

  • Misinterpreting transformed coefficients: They describe the fitted scale, not automatically probabilities or effects on the original outcome.

  • Perfect separation: Coefficients can become extremely large, especially with weak regularization.

  • Class imbalance: Accuracy can be misleading; consider precision, recall, F1, PR-AUC, and class_weight="balanced".

  • Nonlinear predictor relationships: Transform the features or use splines or a more flexible model rather than assuming a dependent-variable transformation fixes model misspecification.

  • Poor calibration: Strong discrimination does not guarantee reliable probabilities; evaluate calibration when probability estimates matter.

31

How should you preprocess data for logistic regression?

The core practice for logistic regression preprocessing is to split the data before fitting any imputation, scaling, encoding, or feature-selection steps. This prevents data leakage and produces a more stable, regularized, and interpretable model. The main trade-off is pipeline complexity versus better numerical behavior and more reliable generalization.

Read full answer

Applies toNumPy 2.x and scikit-learn 1.5+ (Python 3.12)

Recommended Preprocessing Workflow

Step Recommended action Why it matters
Remove leakage Exclude identifiers and post-outcome variables; remove exact duplicate records before splitting Prevents unrealistic validation results and duplicate observations appearing in both data sets
Split data Use a stratified train/test split for classification Preserves class proportions and prevents test information entering training
Handle missing values Impute numerical features with the median and categorical features with a frequent category Logistic regression cannot directly process NaN values
Encode categories Use OneHotEncoder(handle_unknown="ignore") or suitable grouped categories Avoids imposing an artificial order on nominal variables
Scale features Standardize continuous variables before regularization Makes penalty effects comparable and improves optimization
Inspect relationships Add transformations or interaction terms when necessary Logistic regression still has a linear decision boundary
Address imbalance Consider class_weight="balanced" and threshold tuning Accuracy alone can be misleading with minority classes

For mixed data types, use a ColumnTransformer inside a Pipeline. Every fitted transformation—including imputers, encoders, scalers, and feature selectors—must be trained only on the training fold. If multiple rows belong to the same person, customer, or other entity, use a grouped split rather than an ordinary random split.

Model Interpretation and Regularization

Logistic regression calculates a linear score:

z = w · x + b

Then converts it into a probability:

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, x is the feature vector, w contains the coefficients, and b is the intercept. The resulting decision boundary is linear in the original or engineered features.

Without regularization, training maximizes the Bernoulli likelihood, equivalently minimizing binary cross-entropy. Scikit-learn’s default L2 penalty instead minimizes regularized log loss. Both objectives are convex, although regularization changes the solution and scaling affects numerical behavior and how penalty weights compare features.

  • Coefficient interpretation: A one-unit increase in a feature changes the log-odds by its coefficient, holding other features constant.

  • Odds ratio: exp(coefficient) gives the multiplicative change in odds for a one-unit increase.

  • Scaled features: A coefficient represents the change in log-odds for a one-standard-deviation increase.

  • L2 regularization: The default penalty shrinks coefficients and reduces overfitting.

  • L1 regularization: Can set coefficients to zero; use liblinear or saga.

  • Elastic Net: Combines L1 and L2 penalties and requires saga.

  • C parameter: Controls regularization inversely; larger C means weaker regularization.

For multiclass classification, scikit-learn 1.5+ uses the lbfgs solver by default and resolves the default multi_class="auto" setting to multinomial regression for that solver. The multi_class parameter was deprecated in scikit-learn 1.5–1.6 and removed in 1.7+, so it should not be set in current versions.

Evaluation and Class Imbalance

Use metrics appropriate for the task:

  • ROC-AUC: Measures ranking quality across decision thresholds.

  • Precision: Measures how many predicted positives are correct.

  • Recall: Measures how many actual positives are detected.

  • F1: Balances precision and recall.

  • PR-AUC: Often more informative than ROC-AUC when the positive class is rare.

  • Confusion matrix: Shows the types of classification errors.

  • Accuracy: Can be misleading when one class dominates.

Choose the classification threshold based on operational costs rather than always using 0.5. For example, a medical screening system may favor recall, while a spam filter may favor precision.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load a bundled binary classification dataset.
X, y = load_breast_cancer(return_X_y=True)

# Split before fitting any learned preprocessing step.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000)),
])

# Generate out-of-fold probabilities for threshold selection.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_prob = cross_val_predict(
    pipeline,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

precision, recall, thresholds = precision_recall_curve(y_train, oof_prob)
f1_scores = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) != 0,
)

# Select the threshold with the highest out-of-fold F1 score.
best_index = np.argmax(f1_scores)
best_threshold = float(thresholds[best_index])

# Fit preprocessing and the model on the complete training set.
pipeline.fit(X_train, y_train)
test_prob = pipeline.predict_proba(X_test)[:, 1]
tuned_prediction = (test_prob >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, tuned_prediction):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_prob):.3f}")
print(f"Test PR-AUC: {average_precision_score(y_test, test_prob):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, tuned_prediction))
print(classification_report(y_test, tuned_prediction, digits=3))
print("Example probabilities:", pipeline.predict_proba(X_test[:5])[:, 1])

Common Pitfalls

  • Scaling before splitting: This leaks test-set statistics into training.

  • Fitting imputers or encoders outside a pipeline: Validation rows influence fitted transformations.

  • Leaving duplicate records across splits: The model can effectively see the same observation during training and evaluation.

  • Ignoring nonlinear relationships: Feature engineering, interactions, or another model may be needed.

  • Overfitting the validation set: Tune thresholds and hyperparameters using training-fold cross-validation, then evaluate once on the test set.

  • Perfect separation: Coefficients can become extremely large; stronger regularization usually helps.

  • Confusing discrimination with calibration: Good ROC-AUC does not guarantee accurate probability estimates. Use held-out calibration when probabilities drive decisions.

  • Assuming every feature needs scaling: Trees do not, but scaling is generally beneficial for logistic regression with penalties.

32

How do you make logistic regression scalable?

Scalable logistic regression is a linear classifier that can be trained efficiently on large datasets while retaining interpretable coefficients. It solves binary or multiclass classification by modeling log-odds, with the main trade-off being that nonlinear relationships generally require feature engineering.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Model and Objective

For binary classification, logistic regression computes a linear score and converts it to a probability:

z = w · x + b

p = sigmoid(z) = 1 / (1 + e^(-z))

  • x is one observation’s feature vector, w contains feature weights, and b is the intercept.

  • z is the linear score, p is the probability of class 1, and y is the observed binary label.

  • The boundary z = 0 is linear in the original features.

Training maximizes likelihood, equivalently minimizing binary cross-entropy:

log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the number of training observations. A regularized objective adds a term such as λ * ||w||², where λ controls penalty strength and ||w||² is the squared L2 norm of the weights.

The regularized objective is convex, meaning a global optimum exists under suitable conditions. Iterative solvers can still stop at iteration limits, and perfectly separable data can produce extremely large coefficients without sufficient regularization. For a fitted coefficient β, exp(β) is the odds ratio associated with a one-unit increase in that feature, holding other features fixed.

Solver and Data Choices

Solver How it works Strengths Weaknesses Typical use
lbfgs Quasi-Newton optimization Strong convergence; supports multinomial loss Primarily suited to dense data; no L1 or elastic-net penalty Medium-sized dense datasets
liblinear Coordinate descent Fast for small binary problems; supports L1 One-vs-rest only; not intended for very large datasets Small binary datasets or L1 models
saga Stochastic gradient with averaging Supports large sparse matrices, L1, L2, and elastic-net penalties Requires careful scaling and convergence tuning Large sparse datasets
sag Stochastic gradient with averaging Supports large sparse datasets and multinomial loss Convergence can require careful tuning Large sparse binary or multiclass datasets

Gradient-based training typically costs approximately O(n * d) per pass, while prediction costs O(d). Sparse matrices also reduce memory use when the data has many zeros.

In scikit-learn 1.5+, multi_class is deprecated. The lbfgs, sag, and saga solvers use multinomial softmax behavior by default, whereas liblinear uses one-vs-rest. Multinomial regression optimizes all classes jointly; one-vs-rest fits a separate binary model per class.

The C parameter is the inverse of regularization strength: larger C means weaker regularization. The default penalty is l2; l1 requires liblinear or saga, and elasticnet requires saga.

For data that cannot fit in memory, use SGDClassifier(loss="log_loss") with partial_fit over mini-batches. On the first call, pass classes=np.unique(y_train). Fit preprocessing on training data only and apply it consistently to every batch.

Evaluation Strategy

  • Use Pipeline so scaling and other fitted preprocessing steps learn only from training data.

  • Use StratifiedKFold cross-validation for classification model selection.

  • Keep validation data separate from threshold tuning and the final test set.

  • For imbalanced classes, inspect precision, recall, F1, ROC-AUC, and PR-AUC because accuracy can hide minority-class failures. Consider class_weight="balanced".

  • Check calibration when probabilities will drive business decisions.

  • For sparse continuous features, use StandardScaler(with_mean=False) or MaxAbsScaler; default StandardScaler centering does not support sparse input.

Code Example

This example uses saga, reserves separate validation and test partitions, and tunes an F1 decision threshold:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target

# Create independent training, validation, and test partitions.
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.50, stratify=y_temp, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(
        solver="saga",
        penalty="l2",
        C=1.0,
        max_iter=1000,
        random_state=42,
    )),
])

# Each fold fits the scaler and classifier only on its training fold.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(
    model, X_train, y_train, cv=cv,
    scoring=["accuracy", "roc_auc"]
)
print(f"CV accuracy: {results['test_accuracy'].mean():.3f}")
print(f"CV ROC-AUC: {results['test_roc_auc'].mean():.3f}")

model.fit(X_train, y_train)
validation_probabilities = model.predict_proba(X_val)[:, 1]
print("First validation probabilities:", validation_probabilities[:5])

# Select a threshold using validation data only.
thresholds = np.linspace(0.10, 0.90, 81)
f1_scores = [
    f1_score(y_val, validation_probabilities >= threshold, zero_division=0)
    for threshold in thresholds
]
threshold = thresholds[int(np.argmax(f1_scores))]

test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= threshold).astype(int)

print(f"Selected threshold: {threshold:.2f}")
print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Test F1: {f1_score(y_test, test_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))

Common Pitfalls

  • Data leakage: Fitting a scaler, imputer, or encoder before splitting contaminates evaluation data.

  • Perfect separation: Coefficients can become extremely large; review features and consider stronger regularization.

  • Wrong metrics: Accuracy can conceal poor minority-class recall.

  • Unmodeled nonlinearity: Add justified nonlinear features or compare against tree-based models.

  • Poor calibration: Well-ranked scores do not necessarily represent reliable probabilities.

  • Validation overfitting: Repeated threshold or hyperparameter adjustment makes validation results optimistic.

33

How does missing data affect logistic regression models?

Missing data occurs when one or more feature values or outcome labels are absent from the training dataset. In logistic regression, missingness can reduce usable information, bias estimated coefficients, and change prediction performance; the main trade-off is between recovering information through imputation and introducing assumptions that may distort the model.

Read full answer

Applies tologistic regression concepts (all versions); scikit-learn 1.5+ for scikit-learn-specific behavior

How Missingness Can Bias the Model

The effect depends on why values are missing:

  • Missing Completely at Random (MCAR): Missingness is unrelated to observed or unobserved information. Under a correctly specified model, complete-case analysis can still give unbiased coefficient estimates, but it loses statistical power and may substantially reduce the sample size.

  • Missing at Random (MAR): Missingness depends on observed variables. For example, income may be missing more often for certain employment groups. Deleting those rows can therefore bias coefficients and misrepresent the target population.

  • Missing Not at Random (MNAR): Missingness depends on the unobserved value itself. For example, patients with severe symptoms may be less likely to report pain intensity. Standard imputation is unlikely to correct this bias reliably.

Missing outcome labels are especially problematic because those observations cannot directly train the classifier. Missing feature values may be usable when the model has a defined missing-value strategy.

Effects on Coefficients and Predictions

For a binary outcome, logistic regression estimates P(y = 1 | x) using the linear score z = w · x + b followed by the sigmoid p = 1 / (1 + e^(-z)). Here, y is the binary outcome, x is the feature vector, w contains the coefficients, b is the intercept, z is the linear score, and p is the estimated probability of class 1. Fitting maximizes the Bernoulli likelihood, equivalently minimizing binary cross-entropy or log loss; for a linear model this objective is convex.

  • Information loss: Removing incomplete rows increases coefficient uncertainty, especially in a small dataset.

  • Biased estimates: If incomplete cases differ systematically from complete cases, coefficients may not generalize to the intended population.

  • Changed model scope: Complete-case analysis models the population represented by complete observations, which may differ from the deployment population.

  • Higher variance: A smaller effective sample makes coefficients more sensitive to sampling variation.

  • Coefficient interpretation: Each coefficient changes the log-odds of y = 1 for a one-unit increase in that feature, holding other features constant. exp(coefficient) is the corresponding odds ratio.

  • Regularization interaction: C is the inverse of regularization strength. In scikit-learn, l2 is the default penalty, l1 is supported by the liblinear and saga solvers, and elasticnet requires saga. Extreme values introduced by imputation can change coefficient shrinkage or selection, so scale features before fitting regularized models.

  • Multiclass extension: One-vs-rest treats each class as a binary problem, while multinomial logistic regression uses softmax jointly. In scikit-learn 1.5, multi_class was deprecated and lbfgs used the multinomial formulation by default for multiclass; current code should rely on solver defaults rather than set that deprecated parameter.

Common Handling Strategies

  • Complete-case deletion: Simple, but inefficient and potentially biased.

  • Mean, median, or mode imputation: Preserves rows but can dilute relationships and treats imputed values as exact.

  • Model-based imputation: Predicts missing values from other features, often iteratively, but adds complexity and assumptions.

  • Missingness indicators: Adds binary indicators for missing features, allowing the model to distinguish absence from an observed value at the cost of more features.

  • Multiple imputation: Creates several plausible datasets, fits a model on each, and combines the estimates. This is useful for statistical inference because it accounts for imputation uncertainty.

  • Native handling: Some estimators learn how to represent missing observations. Ordinary scikit-learn LogisticRegression requires missing values to be handled before fitting.

For categorical features, treating missingness as its own category can be preferable to automatically replacing it with the most frequent category, provided it is meaningful and has enough observations.

Leakage, Evaluation, and Calibration

Any preprocessing that learns from data—including imputers, scalers, and encoders—must be fitted only on the training partition and then applied to validation or test data. A scikit-learn Pipeline helps enforce this separation.

  • Include naturally occurring missing values in the cross-validation data when they occur in production.

  • Compare complete-case, imputed, and missing-indicator strategies using the same validation folds.

  • Report precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classes.

  • Check probability calibration because a model may rank cases well while assigning unreliable probabilities.

  • Account for imputation uncertainty; treating every imputed value as exact usually makes uncertainty intervals too narrow.

Common Pitfalls

  • Deleting rows indiscriminately: Check the missingness mechanism and whether incomplete cases represent a distinct population.

  • Leakage: Never fit an imputer, scaler, or encoder on the full dataset before splitting it.

  • Perfect separation: Separated classes can produce extremely large coefficients and unstable probabilities. Missing-value treatment does not itself cause separation, but poor imputation can make estimation less stable.

  • Class imbalance: Missingness can alter class prevalence. Consider metrics such as PR-AUC and, when appropriate, class_weight="balanced"; this reweights classes but does not repair missingness or selection bias.

  • Nonlinear relationships: Logistic regression remains linear in its features. Use suitable transformations, interactions, or other feature engineering when the log-odds are nonlinear.

  • Overconfident probabilities: Imputation and model fitting can produce poorly calibrated probabilities, so evaluate calibration on held-out data.

  • Tuning against the test set: Use validation folds or nested cross-validation rather than repeatedly selecting strategies on the test set.

Interview follow-ups commonly ask how MCAR, MAR, and MNAR differ, why missingness indicators can help, how uncertainty should be represented, and how preprocessing can be fitted without leakage.

34

How do I add class weights to scikit-learn LogisticRegression?

Class weighting in logistic regression gives observations from selected classes greater influence on the training loss. It helps a model learn minority classes such as fraud or disease cases. The decision boundary stays linear, but the fitted coefficients and intercept change, so the boundary can shift or rotate. The main trade-off is that the fitted probabilities reflect a shifted class prior, so calibration and the decision threshold should be validated against deployment data.

Read full answer

Applies toscikit-learn 1.5.x (Python 3.12)

How Class Weighting Works

Logistic regression computes a linear score and converts it to a probability:

z_i = w · x_i + b

σ(z_i) = 1 / (1 + e^(-z_i)) = P(y_i = 1 | x_i)

A class-weighted objective can be written as:

min_w C * sum_i c_{y_i} * log_loss(y_i, σ(z_i)) + 1/2 * ||w||²

  • w contains the feature coefficients, and x_i is observation i’s feature vector.

  • b is the intercept, which scikit-learn does not penalize by default.

  • c_{y_i} is the weight assigned to observation i’s class.

  • C is the inverse regularization strength, so a larger C means weaker regularization.

  • ||w||² is the squared L2 norm of the coefficient vector.

The weighted log-loss objective remains convex. Each coefficient β_j changes the log-odds by β_j for a one-unit increase in feature j; therefore, exp(β_j) is the corresponding odds multiplier under the fitted, reweighted class distribution.

Selecting and Comparing Weights

  • class_weight="balanced": scikit-learn computes each class weight as n_samples / (n_classes * count_per_class).

  • Manual weights: pass a dictionary such as {0: 1.0, 1: 3.0} to give class 1 three times the loss contribution of class 0.

  • sample_weight: assigns a weight to each observation. It can be combined with class_weight; scikit-learn combines the two multiplicatively.

  • Regularization: L2 is the default. Use penalty="l1" with liblinear or saga, and penalty="elasticnet" with saga.

Approach How it works Strength Weakness When to use
class_weight Scales each class’s loss contribution Simple and effective Can shift probability calibration Imbalanced classification
sample_weight Scales each observation’s loss contribution Supports row-level information Must be aligned with training rows Unequal row importance
Resampling Changes the training distribution Can improve minority representation May distort data and still require calibration Minority learning needs additional data
Threshold tuning Changes only the decision cutoff No retraining required Does not improve the learned ranking Validating an operating threshold

For multiclass classification, liblinear uses one-vs-rest, while lbfgs uses the multinomial softmax formulation. In scikit-learn 1.5, multi_class is deprecated, so omit it and select behavior through the solver.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    balanced_accuracy_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

def make_model(class_weight):
    # Scaling keeps the L2 penalty comparable across features.
    return Pipeline([
        ("scaler", StandardScaler()),
        ("clf", LogisticRegression(
            max_iter=1000,
            solver="lbfgs",
            class_weight=class_weight,
        )),
    ])

balanced_model = make_model("balanced").fit(X_train, y_train)
y_prob = balanced_model.predict_proba(X_test)[:, 1]
y_pred = balanced_model.predict(X_test)

print("Balanced class weights")
print("accuracy:", round(accuracy_score(y_test, y_pred), 3))
print("balanced accuracy:", round(balanced_accuracy_score(y_test, y_pred), 3))
print("precision:", round(precision_score(y_test, y_pred), 3))
print("recall:", round(recall_score(y_test, y_pred), 3))
print("F1:", round(f1_score(y_test, y_pred), 3))
print("ROC-AUC:", round(roc_auc_score(y_test, y_prob), 3))
print("PR-AUC:", round(average_precision_score(y_test, y_prob), 3))
print("confusion matrix:\n", confusion_matrix(y_test, y_pred))
print("first predicted probabilities:", y_prob[:5])

# Manual weighting: observations in class 1 receive twice the weight.
manual_model = make_model({0: 1.0, 1: 2.0}).fit(X_train, y_train)
manual_prob = manual_model.predict_proba(X_test)[:, 1]
print("manual-weight ROC-AUC:", round(roc_auc_score(y_test, manual_prob), 3))

# Select a threshold using out-of-fold training predictions only.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_prob = cross_val_predict(
    make_model("balanced"),
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

thresholds = np.linspace(0.05, 0.95, 91)
threshold_scores = [
    balanced_accuracy_score(y_train, (oof_prob >= threshold).astype(int))
    for threshold in thresholds
]
best_index = int(np.argmax(threshold_scores))
best_threshold = thresholds[best_index]

tuned_pred = (y_prob >= best_threshold).astype(int)
print("best threshold:", round(best_threshold, 2))
print("tuned balanced accuracy:",
      round(balanced_accuracy_score(y_test, tuned_pred), 3))
print("tuned confusion matrix:\n", confusion_matrix(y_test, tuned_pred))

Common Pitfalls

  • Leakage: "balanced" is computed from the labels passed to fit, so it is computed from each training fold rather than the complete cross-validation dataset.

  • Calibration: balanced weights change the effective class prior. Validate probabilities on representative held-out data and use calibration methods such as CalibratedClassifierCV when needed.

  • Perfect separation: large weights can produce very large coefficients. Use regularization and tune C without selecting it from test performance.

  • Metrics: accuracy can hide poor minority-class performance. Report precision, recall, F1, balanced accuracy, ROC-AUC, and PR-AUC.

  • Threshold selection: choose the threshold on training or validation predictions, then evaluate it once on the untouched test set.

35

How can logistic regression perform variable selection?

Logistic regression can perform variable selection through L1 regularization, which drives some coefficients to zero and creates a sparse model. It solves classification problems with a linear decision boundary, trading some model stability or fit for interpretability. Its main weakness is that variable selection can be unstable when predictors are highly correlated.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How L1 Performs Variable Selection

Logistic regression first calculates a linear score and converts it into a probability:

z = w·x + b

P(y = 1 | x) = 1 / (1 + exp(-z))

Training maximizes the Bernoulli likelihood, which is equivalent to minimizing binary cross-entropy, also called log loss. With L1 regularization, the objective is conceptually:

minimize_w Σ_i log(1 + exp(-y_i z_i)) + λ Σ_j |w_j|

  • x_i: feature values for observation i.

  • y_i: binary target, either 0 or 1.

  • z_i = w·x_i + b: the model’s linear score.

  • w_j: coefficient for feature j; the intercept b is generally not penalized.

  • λ: regularization strength. Larger values generally produce more zero coefficients.

  • Σ_j |w_j|: L1 penalty, which encourages coefficients to become exactly zero.

The log-loss and L1 penalty are convex, so their combined objective is convex, although the optimum need not be unique when predictors are correlated. Each coefficient represents the change in log-odds for a one-unit increase in that feature, holding other features fixed; exp(coefficient) is the corresponding odds ratio.

In scikit-learn, C is the inverse of regularization strength: a smaller C applies stronger regularization.

Penalty Choices

Penalty Variable selection Strengths Weaknesses
l1 Sets coefficients to zero Sparse, interpretable models Unstable with correlated predictors
l2 Usually does not set coefficients to zero Stable when predictors overlap Retains many variables
elasticnet Can create sparse models Combines L1 sparsity and L2 stability Requires saga and additional tuning

Use solver="liblinear" or solver="saga" for l1, and solver="saga" for elasticnet. Select C using cross-validation rather than choosing it after inspecting the test results.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    GridSearchCV,
    StratifiedKFold,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load bundled data and reserve a final test set.
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.25,
    stratify=y_train_val,
    random_state=42,
)

# Scaling and logistic regression are fitted only within each CV fold.
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    (
        "logisticregression",
        LogisticRegression(
            penalty="l1",
            solver="liblinear",
            max_iter=1000,
            random_state=42,
        ),
    ),
])

search = GridSearchCV(
    pipeline,
    param_grid={"logisticregression__C": np.logspace(-3, 2, 12)},
    scoring="neg_log_loss",
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    n_jobs=-1,
)
search.fit(X_train, y_train)
model = search.best_estimator_

# Nonzero coefficients identify variables retained by L1.
coefficients = model.named_steps["logisticregression"].coef_[0]
selected = np.abs(coefficients) > 1e-8

# Tune the decision threshold on validation data, not test data.
validation_probabilities = model.predict_proba(X_val)[:, 1]
thresholds = np.linspace(0.1, 0.9, 161)
f1_scores = [
    f1_score(
        y_val,
        (validation_probabilities >= threshold).astype(int),
        zero_division=0,
    )
    for threshold in thresholds
]
best_threshold = float(thresholds[np.argmax(f1_scores)])

# Final evaluation on untouched test data.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

print("Selected C:", search.best_params_["logisticregression__C"])
print("Selected features:", feature_names[selected].tolist())
print("Coefficients:", dict(zip(feature_names[selected], coefficients[selected])))
print("Decision threshold:", best_threshold)
print("First five test probabilities:", test_probabilities[:5])
print("Accuracy:", accuracy_score(y_test, test_predictions))
print("Precision:", precision_score(y_test, test_predictions, zero_division=0))
print("Recall:", recall_score(y_test, test_predictions, zero_division=0))
print("F1:", f1_score(y_test, test_predictions, zero_division=0))
print("ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("PR-AUC:", average_precision_score(y_test, test_probabilities))
print("Confusion matrix:\n", confusion_matrix(y_test, test_predictions))

Practical Guidance

  • Standardize before L1: Without scaling, the penalty unfairly favors variables measured on smaller numerical scales. With standardized features, exp(coefficient) represents the odds ratio for a one-standard-deviation increase in that feature, holding other features fixed.

  • Separate threshold tuning from feature selection: The probability threshold controls classification decisions but does not change the fitted coefficients.

  • Assess selection stability: Repeated cross-validation or bootstrap resampling can reveal whether the same variables are consistently retained.

  • For multiclass targets: In scikit-learn 1.5–1.6, lbfgs uses multinomial softmax by default and multi_class was deprecated. The parameter was removed in scikit-learn 1.7, so do not pass it to current releases. Use saga with penalty="l1" for multinomial L1 or fit one-vs-rest binary models when class-specific selection is required.

Common Pitfalls

  • Data leakage: Do not fit the scaler or model on the full dataset before splitting or cross-validation.

  • Correlated predictors: L1 may select one variable arbitrarily from a correlated group, so coefficient signs and exact selections can change.

  • Perfect separation: It can produce extremely large coefficients; regularization or additional data may be needed.

  • Class imbalance: Consider class_weight="balanced" and prioritize PR-AUC, precision, recall, or F1 because accuracy can be misleading.

  • Nonlinear relationships: Logistic regression needs meaningful feature engineering when relationships are nonlinear.

  • No causal interpretation: A retained or excluded predictor does not establish causal importance.

36

How do I implement logistic regression from scratch with NumPy?

Logistic regression is a linear classification model that estimates the probability of a binary outcome by applying the sigmoid function to a linear score. Its main trade-off is an interpretable, linear decision boundary that may be inadequate for nonlinear relationships.

Read full answer

Applies toPython 3.12 with NumPy 2.x and scikit-learn 1.5+

How It Works

For each observation, the model computes:

z = w . x + b

Here, w is the coefficient vector, x is the feature vector, . denotes a dot product, and b is the intercept. The sigmoid converts the score into a probability:

p = 1 / (1 + e^(-z))

A common classification rule predicts 1 when p >= 0.5 and 0 otherwise, although the threshold can be tuned for the desired precision-recall balance. The resulting decision boundary is linear in the original feature space.

Each coefficient changes the log-odds for a one-unit increase in that feature while holding other features constant. On the scale used by the fitted model, exp(coefficient) is the corresponding odds ratio.

Training Objective

Binary cross-entropy, also called log loss, is:

BCE = -1/n * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the number of training examples, y is the true binary label, and p is the predicted probability. Minimizing BCE maximizes the likelihood of the observed labels. The unregularized objective is convex, so optimization is generally reliable, although gradient descent can converge slowly.

Code Example

This implementation uses NumPy for the model, standardization, and gradient descent. Stratified out-of-fold predictions are used to tune the threshold without leaking test information.

import numpy as np
from sklearn.metrics import f1_score
from sklearn.model_selection import StratifiedKFold, train_test_split

rng = np.random.default_rng(42)


def sigmoid(z):
    z = np.clip(z, -500, 500)
    return 1 / (1 + np.exp(-z))


n_samples, n_features = 1_000, 5
X = rng.normal(size=(n_samples, n_features))
true_logit = (
    1.5 * X[:, 0]
    - 1.2 * X[:, 1]
    + 0.7 * X[:, 2]
    + 0.4 * X[:, 3]
    - 0.8 * X[:, 4]
)
y = (rng.random(n_samples) < sigmoid(true_logit)).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)


def fit_logistic_regression(
    X_fit, y_fit, learning_rate=0.1, regularization=0.01, epochs=5_000
):
    mean = X_fit.mean(axis=0)
    std = X_fit.std(axis=0)
    std[std == 0] = 1.0
    X_scaled = (X_fit - mean) / std

    weights = np.zeros(X_fit.shape[1])
    intercept = 0.0

    for _ in range(epochs):
        probabilities = sigmoid(X_scaled @ weights + intercept)
        error = probabilities - y_fit
        weights -= learning_rate * (
            (X_scaled.T @ error) / len(y_fit)
            + regularization * weights
        )
        intercept -= learning_rate * error.mean()

    return weights, intercept, mean, std


def predict_probabilities(X_data, weights, intercept, mean, std):
    return sigmoid(((X_data - mean) / std) @ weights + intercept)


# Generate out-of-fold probabilities for threshold selection.
splitter = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = np.zeros(len(y_train))

for fit_idx, validation_idx in splitter.split(X_train, y_train):
    weights, intercept, mean, std = fit_logistic_regression(
        X_train[fit_idx], y_train[fit_idx]
    )
    oof_probabilities[validation_idx] = predict_probabilities(
        X_train[validation_idx], weights, intercept, mean, std
    )

thresholds = np.linspace(0.1, 0.9, 161)
f1_scores = [
    f1_score(
        y_train,
        (oof_probabilities >= threshold).astype(int),
        average="binary",
        zero_division=0,
    )
    for threshold in thresholds
]
best_threshold = thresholds[int(np.argmax(f1_scores))]

# Fit the final model on all training data, then evaluate once on the test set.
weights, intercept, mean, std = fit_logistic_regression(X_train, y_train)
test_probabilities = predict_probabilities(
    X_test, weights, intercept, mean, std
)
test_predictions = (test_probabilities >= best_threshold).astype(int)

accuracy = np.mean(test_predictions == y_test)
epsilon = 1e-15
log_loss = -np.mean(
    y_test * np.log(np.clip(test_probabilities, epsilon, 1))
    + (1 - y_test) * np.log(np.clip(1 - test_probabilities, epsilon, 1))
)
confusion_matrix = np.array([
    [
        np.sum((y_test == 0) & (test_predictions == 0)),
        np.sum((y_test == 0) & (test_predictions == 1)),
    ],
    [
        np.sum((y_test == 1) & (test_predictions == 0)),
        np.sum((y_test == 1) & (test_predictions == 1)),
    ],
])

print("Accuracy:", accuracy)
print("Log loss:", log_loss)
print("Best threshold:", best_threshold)
print("Confusion matrix:", confusion_matrix)

A production workflow can use scikit-learn’s Pipeline so preprocessing is fitted only on the training data:

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1_000)),
])
model.fit(X_train, y_train)

validation_probabilities = model.predict_proba(X_val)[:, 1]
thresholds = np.linspace(0.1, 0.9, 161)
f1_scores = [
    f1_score(
        y_val,
        (validation_probabilities >= threshold).astype(int),
        average="binary",
        zero_division=0,
    )
    for threshold in thresholds
]
best_threshold = thresholds[int(np.argmax(f1_scores))]

test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

print("Accuracy:", accuracy_score(y_test, test_predictions))
print("ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("Average precision:", average_precision_score(y_test, test_probabilities))
print("Precision:", precision_score(y_test, test_predictions))
print("Recall:", recall_score(y_test, test_predictions))
print("F1:", f1_score(y_test, test_predictions))
print("Confusion matrix:", confusion_matrix(y_test, test_predictions))
print("Best threshold:", best_threshold)

Practical Notes

  • Regularization: C is the inverse of regularization strength. The default penalty is l2; use l1 with liblinear or saga, and elasticnet with saga. Scale features first.

  • Thresholding: predict_proba supports threshold selection, but tune the threshold on validation or cross-validation data rather than on the test set.

  • Imbalance: Accuracy can mislead when classes are imbalanced. Consider class_weight="balanced", precision, recall, F1, ROC-AUC, and PR-AUC.

  • Calibration: Predicted probabilities may require calibration when reliable probability estimates are important.

For multiclass targets, scikit-learn supports two main formulations:

Approach How it works Typical use
One-vs-rest Fits one binary classifier per class liblinear; simple binary models per class
Multinomial Jointly fits a softmax model over all classes Current lbfgs default; coherent multiclass probabilities

The multi_class parameter was deprecated in scikit-learn 1.5, so leave it unset: lbfgs fits a multinomial model for multiclass targets and liblinear uses one-vs-rest.

Common Pitfalls

  • Data leakage: Fit scaling, encoding, and imputation only on training folds using Pipeline or ColumnTransformer.

  • Perfect separation: Coefficients can become extremely large; regularization helps control them.

  • Missing values: Impute missing values inside the training pipeline rather than before splitting.

  • Nonlinear relationships: Add meaningful interaction or polynomial features, or use a more flexible model.

  • Overfitting the validation set: Tune thresholds and hyperparameters using validation or cross-validation data, then evaluate once on untouched test data.

37

How do you standardize data for logistic regression in Python?

Data standardization rescales each numerical feature using the training set’s mean and standard deviation. For logistic regression, it improves numerical conditioning and applies regularization more consistently across features while preserving the model’s linear decision boundary. The main trade-off is that coefficients are interpreted per standard deviation rather than per original unit, and the scaling statistics must come only from training data to prevent leakage.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Standardization Works

For each feature, StandardScaler calculates:

x_scaled = (x - mean_train) / standard_deviation_train

  • x: Original feature value.

  • mean_train: Mean of that feature in the training set.

  • standard_deviation_train: Population standard deviation calculated by scikit-learn from the training set.

  • x_scaled: Standardized feature value.

The transformed training features have a mean of approximately 0 and a standard deviation of 1. The scaler learns these values during fit and reuses them during transform, preventing validation or test information from entering the model.

Logistic Regression Context

Logistic regression computes a linear score:

z = w · x + b

The sigmoid converts that score into the positive-class probability:

P(y = 1 | x) = 1 / (1 + e^(-z))

Training minimizes binary cross-entropy, also called log loss. This objective is convex, which makes optimization easier than for many non-convex models.

With standardized inputs:

  • A coefficient represents the change in log-odds for a one-standard-deviation increase in that feature, holding other features constant.

  • exp(coefficient) is the corresponding odds ratio.

  • C is the inverse of regularization strength: larger C means weaker regularization.

  • l2 is the default penalty. Use liblinear or saga for l1, and saga for elasticnet.

Code Example

This example uses a Pipeline so that StandardScaler is fitted only on each fold’s training data. Cross-validation provides a stratified model estimate, the validation set selects the classification threshold, and the untouched test set provides the final evaluation.

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load a bundled binary classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Create 60% training, 20% validation, and 20% test splits.
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.40, random_state=42, stratify=y
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp,
    y_temp,
    test_size=0.50,
    random_state=42,
    stratify=y_temp,
)

model = Pipeline(
    steps=[
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=1000, penalty="l2")),
    ]
)

# Each fold fits the scaler only on its own training partition.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(model, X_train, y_train, cv=cv, scoring="roc_auc")

model.fit(X_train, y_train)
val_probabilities = model.predict_proba(X_val)[:, 1]
test_probabilities = model.predict_proba(X_test)[:, 1]

# Select the threshold using validation data, not test data.
thresholds = np.linspace(0.05, 0.95, 181)
best_threshold = max(
    thresholds,
    key=lambda threshold: f1_score(
        y_val,
        val_probabilities >= threshold,
        zero_division=0,
    ),
)

test_predictions = (test_probabilities >= best_threshold).astype(int)

precision_curve, recall_curve, _ = precision_recall_curve(
    y_test, test_probabilities
)
pr_auc = auc(recall_curve, precision_curve)

print(f"Mean CV ROC-AUC: {cv_results['test_score'].mean():.3f}")
print(f"Selected threshold: {best_threshold:.3f}")
print(f"Accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Precision: {precision_score(y_test, test_predictions, zero_division=0):.3f}")
print(f"Recall: {recall_score(y_test, test_predictions, zero_division=0):.3f}")
print(f"F1: {f1_score(y_test, test_predictions, zero_division=0):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"PR-AUC: {pr_auc:.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("First five positive-class probabilities:")
print(test_probabilities[:5])

Common Pitfalls

  • Data leakage: Calling fit_transform on the complete dataset before splitting exposes test information to the scaler.

  • Validation overfitting: Repeatedly tuning the threshold on one validation set can overfit it; use an independent validation set or out-of-fold predictions and keep the test set untouched.

  • Missing values: StandardScaler does not impute missing data. Add an imputer before it when necessary.

  • Perfect separation: A feature that perfectly determines the class can produce very large coefficients; regularization helps control them.

  • Class imbalance: Accuracy can be misleading when classes are imbalanced. Consider class_weight="balanced", precision, recall, F1, PR-AUC, and thresholds based on the costs of false positives and false negatives.

  • Probability calibration: Standardization does not guarantee calibrated probabilities. Evaluate calibration when probability quality is important.

  • Non-linear relationships: Logistic regression remains linear in its inputs. Feature engineering may be needed for non-linear decision boundaries.

Interview Tips

  • Explain that scaling is especially important with l1, l2, or elasticnet penalties because unscaled features can receive inconsistent effective penalties.

  • State that scaling is recommended for gradient-based optimization, although logistic regression can still converge without it.

  • For multiclass logistic regression, recent scikit-learn versions use a multinomial formulation by default with the lbfgs solver; the multi_class parameter is deprecated.

  • Expect follow-up questions about how a Pipeline prevents leakage, how threshold tuning differs from probability estimation, and why accuracy may be unsuitable for imbalanced data.

38

How do you calculate the ROC-AUC curve for logistic regression in Python?

ROC-AUC measures how well a logistic regression model ranks positive examples above negative examples across all classification thresholds. Its main trade-off is that it summarizes threshold-independent discrimination but does not identify the best operating threshold or show whether predicted probabilities are well calibrated.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Logistic regression computes a linear score and converts it into a positive-class probability:

P(y = 1 | x) = 1 / (1 + e^(-z)), where z = w · x + b

Here, P denotes conditional probability, x is the input feature vector, w contains the learned coefficients, b is the intercept, z is the linear score, and e is Euler’s number. Training maximizes the likelihood of the observed labels, equivalently minimizing binary cross-entropy or log loss; this objective is convex.

Each coefficient measures the change in log-odds for a one-unit increase in the corresponding model input. For standardized inputs, this is a one-standard-deviation increase; exp(coefficient) gives the associated odds ratio.

The ROC curve plots:

  • True positive rate (TPR): the proportion of actual positives classified as positive.

  • False positive rate (FPR): the proportion of actual negatives classified as positive.

Lowering the threshold usually increases TPR but may also increase FPR. ROC-AUC is the area under this curve: 1.0 indicates perfect ranking, while 0.5 indicates random ranking.

For multiclass classification, scikit-learn’s LogisticRegression uses multinomial logistic regression, equivalent to softmax, by default with the lbfgs solver. Evaluate multiclass ROC-AUC in a one-vs-rest form per class and aggregate the results; the multi_class parameter was deprecated beginning in scikit-learn 1.5 and should not be set.

Calculation Steps

  1. Split the data into training and test sets before fitting preprocessing.

  2. Fit scaling and logistic regression inside a Pipeline.

  3. Use predict_proba() to obtain positive-class scores.

  4. Call roc_curve() for the curve coordinates and auc() for its area.

Code Example

import numpy as np

from sklearn.base import clone
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    classification_report,
    confusion_matrix,
    precision_recall_curve,
    roc_curve,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


# Load a bundled binary dataset with labels 0 and 1.
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the final test set isolated from all fitting operations.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

candidate_model = Pipeline(
    [
        ("scaler", StandardScaler()),
        (
            "classifier",
            LogisticRegression(max_iter=1000, random_state=42),
        ),
    ]
)

# Generate out-of-fold predictions for threshold selection.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_proba = cross_val_predict(
    candidate_model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

# Select the threshold maximizing Youden's J: TPR - FPR.
oof_fpr, oof_tpr, oof_thresholds = roc_curve(
    y_train,
    oof_proba,
    drop_intermediate=False,
)
finite = np.isfinite(oof_thresholds)
best_index = np.flatnonzero(finite)[
    np.argmax(oof_tpr[finite] - oof_fpr[finite])
]
tuned_threshold = float(oof_thresholds[best_index])

# Fit the final model on all training data.
model = clone(candidate_model).fit(X_train, y_train)

# Positive-class probabilities drive ROC and PR analysis.
test_proba = model.predict_proba(X_test)[:, 1]
fpr, tpr, _, roc_auc = roc_curve(
    y_test,
    test_proba,
    drop_intermediate=False,
)

default_predictions = (test_proba >= 0.5).astype(int)
tuned_predictions = (test_proba >= tuned_threshold).astype(int)

# Area under the precision-recall curve.
pr_precision, pr_recall, _ = precision_recall_curve(y_test, test_proba)
pr_auc = auc(pr_recall[::-1], pr_precision[::-1])

print(f"Accuracy at 0.50: {accuracy_score(y_test, default_predictions):.3f}")
print(f"Accuracy at tuned threshold: {accuracy_score(y_test, tuned_predictions):.3f}")
print(f"ROC-AUC: {roc_auc:.3f}")
print(f"PR-AUC: {pr_auc:.3f}")
print("\nClassification report at 0.50:")
print(classification_report(y_test, default_predictions, digits=3))
print("Confusion matrix at 0.50:")
print(confusion_matrix(y_test, default_predictions, labels=[0, 1]))
print("\nFirst five positive-class probabilities:")
print(test_proba[:5])

Practical Guidance

  • Scaling: StandardScaler is appropriate for L2-regularized logistic regression using iterative optimization. Keeping it in a pipeline prevents preprocessing leakage.

  • Thresholding: 0.5 is the usual default, but Youden’s TPR - FPR is only one choice. Medical screening may prioritize sensitivity instead.

  • Regularization: The default penalty is L2, and C is the inverse of regularization strength; smaller C means stronger regularization.

  • Imbalance: Consider class_weight="balanced" and examine precision, recall, F1, and PR-AUC. Accuracy can mislead when one class dominates.

  • Limitations: The decision boundary is linear, so feature engineering or another model may be needed for nonlinear relationships. High ROC-AUC does not imply calibrated probabilities.

Common Pitfalls

  • Data leakage: Never fit the scaler, imputer, or feature selection on the full dataset before splitting or before cross-validation folds.

  • Threshold leakage: Do not select a threshold using final test-set results.

  • Perfect separation: It can produce extreme coefficient estimates; regularization and diagnostics are important.

  • Confusing ranking with classification: predict_proba() provides scores for curve analysis, while predict() applies the default classification threshold.

39

How do I one-hot encode features for logistic regression?

One-hot encoding represents each nominal category with binary indicator columns, so logistic regression can use it without treating category labels as ordered numbers. Logistic regression predicts a binary probability from a linear score and is interpretable, but its decision boundary is linear and can underfit nonlinear relationships. In practice, fit the encoder only on training data and place it in a pipeline with the model.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Logistic regression calculates a linear score:

z = w · x + b

It then converts that score into a probability:

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, x is the feature vector, w contains the feature weights, b is the intercept, z is the linear decision score, and e is Euler’s number. A threshold, commonly 0.5, converts the probability into a class prediction.

Training minimizes binary cross-entropy:

BCE = -[y * log(p) + (1 - y) * log(1 - p)]

Here, y is the true class (0 or 1), p is the predicted probability of class 1, and log is the natural logarithm. This objective is convex, so a suitable solver can find a global optimum.

  • One-hot encoding: Creates binary indicators for category levels. For a linear model, drop="first" commonly omits one level per categorical feature to provide a reference and avoid redundant columns with the intercept.

  • Numeric scaling: StandardScaler is applied to numeric columns because regularization is sensitive to feature scale. One-hot indicators generally do not need scaling.

  • Regularization: LogisticRegression uses L2 regularization by default. C is the inverse of regularization strength: larger C means less regularization.

  • Coefficients: A numeric coefficient changes log-odds per one unit of the scaled feature. A one-hot coefficient compares a category with the omitted reference category. exp(coefficient) is the corresponding odds ratio.

  • Multiclass: In scikit-learn 1.5–1.6, multi_class was deprecated and was removed in 1.7. With lbfgs, multiclass logistic regression uses multinomial loss by default.

Code Example

The bundled breast-cancer data contain no nominal predictor, so this example creates a fixed radius band to demonstrate the same encoding pattern used for a real categorical column.

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


data = load_breast_cancer(as_frame=True)

X = data.frame.copy()
X["radius_band"] = pd.cut(
    X["mean radius"],
    bins=[float("-inf"), 14.0, 20.0, float("inf")],
    labels=["narrow", "medium", "wide"],
)
X = X.drop(columns=["mean radius"])
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

numeric_features = [column for column in X.columns if column != "radius_band"]
categorical_features = ["radius_band"]

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_features),
        (
            "categorical",
            OneHotEncoder(
                drop="first",
                handle_unknown="ignore",
                sparse_output=False,
            ),
            categorical_features,
        ),
    ],
    remainder="drop",
)

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000)),
])

# Select the threshold from out-of-fold training predictions.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

precision, recall, thresholds = precision_recall_curve(
    y_train,
    oof_probabilities,
)
f1_values = (
    2 * precision[:-1] * recall[:-1]
    / (precision[:-1] + recall[:-1] + 1e-12)
)
best_threshold = thresholds[np.argmax(f1_values)]

# Fit the final pipeline on all training data and evaluate once on test data.
model.fit(X_train, y_train)
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

feature_names = model.named_steps["preprocessor"].get_feature_names_out()
print("One-hot columns:", [name for name in feature_names if "radius_band" in name])
print("Test probabilities (first 5):", test_probabilities[:5])
print("Selected threshold:", best_threshold)
print("Test accuracy:", accuracy_score(y_test, test_predictions))
print("Test ROC-AUC:", roc_auc_score(y_test, test_probabilities))
print("Test PR-AUC (average precision):", average_precision_score(y_test, test_probabilities))
print("Test F1:", f1_score(y_test, test_predictions))
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions))

Evaluation Guidance

  • Accuracy is useful for reasonably balanced classes but can hide poor minority-class performance.

  • ROC-AUC measures ranking quality across thresholds.

  • PR-AUC is often more informative for rare positive classes.

  • F1 summarizes precision and recall when a fixed threshold is required.

  • A confusion matrix distinguishes false positives, false negatives, true positives, and true negatives.

Best Practices

  • Keep OneHotEncoder, StandardScaler, and LogisticRegression inside a Pipeline; each cross-validation fold then learns preprocessing only from its training partition.

  • Impute missing categorical and numeric values inside the pipeline when necessary so that imputation rules are also learned without leakage.

  • Use handle_unknown="ignore" to handle previously unseen categories at prediction time.

  • For severe imbalance, consider class_weight="balanced" and evaluate precision, recall, F1, ROC-AUC, and PR-AUC.

  • Check probability calibration on held-out data; logistic-regression probabilities are not guaranteed to be calibrated and can be recalibrated when needed.

  • Add interaction or polynomial features when the target depends on nonlinear relationships.

Common Pitfalls

  • Data leakage: Fitting the encoder or scaler before splitting exposes validation and test information to training.

  • Dummy-variable trap: Encoding every category alongside an intercept creates redundant columns; use a reference level when coefficient interpretation matters.

  • Unseen categories: Omitting handle_unknown="ignore" can cause prediction errors.

  • Perfect separation: Coefficients may become extremely large and unstable; increase regularization or inspect the data.

  • Wrong metric: High accuracy can conceal poor recall or precision for an imbalanced target.

  • Overfitting: Excessive flexibility or repeatedly tuning against the test set produces optimistic results.

40

How do you tune logistic regression C with cross-validation in Python?

Cross-validation tunes logistic regression’s C by evaluating candidate values on validation folds and selecting the one with the best mean score. Because C is the inverse of regularization strength, larger C means weaker regularization and more flexible fits; the main trade-off is training fit versus generalization.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Logistic regression forms z = w · x + b, where x is a feature vector, w contains the weights, · is the dot product, and b is the intercept. The sigmoid p = 1 / (1 + e^(-z)) converts this score into P(y = 1 | x), giving logistic regression a linear decision boundary.

Training maximizes likelihood, equivalently minimizing log loss:

log_loss = -1/n * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i))

Here, n is the number of examples, y_i is a binary label, and p_i is its predicted probability of class 1. The regularized objective is convex.

  • Small C imposes stronger regularization and usually produces smaller coefficients.

  • Large C imposes weaker regularization and permits more flexible fits.

GridSearchCV evaluates candidate C values using cross-validation. Putting StandardScaler inside a Pipeline ensures it is fitted separately on each training fold, preventing preprocessing leakage.

For multiclass targets, one-vs-rest fits one binary model per class, while multinomial logistic regression jointly models classes with softmax. With lbfgs, current scikit-learn releases use multinomial behavior by default; multi_class was deprecated in 1.5–1.6 and removed in 1.7+.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    GridSearchCV,
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load a bundled binary-classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the test set untouched during model and threshold selection.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

# Scaling is fitted within each training fold, preventing leakage.
pipeline = Pipeline(
    [
        ("scale", StandardScaler()),
        (
            "model",
            LogisticRegression(
                solver="lbfgs",
                penalty="l2",
                max_iter=1000,
            ),
        ),
    ]
)

param_grid = {"model__C": np.logspace(-3, 3, 13)}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

search = GridSearchCV(
    estimator=pipeline,
    param_grid=param_grid,
    scoring="roc_auc",
    cv=cv,
    refit=True,
    n_jobs=-1,
)
search.fit(X_train, y_train)

best_model = search.best_estimator_
best_C = search.best_params_["model__C"]
print(f"Best C: {best_C:.6g}")
print(f"Best CV ROC-AUC: {search.best_score_:.4f}")

# predict_proba returns one probability per class; column 1 is class 1.
test_prob = best_model.predict_proba(X_test)[:, 1]
print("First five test probabilities:", test_prob[:5])

# Select an F1-maximizing threshold using cross-fitted training probabilities.
oof_prob = cross_val_predict(
    best_model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

precision, recall, thresholds = precision_recall_curve(y_train, oof_prob)
denominator = precision[:-1] + recall[:-1]
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    denominator,
    out=np.zeros_like(denominator),
    where=denominator > 0,
)
best_index = int(np.argmax(f1))
best_threshold = float(thresholds[best_index])

# Evaluate the selected model and threshold only on the untouched test set.
test_pred = (test_prob >= best_threshold).astype(int)
print(f"Selected threshold: {best_threshold:.4f}")
print(f"Accuracy: {accuracy_score(y_test, test_pred):.4f}")
print(
    f"Precision: {precision_score(y_test, test_pred, zero_division=0):.4f}"
)
print(f"Recall: {recall_score(y_test, test_pred, zero_division=0):.4f}")
print(f"F1: {f1_score(y_test, test_pred, zero_division=0):.4f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_prob):.4f}")
print(
    f"PR-AUC: {average_precision_score(y_test, test_prob):.4f}"
)
print(
    "Confusion matrix:\n",
    confusion_matrix(y_test, test_pred, labels=[0, 1]),
)

Interpreting the Results

  • Best C: The candidate with the highest mean validation ROC-AUC.

  • ROC-AUC: Measures probability ranking across thresholds.

  • Threshold tuning: The default threshold is 0.5, but the best threshold depends on the cost of false positives versus false negatives. F1 is appropriate when precision and recall matter similarly.

  • Coefficients: For a non-intercept coefficient beta, exp(beta) is the estimated odds ratio for a one-unit increase in that feature, holding other features constant.

Best Practices

  • Scale features: Regularization penalizes coefficient magnitude, so features with larger scales can be penalized disproportionately.

  • Handle missing values: Fit an imputer inside the pipeline before scaling and modeling.

  • Address imbalance: Consider class_weight="balanced" and report precision, recall, F1, and PR-AUC because accuracy can be misleading.

  • Avoid leakage: Split before fitting scalers, imputers, encoders, or feature selectors; never tune against test results.

  • Assess calibration: A model selected by ROC-AUC is not necessarily well calibrated.

  • Use nested CV when needed: If the test set is unavailable, nested CV provides a more honest estimate of the complete model-selection procedure.

Common Pitfalls

  • Reversing C: Larger C means weaker regularization.

  • Scaling before splitting: This leaks test-set statistics into training.

  • Tuning repeatedly on the test set: This turns the test set into training data.

  • Ignoring separation: Perfect separation can produce extreme coefficients and poor generalization.

  • Missing nonlinear structure: Add suitable transformations or interactions when relationships are nonlinear.

41

How to interpret logistic regression coefficients as odds ratios?

Logistic regression predicts binary probabilities from a linear score. Each coefficient measures a change in log-odds, so exponentiating it gives the multiplicative change in odds for one unit of the modeled feature. Its main trade-off is transparent, interpretable coefficients versus an inability to represent nonlinear relationships without feature engineering.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For a binary outcome:

p = P(y = 1 | x)

odds = p / (1 - p)

z = b + sum(w_j * x_j)

p = 1 / (1 + exp(-z))

  • p is the predicted probability of class 1; y is the binary outcome.

  • x_j is feature j, and w_j is its coefficient.

  • b is the intercept.

  • j indexes features, and sum runs over all features.

  • z is the linear score or log-odds; exp is the exponential function.

The decision boundary is linear. Training maximizes the likelihood, equivalently minimizing binary cross-entropy or log loss, which is convex in the coefficients.

Interpreting Coefficients

The odds ratio for feature j is:

odds_ratio_j = exp(w_j)

Holding the other features constant, a one-unit increase in the modeled feature multiplies the odds of class 1 by this factor. An odds ratio above 1 increases the odds, one below 1 decreases them, and one has no multiplicative effect.

For example, a coefficient of 0.70 gives exp(0.70) ≈ 2.01, so the odds are multiplied by about 2.01. This is not a probability ratio because probability also depends on the starting probability and the other features.

The intercept is the log-odds when every modeled feature is zero; exp(intercept) is the corresponding baseline odds. With StandardScaler, zero means the feature’s training-set mean, so a coefficient represents a one-training-standard-deviation change rather than a raw-unit change. Binary predictions normally use a 0.5 probability threshold, although predict_proba allows the threshold to be tuned.

Multiclass, Regularization, and Evaluation

For multiclass logistic regression, coefficient interpretations depend on the formulation:

Formulation Coefficient meaning
One-vs-rest Odds of one class against all remaining classes
Multinomial/softmax Odds of one class relative to a reference class

With the lbfgs solver, scikit-learn automatically uses multinomial regression for multiclass problems. The multi_class option was deprecated in scikit-learn 1.5 and removed in later releases, so it should not be set.

C is the inverse of regularization strength: larger C means weaker regularization. The default penalty is l2; l1 is supported with liblinear or saga, while elasticnet requires saga. Scale features before fitting so regularization and optimization do not depend on arbitrary feature units.

Keep validation data separate for threshold selection and test data separate for final evaluation. For imbalanced classes, emphasize precision, recall, F1, and PR-AUC because accuracy can be misleading.

Code Example

import numpy as np
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    auc,
    confusion_matrix,
    f1_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


data = load_breast_cancer()
X, y = data.data, data.target

# Reserve the test set and a separate validation set.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)
X_train, X_validation, y_train, y_validation = train_test_split(
    X_train, y_train, test_size=0.25, stratify=y_train, random_state=42
)

model = Pipeline(
    [
        ("scaler", StandardScaler()),
        ("logistic", LogisticRegression(max_iter=1000, random_state=42)),
    ]
)

# Each fold fits the scaler and model only on its training fold.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc")
print(f"CV ROC-AUC: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

model.fit(X_train, y_train)

# Coefficients use standardized feature units.
coefficients = model.named_steps["logistic"].coef_[0]
coefficient_table = pd.DataFrame(
    {
        "feature": data.feature_names,
        "coefficient_log_odds": coefficients,
        "odds_ratio": np.exp(coefficients),
    }
)
print(
    coefficient_table.sort_values("odds_ratio", ascending=False)
    .to_string(index=False)
)

# Tune the classification threshold only on validation probabilities.
validation_probabilities = model.predict_proba(X_validation)[:, 1]
thresholds = np.linspace(0.10, 0.90, 33)
f1_values = [
    f1_score(y_validation, validation_probabilities >= threshold)
    for threshold in thresholds
]
best_threshold = float(thresholds[np.argmax(f1_values)])

# Evaluate once on the untouched test set.
test_probabilities = model.predict_proba(X_test)
test_predictions = test_probabilities[:, 1] >= best_threshold

precision_curve, recall_curve, _ = precision_recall_curve(
    y_test, test_probabilities[:, 1]
)
pr_auc = auc(recall_curve[::-1], precision_curve[::-1])

print("Sample probabilities:", test_probabilities[:5, 1])
print("Selected threshold:", best_threshold)
print(f"Accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities[:, 1]):.3f}")
print(f"PR-AUC: {pr_auc:.3f}")
print(f"Precision: {precision_score(y_test, test_predictions):.3f}")
print(f"Recall: {recall_score(y_test, test_predictions):.3f}")
print(f"F1: {f1_score(y_test, test_predictions):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions))

Common Pitfalls

  • Odds are not probabilities: an odds ratio does not directly describe the change in probability.

  • Scaling changes the unit: with StandardScaler, interpret coefficients as per training-set standard deviation.

  • Perfect separation: coefficients and odds ratios can become extreme; regularization helps control them.

  • Class imbalance: consider class_weight="balanced" and prioritize suitable metrics such as PR-AUC, precision, and recall.

  • Nonlinear relationships: engineer suitable features or use a more flexible model.

  • Calibration and causality: ranking metrics do not guarantee calibrated probabilities, and odds ratios do not establish causation.

  • Common follow-up questions: How does C affect regularization? Why does scaling change coefficient interpretation? When is multinomial logistic regression preferable?

42

How do I build class-weighted logistic regression in scikit-learn?

Weighted logistic regression compensates for imbalanced classes by increasing the influence of underrepresented observations during training. In scikit-learn, class_weight="balanced" assigns class weights inversely related to class frequency, and the classifier models a linear log-odds score whose sigmoid produces class probabilities. Its main trade-off is stronger minority-class recognition at the cost of more false positives, lower majority-class accuracy, and possible degradation of probability calibration for the original class prevalence.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Model and Objective

Logistic regression computes a linear score:

z = w · x + b

Then converts it into the probability of the positive class:

P(y = 1 | x) = 1 / (1 + e^(-z))

Here, w is the coefficient vector, x is the observation’s feature vector, b is the intercept, and z is the decision score. The predicted class is selected using a threshold, commonly 0.5.

The model is trained by minimizing weighted binary cross-entropy, also called log loss. Together with L2 regularization, this objective is convex, so optimization can find a global minimum.

  • Class weighting: class_weight="balanced" uses n_samples / (n_classes * class_count) as each class’s weight.

  • Coefficient interpretation: holding other features constant, a one-unit increase in feature j changes the fitted model’s log-odds by its coefficient. exp(coefficient) is the corresponding odds multiplier.

  • Regularization: scikit-learn uses L2 regularization by default. C is the inverse of regularization strength: larger C means weaker regularization.

  • Alternative penalties: L1 requires liblinear or saga; elastic net requires saga.

  • Multiclass extension: liblinear uses one-vs-rest. In scikit-learn 1.5–1.6, multi_class was deprecated and lbfgs defaults to multinomial softmax; it was removed in 1.7, so it should not be set.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    classification_report,
    confusion_matrix,
    f1_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


# Load a bundled binary classification dataset.
data = load_breast_cancer()
X = data.data

# sklearn stores 0 = malignant and 1 = benign. Recode so that
# 1 = malignant (the minority class) and 0 = benign.
malignant_index = list(data.target_names).index("malignant")
y = (data.target == malignant_index).astype(int)

# Hold out an untouched test set before fitting any preprocessing.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

# Scaling is learned inside each training fold to prevent leakage.
model = Pipeline(
    steps=[
        ("scaler", StandardScaler()),
        (
            "classifier",
            LogisticRegression(
                class_weight="balanced",
                penalty="l2",
                solver="liblinear",
                max_iter=1000,
                random_state=42,
            ),
        ),
    ]
)

# Generate out-of-fold probabilities for threshold selection.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

# Select the threshold that maximizes minority-class F1.
candidate_thresholds = np.unique(
    np.concatenate(([0.0, 1.0], oof_probabilities))
)
f1_scores = [
    f1_score(
        y_train,
        (oof_probabilities >= threshold).astype(int),
        zero_division=0,
    )
    for threshold in candidate_thresholds
]
best_threshold = float(candidate_thresholds[np.argmax(f1_scores)])

# Refit the pipeline on the complete training partition.
model.fit(X_train, y_train)

test_probabilities = model.predict_proba(X_test)[:, 1]
default_predictions = model.predict(X_test)
tuned_predictions = (test_probabilities >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.3f}")
print(f"Default-threshold accuracy: {accuracy_score(y_test, default_predictions):.3f}")
print(f"Tuned-threshold accuracy: {accuracy_score(y_test, tuned_predictions):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, test_probabilities):.3f}")
print("Confusion matrix at the selected threshold:")
print(confusion_matrix(y_test, tuned_predictions))
print(
    classification_report(
        y_test,
        tuned_predictions,
        target_names=["benign", "malignant"],
        zero_division=0,
    )
)

Evaluation Guidance

  • Accuracy can be misleading when the minority class is rare because a model can score well by predicting only the majority class.

  • Precision measures how many predicted positives are genuine, while recall measures how many genuine positives are detected.

  • F1 balances precision and recall and is used above to select the threshold.

  • ROC-AUC measures ranking quality across thresholds; PR-AUC is often more informative for severe imbalance.

  • Calibration should be checked separately. Class weighting changes the effective class distribution in the loss, so its probabilities may require recalibration to reflect the original prevalence; decision thresholds should be evaluated separately.

Common Pitfalls

  • Using class_weight="balanced" without comparing against an unweighted baseline.

  • Selecting a threshold on the test set, which leaks test information into model selection.

  • Reporting accuracy without precision, recall, F1, ROC-AUC, or PR-AUC.

  • Expecting class weighting to create nonlinear relationships; feature engineering or a more flexible model may still be needed.

  • Interpreting coefficients from heavily regularized or standardized data without stating those conditions.

  • Failing to investigate extreme coefficients caused by perfect separation.

  • Assuming weighted probabilities are directly calibrated for the original class prevalence.

43

How do I implement multiclass logistic regression with TensorFlow and Keras?

Multiclass logistic regression is a linear classifier that assigns each example to one of several mutually exclusive classes. TensorFlow implements it as a linear layer followed by a softmax activation, producing class probabilities; its main trade-off is an interpretable linear decision boundary that cannot capture complex nonlinear relationships without feature engineering.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3 and scikit-learn 1.5+ (Python 3.12)

Model and Loss

For class k, the model computes a linear score:

z_k = w_k · x + b_k

Here, w_k is the weight vector for class k, x is the input feature vector, · denotes a dot product, and b_k is the class-specific intercept.

Softmax converts these scores into probabilities:

p_k = exp(z_k) / sum(exp(z_j) for all classes j)

The predicted class is the index with the largest probability. Training minimizes categorical cross-entropy, equivalent to maximizing the likelihood of the observed labels. The unregularized multinomial logistic-regression objective is convex, although optimization details can affect convergence.

For binary classification, one sigmoid output is equivalent to a two-logit softmax with one logit fixed at zero; the latter has a redundant shift between its logits.

With a softmax model, choose a reference class r:

log(p_k / p_r) = (w_k - w_r) · x + (b_k - b_r)

Thus, a one-unit increase in feature i changes the log-odds of class k relative to the reference class by w_k,i - w_r,i. Exponentiating that value gives the multiplicative odds ratio for the one-unit increase, holding the other features fixed.

Multiclass Approaches

Formulation How it works Strengths Weaknesses When to use Typical scale
Multinomial softmax Trains one model with K logits and normalizes them into probabilities Coherent, mutually exclusive probabilities; direct multiclass likelihood Linear decision boundaries Classes are mutually exclusive Approximately O(n * d * K) training work per epoch
One-vs-rest Trains K binary classifiers, one per class Reuses binary-classification workflows Scores may not sum to one and need separate calibration Binary tooling is preferred or classes can be treated independently Approximately O(n * d * K) total training work per epoch

Here, n is the number of training examples, d is the number of features, and K is the number of classes.

Data Preparation and Evaluation

  • Split first: Divide the data into training and test sets before fitting a scaler or model.

  • Scale features: Standardization improves optimizer conditioning and makes coefficient penalties more comparable across features.

  • Use stratified splits: This preserves class proportions, especially with smaller datasets.

  • Evaluate several metrics: Report accuracy, confusion matrix, and multiclass ROC-AUC using one-vs-rest (ovr) scoring.

  • Watch class imbalance: Accuracy can be misleading when one class dominates. Also inspect per-class precision, recall, and F1 score.

  • Cross-validate: StratifiedKFold provides a more reliable estimate, with preprocessing fitted separately within each fold.

Code Example

import numpy as np
import tensorflow as tf

from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.preprocessing import label_binarize, StandardScaler

# Reproducibility
np.random.seed(42)
tf.keras.utils.set_random_seed(42)

# Synthetic multiclass dataset
X, y = make_classification(
    n_samples=1200,
    n_features=8,
    n_informative=5,
    n_redundant=0,
    n_classes=3,
    class_sep=1.2,
    random_state=42,
)

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

n_features = X_train_val.shape[1]
n_classes = np.unique(y_train_val).size


def build_model():
    model = tf.keras.Sequential(
        [
            tf.keras.layers.Input(shape=(n_features,)),
            tf.keras.layers.Dense(n_classes, activation="softmax"),
        ]
    )

    model.compile(
        optimizer=tf.keras.optimizers.Adam(),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(
            from_logits=False
        ),
        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],
    )
    return model


# Cross-validation with fold-local preprocessing
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = []

for fold, (train_index, validation_index) in enumerate(
    cv.split(X_train_val, y_train_val), start=1
):
    fold_scaler = StandardScaler()
    X_fold_train = fold_scaler.fit_transform(X_train_val[train_index])
    X_fold_validation = fold_scaler.transform(X_train_val[validation_index])

    fold_model = build_model()
    fold_model.fit(
        X_fold_train,
        y_train_val[train_index],
        epochs=100,
        batch_size=32,
        verbose=0,
    )

    validation_probabilities = fold_model.predict(
        X_fold_validation, verbose=0
    )
    validation_predictions = validation_probabilities.argmax(axis=1)
    score = accuracy_score(y_train_val[validation_index], validation_predictions)
    cv_scores.append(score)
    print(f"Fold {fold} accuracy: {score:.3f}")

# Fit the final model using all non-test data
final_scaler = StandardScaler()
X_train_scaled = final_scaler.fit_transform(X_train_val)
X_test_scaled = final_scaler.transform(X_test)

final_model = build_model()
final_model.fit(
    X_train_scaled,
    y_train_val,
    epochs=100,
    batch_size=32,
    verbose=0,
)

# Softmax output contains one probability per class
test_probabilities = final_model.predict(X_test_scaled, verbose=0)
test_predictions = test_probabilities.argmax(axis=1)

test_accuracy = accuracy_score(y_test, test_predictions)
confusion = confusion_matrix(y_test, test_predictions)
y_test_binary = label_binarize(y_test, classes=np.arange(n_classes))
roc_auc = roc_auc_score(
    y_test_binary,
    test_probabilities,
    multi_class="ovr",
    average="macro",
)

print(f"Mean CV accuracy: {np.mean(cv_scores):.3f}")
print(f"Test accuracy: {test_accuracy:.3f}")
print(f"Test ROC-AUC (OVR macro): {roc_auc:.3f}")
print("Confusion matrix:")
print(confusion)
print("First test example probabilities:", test_probabilities[0])

Practical Considerations

  • Regularization: Add tf.keras.regularizers.L2() to the Dense layer when overfitting occurs. Standardize features so regularization does not disproportionately penalize variables measured on larger scales.

  • Class imbalance: Use sample weights during fit, or collect representative training examples rather than relying only on accuracy.

  • Nonlinear patterns: Polynomial features, interaction terms, or a more flexible neural architecture may outperform a linear model.

  • Probability quality: Softmax outputs are probabilities, but they may require calibration before being interpreted as reliable real-world risks.

  • scikit-learn comparison: In scikit-learn 1.5+, multiclass logistic regression with the lbfgs solver uses the multinomial formulation by default. The multi_class parameter is deprecated; the TensorFlow implementation above directly uses a softmax output layer.

Common Pitfalls

  • Scaling the complete dataset before splitting leaks test information into training.

  • Using independent sigmoid outputs for mutually exclusive classes does not produce normalized multiclass probabilities.

  • Match the loss to the labels: use SparseCategoricalCrossentropy for integer labels and CategoricalCrossentropy for one-hot labels.

  • Reporting only accuracy on imbalanced data can hide poor minority-class performance.

  • Training for too few epochs can underfit; excessive epochs without regularization can overfit.

  • Selecting models repeatedly against the test set turns it into a validation set and biases the reported result.

  • Interview follow-ups commonly ask about softmax versus one-vs-rest, coefficient interpretation, regularization, multiclass metrics, and data leakage.

44

How do you implement stepwise logistic regression in Python?

Stepwise logistic regression builds a logistic-regression model by repeatedly adding or removing predictors according to a performance criterion, giving a simpler, more interpretable model instead of one that uses every column. Its main trade-off is instability: the chosen variables can change with correlated predictors or small samples, and the selection itself can overfit, so cross-validation and a strict stopping rule matter.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, NumPy 2.x (Python 3.12)

How It Works

For binary classification, logistic regression forms the linear score z = w · x + b, where w is the coefficient vector, x the feature vector and b the intercept, and passes it through the sigmoid p(y = 1 | x) = 1 / (1 + e^(-z)) to give the probability of class 1. The decision boundary is therefore linear in the original feature space.

Training minimizes binary cross-entropy (log loss), log_loss = -[y log(p) + (1 - y) log(1 - p)], where y is the true 0/1 label, p the predicted probability of class 1 and the logarithms are natural; the objective is convex, so a global optimum exists.

A coefficient shifts the log-odds by its value per one-unit increase in that feature (others held fixed), and exp(coefficient) is the multiplicative change in odds. C is the inverse of regularization strength, so a larger C means a weaker penalty; the default l2 penalty fits here, while l1 needs liblinear or saga and elasticnet needs saga. For multiclass data with the lbfgs solver, current scikit-learn versions fit a multinomial (softmax) model; the multi_class parameter is deprecated.

Selection Strategies

Approach Start Candidate step
Forward selection empty model add the feature that most improves CV log loss
Backward elimination all features drop the feature whose removal hurts CV log loss least
  • Criterion: cross-validated negative log loss computed on training folds only.

  • Stopping rule: stop when no candidate improves the score (tolerance 1e-4 here).

  • Leakage control: scaler and model are refitted inside each fold, so selection never touches validation or test data.

Code Example

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, average_precision_score, confusion_matrix,
                             f1_score, precision_recall_curve, precision_score,
                             recall_score, roc_auc_score)
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler


def stepwise_logistic_regression(X, y, direction="forward", max_features=None,
                                 cv=3, C=1.0, seed=42):
    """Select features by forward/backward steps scored with CV log loss."""
    if direction not in {"forward", "backward"}:
        raise ValueError("direction must be 'forward' or 'backward'")
    y = np.asarray(y)
    if set(np.unique(y)) != {0, 1}:
        raise ValueError("labels must be encoded as 0 and 1")
    limit = X.shape[1] if max_features is None else max_features
    if not 1 <= limit <= X.shape[1]:
        raise ValueError("max_features must be between 1 and the number of columns")
    splitter = StratifiedKFold(n_splits=cv, shuffle=True, random_state=seed)

    def build():
        # Scaler is refitted inside every CV fold, so no leakage across folds
        return Pipeline([("scale", StandardScaler()),
                         ("lr", LogisticRegression(penalty="l2", C=C, solver="liblinear",
                                                  max_iter=1000, random_state=seed))])

    def score(features):
        return cross_val_score(build(), X.loc[:, features], y, cv=splitter,
                               scoring="neg_log_loss", n_jobs=-1).mean()

    history, selected = [], []
    if direction == "forward":
        remaining, current = list(X.columns), None
        while len(selected) < limit and remaining:
            scores = {f: score(selected + [f]) for f in remaining}
            best = max(scores, key=scores.get)
            if current is not None and scores[best] <= current + 1e-4:
                break
            selected.append(best)
            remaining.remove(best)
            history.append((len(selected), "added", best, scores[best]))
            current = scores[best]
    else:
        selected = list(X.columns)
        current = score(selected)
        while len(selected) > 1:
            scores = {f: score([c for c in selected if c != f]) for f in selected}
            best = max(scores, key=scores.get)
            # keep removing past the tolerance until max_features is respected
            if scores[best] <= current + 1e-4 and len(selected) <= limit:
                break
            selected.remove(best)
            history.append((len(history) + 1, "removed", best, scores[best]))
            current = scores[best]

    fitted = build().fit(X.loc[:, selected], y)   # refit on the whole training set
    return selected, fitted, history


def positive_probabilities(fitted, frame):
    lr = fitted.named_steps["lr"]
    positive = np.flatnonzero(lr.classes_ == 1)[0]
    return fitted.predict_proba(frame)[:, positive]


data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.20, stratify=y,
                                                   random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25,
                                                    stratify=y_temp, random_state=42)
selected, model, history = stepwise_logistic_regression(X_train, y_train,
                                                        max_features=8, cv=3)

# Tune the decision threshold on validation data only
val_p = positive_probabilities(model, X_val)
precision, recall, thresholds = precision_recall_curve(y_val, val_p)
best_threshold, _ = max(
    ((t, f1_score(y_val, (val_p >= t).astype(int), zero_division=0)) for t in thresholds),
    key=lambda pair: pair[1])

# Evaluate once on the untouched test set
test_p = positive_probabilities(model, X_test)
test_pred = (test_p >= best_threshold).astype(int)
print("Selected features:", selected)
print("Selection history:", history)
print("Best validation threshold:", round(float(best_threshold), 3))
print("Test accuracy:", accuracy_score(y_test, test_pred))
print("Test ROC-AUC:", roc_auc_score(y_test, test_p))
print("Test PR-AUC:", average_precision_score(y_test, test_p))
print("Test precision:", precision_score(y_test, test_pred))
print("Test recall:", recall_score(y_test, test_pred))
print("Test F1:", f1_score(y_test, test_pred))
print("Test confusion matrix:\n", confusion_matrix(y_test, test_pred))

Practical Guidance

  • Scaling matters: regularized logistic regression is sensitive to feature scale, so scale before fitting (done inside the pipeline).

  • Metrics: accuracy can mislead on imbalanced data; inspect precision, recall, F1, ROC-AUC and PR-AUC, and choose the threshold by business cost.

  • Stability: compare against an all-features model; elastic-net logistic regression is often more stable than stepwise selection when predictors are highly correlated.

Common Pitfalls

  • Selecting features on the full dataset leaks information; select on training folds only.

  • Choosing variables by training accuracy almost always selects too many.

  • Ignoring class imbalance hides poor minority recall; consider class_weight="balanced".

  • Perfect separation produces huge coefficients; regularization damps it.

  • Assuming causality: a selected coefficient is a conditional predictive association, not necessarily a causal effect.

  • Skipping calibration: high ROC-AUC does not mean probabilities are accurate.

45

How do I build polynomial-feature logistic regression with a scikit-learn Pipeline?

Polynomial logistic regression applies logistic regression to engineered polynomial features, allowing it to represent nonlinear relationships in the original inputs while remaining linear in the expanded feature space. Its main trade-off is greater flexibility versus increased overfitting risk, dimensionality, and computational cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For binary classification, each observation with feature vector x is transformed into polynomial features phi_j(x). The model computes:

z = b + sum(beta_j * phi_j(x))

P(y = 1 | x) = sigmoid(z) = 1 / (1 + exp(-z))

  • x: the original observation and y: its class label.

  • phi_j(x): an original, squared, or interaction feature.

  • beta_j: the coefficient associated with that engineered feature.

  • b: the intercept and z: the linear score passed through the sigmoid.

Binary logistic regression maximizes the likelihood of the observed labels, equivalently minimizing binary cross-entropy or log loss. This objective is convex, so suitable solvers find a global optimum.

The pipeline should contain these steps:

  1. PolynomialFeatures creates powers and interactions.

  2. StandardScaler standardizes the expanded features.

  3. LogisticRegression fits the classifier.

Using include_bias=False avoids duplicating a constant feature because logistic regression already has an intercept. With degree 2 and d original features, the expanded representation contains d + d + d(d-1)/2 features, so memory use grows quadratically with d.

Coefficients and Regularization

For binary logistic regression, each coefficient changes the predicted log-odds for a one-unit increase in its engineered feature. Its exponential, exp(beta_j), is the corresponding odds ratio. Because the example standardizes the expanded features, a one-unit increase means one standard deviation rather than one raw unit.

C is the inverse of regularization strength, so larger C means weaker regularization. The default penalty is l2; l1 requires liblinear or saga, while elasticnet requires saga. Scaling features before fitting is important when regularization is used.

Multiclass

Scikit-learn supports two multiclass formulations:

Formulation How it works Main trade-off
One-vs-rest Trains one binary classifier for each class Simple, but probabilities may not sum coherently across separately fitted models
Multinomial Uses softmax to produce one probability per class Models class competition directly and is generally preferred for multiclass logistic regression

With lbfgs, scikit-learn 1.5 made multinomial the default for multiclass problems. The multi_class parameter was deprecated in 1.5 and removed in 1.7, so current code should not specify it.

Evaluation Strategy

The example selects its threshold from out-of-fold training predictions and then evaluates the final model once on the test set. This avoids threshold-selection leakage. For imbalanced classes, inspect precision, recall, F1, ROC-AUC, and PR-AUC because accuracy alone can hide poor minority-class performance.

Code Example

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

RANDOM_STATE = 42

# In this dataset, 0 is malignant and 1 is benign.
# Remap the labels so the positive class is malignant.
X, y = load_breast_cancer(return_X_y=True)
y = (y == 0).astype(np.int64)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=RANDOM_STATE,
)

model = Pipeline(
    steps=[
        ("polynomial", PolynomialFeatures(degree=2, include_bias=False)),
        ("scaler", StandardScaler()),
        (
            "classifier",
            LogisticRegression(
                penalty="l2",
                solver="lbfgs",
                max_iter=1000,
            ),
        ),
    ]
)

# Generate leakage-safe out-of-fold probabilities for threshold selection.
cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE,
)
oof_probabilities = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
    n_jobs=-1,
)[:, 1]

candidate_thresholds = np.linspace(0.05, 0.95, 181)
f1_scores = [
    f1_score(
        y_train,
        oof_probabilities >= threshold,
        zero_division=0,
    )
    for threshold in candidate_thresholds
]
best_threshold = candidate_thresholds[np.argmax(f1_scores)]

# Pipeline fitting learns preprocessing only from the training data.
model.fit(X_train, y_train)

test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = test_probabilities >= best_threshold

print(f"Selected threshold: {best_threshold:.2f}")
print(f"Accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, test_probabilities):.3f}")
print(f"Precision: {precision_score(y_test, test_predictions):.3f}")
print(f"Recall: {recall_score(y_test, test_predictions):.3f}")
print(f"F1: {f1_score(y_test, test_predictions):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("First five malignant probabilities:", test_probabilities[:5])

Common Pitfalls

  • Data leakage: Keep feature expansion, scaling, and classification inside the Pipeline; do not select a threshold on the test set.

  • Perfect separation: This can produce very large coefficients and unstable probabilities, so retain regularization and validate on unseen data.

  • Class imbalance: Consider class_weight="balanced", threshold tuning, and PR-AUC rather than accuracy alone.

  • Overfitting: High polynomial degrees greatly expand the feature space; cross-validate the degree and regularization strength.

  • Calibration: Logistic probabilities are not guaranteed to be calibrated; use held-out calibration when reliable probabilities are essential.

  • Typical follow-up questions: Why scale after polynomial expansion? How does standardization affect coefficient interpretation? When is a 0.5 threshold inappropriate? How should polynomial degree be selected without test-set leakage?

46

How can logistic regression predict marketing campaign conversions?

Logistic regression predicts the probability that a customer converts by combining customer and campaign features into a linear score. It is fast, interpretable, and supports threshold-based decisions, but its linear decision boundary can underperform when important relationships are nonlinear.

Read full answer

Applies toscikit-learn 1.5–1.7 (Python 3.12)

How Logistic Regression Works

For binary conversion outcomes, the model computes:

z = w · x + b

Here, w contains feature weights, x contains feature values, and b is the intercept. The sigmoid converts this score into the probability of conversion:

p = P(y = 1 | x) = 1 / (1 + e^(-z))

Training maximizes the observed likelihood by minimizing binary cross-entropy:

log_loss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the number of customers, y is the observed conversion indicator, and p is the predicted probability. This objective is convex, so optimization is generally reliable.

Each coefficient changes the log-odds:

log(p / (1 - p)) = w · x + b

Thus, exp(coefficient) is the estimated odds ratio for a one-unit increase in an original feature, holding other features constant. With standardized features, it represents the effect of a one-standard-deviation increase.

Campaign Workflow

  1. Define conversion: Use a specific outcome, such as a purchase within 30 days of campaign exposure.

  2. Build pre-campaign features: Include historical spending, purchase frequency, recency, tenure, channel, offer type, and relevant demographics. Exclude eventual purchase value and other post-exposure variables.

  3. Handle missing data: Impute values inside the pipeline so the imputation is learned only from training data.

  4. Split appropriately: Use a chronological split for future campaigns or a customer-group split when customers appear repeatedly.

  5. Fit a pipeline: Combine StandardScaler, optional imputation, and LogisticRegression so preprocessing is not fitted on validation or test data.

  6. Interpret and validate: Review odds ratios and domain plausibility, and compare performance with suitable nonlinear alternatives.

Evaluation and Thresholds

predict_proba returns the estimated probability of conversion. Select a threshold from validation or out-of-fold predictions rather than the test set:

  • Precision: How many predicted conversions were correct.

  • Recall: How many actual conversions were found.

  • F1: Harmonic balance of precision and recall.

  • ROC-AUC: Ranking performance across thresholds.

  • PR-AUC: Often more informative for rare conversions.

  • Accuracy: Can be misleading when conversions are uncommon.

An F1-maximizing threshold is useful when precision and recall are similarly important; an expected-profit threshold is better when false positives and false negatives have different costs.

Regularization and Multiclass Variants

LogisticRegression uses L2 regularization by default. Its C parameter is the inverse of regularization strength: larger C means weaker regularization, while smaller C means stronger regularization.

  • L1: Available with liblinear or saga; can produce sparse coefficients.

  • Elastic net: Available with saga; combines L1 and L2 behavior.

  • Class weighting: class_weight="balanced" can help with imbalanced conversions.

Historically, multiclass classification used one-vs-rest, fitting one binary classifier per class. Current scikit-learn versions use multinomial logistic regression, based on a softmax formulation, by default with the lbfgs solver. The multi_class parameter is deprecated, so it should not be set manually.

Code Example

This reproducible example uses bundled breast-cancer data. For a campaign, replace the data with pre-campaign features and conversion labels, and use a chronological or customer-group split where appropriate.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    precision_recall_curve,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load the bundled binary-classification dataset.
data = load_breast_cancer()
X, y = data.data, data.target

# Keep the test set separate from model and threshold selection.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

# Each fold fits the scaler and classifier on training data only.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probability = cross_val_predict(
    model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]

# Select an F1-maximizing threshold from out-of-fold predictions.
precision, recall, thresholds = precision_recall_curve(y_train, oof_probability)
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(thresholds),
    where=(precision[:-1] + recall[:-1]) != 0,
)
best_threshold = float(thresholds[np.argmax(f1)])

# Fit on all training data and evaluate once on the test set.
model.fit(X_train, y_train)
test_probability = model.predict_proba(X_test)[:, 1]
test_prediction = (test_probability >= best_threshold).astype(int)

print(f"Tuned threshold: {best_threshold:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_prediction):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probability):.3f}")
print(f"First test probability: {test_probability[0]:.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_prediction, labels=[0, 1]))

Common Pitfalls

  • Leakage: Do not fit scaling, imputation, feature selection, or thresholds on test data.

  • Temporal or customer dependence: Random row splits can overstate campaign performance.

  • Class imbalance: Accuracy can hide poor conversion recall; consider balanced weighting and PR-AUC.

  • Perfect separation: It can produce extreme coefficients; use regularization and validation.

  • Nonlinear relationships: Add meaningful transformations or interactions and compare alternatives.

  • Poor calibration: If probabilities drive budget decisions, validate them with calibration curves.

  • Overfitting: Regularization and cross-validation help control excessive fitting.

47

How is logistic regression used in credit scoring?

Logistic regression is a linear model that estimates the probability of a binary event, such as loan default. In credit scoring, it supports approval rules, risk ranking, pricing, and limits; its main trade-off is interpretability and stability versus limited ability to model nonlinear relationships.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For each applicant, the model computes a linear score:

z = w · x + b

It then converts that score into a probability:

p = P(y = 1 | x) = 1 / (1 + e^(-z))

Here:

  • x contains applicant features such as income, debt-to-income ratio, and credit history.

  • w contains the fitted feature coefficients.

  • · denotes a dot product.

  • b is the intercept.

  • z is the linear score.

  • p is the probability of the positive class.

  • e is Euler’s number.

Without regularization, training maximizes likelihood, equivalently minimizing binary cross-entropy:

log loss = -(1/n) * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i))

In this formula, n is the number of observations, i indexes them, y_i is an observed label, p_i is its predicted probability, and log is the natural logarithm. Scikit-learn instead minimizes log loss plus a regularization penalty. These objectives are convex, so a converged solver finds a global optimum rather than a local optimum.

Credit-Scoring Workflow

  1. Define the target: For example, 1 means default within a specified period.

  2. Prepare the data: Handle missing values, remove invalid predictors, and engineer ratios such as debt-to-income ratio.

  3. Prevent leakage: Fit preprocessing only on training data and exclude information observed after the prediction date.

  4. Validate chronologically: Prefer out-of-time validation because economic conditions and applicant behavior change.

  5. Choose actions: Use thresholds for approval, review, or rejection, and use probabilities for ranking or pricing.

  6. Monitor production behavior: Track calibration, ranking quality, default rates, approval rates, drift, and fairness.

Interpretation and Regularization

  • Coefficients: Each coefficient changes the log-odds for a one-unit increase in its model input while other inputs remain fixed.

  • Odds ratios: exp(w_j), meaning e raised to w_j, is the estimated odds ratio. For example, a value of 2 means the odds double.

  • Scaling: Standardize numerical features before fitting so coefficients, optimization, and regularization behave sensibly.

  • C: This is the inverse of regularization strength; larger C means weaker regularization.

  • Penalties: l2 is the default; l1 is supported by the liblinear and saga solvers, while elasticnet requires saga and an appropriate l1_ratio.

With standardized inputs, a coefficient represents a change per standard deviation rather than per original unit. Credit scorecards often therefore use transformed or binned features.

Evaluation and Thresholding

  • ROC-AUC measures ranking quality across thresholds.

  • PR-AUC is often more informative when defaults are rare.

  • Precision is the proportion of predicted defaults that actually default.

  • Recall is the proportion of actual defaults that the model identifies.

  • Calibration checks whether predicted probabilities match observed frequencies.

  • Accuracy can mislead when defaults are a small fraction of applicants.

Select thresholds using validation data and business costs rather than accuracy alone. A lower threshold flags more applicants as likely defaulters, raising recall but rejecting more healthy applicants; a higher threshold does the reverse.

Code Example

This complete example treats malignant breast-cancer cases as the positive risk event, analogous to default in credit scoring.

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# Original labels: 0 = malignant, 1 = benign.
# Remap so that class 1 is the positive risk event.
y = (y == 0).astype(int)

X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.40, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val,
    y_train_val,
    test_size=0.50,
    random_state=42,
    stratify=y_train_val,
)

# The pipeline prevents scaling leakage during cross-validation.
model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(
        C=1.0,
        penalty="l2",
        solver="lbfgs",
        max_iter=1000,
    )),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(
    model, X_train, y_train, cv=cv, scoring="roc_auc"
)
print(f"Cross-validated ROC-AUC: {cv_scores.mean():.3f}")

model.fit(X_train, y_train)

# Tune the decision threshold using validation probabilities only.
val_probabilities = model.predict_proba(X_val)[:, 1]
thresholds = np.linspace(0.05, 0.95, 181)
f1_scores = [
    f1_score(y_val, (val_probabilities >= threshold).astype(int))
    for threshold in thresholds
]
best_threshold = thresholds[int(np.argmax(f1_scores))]

# Evaluate the selected threshold once on the test set.
test_probabilities = model.predict_proba(X_test)[:, 1]
test_predictions = (test_probabilities >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.2f}")
print(f"Test accuracy: {accuracy_score(y_test, test_predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_probabilities):.3f}")
print(f"Test PR-AUC: {average_precision_score(y_test, test_probabilities):.3f}")
print(f"Test precision: {precision_score(y_test, test_predictions):.3f}")
print(f"Test recall: {recall_score(y_test, test_predictions):.3f}")
print(f"Test F1: {f1_score(y_test, test_predictions):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions, labels=[0, 1]))
print("Example positive-class probabilities:")
print(model.predict_proba(X_test[:3])[:, 1])

Common Pitfalls

  • Data leakage: Never use repayment behavior recorded after the prediction date.

  • Class imbalance: Consider class_weight="balanced" and emphasize PR-AUC, recall, and economic costs over accuracy.

  • Nonlinear relationships: Engineer interactions or nonlinear features when necessary.

  • Perfect separation: Coefficients can become extremely large and probabilities unstable; stronger regularization or data review may help.

  • Overfitting: Keep regularization enabled, validate out of time, and avoid repeatedly tuning against the test set.

  • Poor calibration: Good ROC-AUC does not guarantee reliable probabilities; validate calibration on out-of-time data.

  • Unfair proxies: Omitting protected attributes does not eliminate discrimination through correlated features.

  • Multiclass choice: With lbfgs, scikit-learn uses multinomial softmax by default for multiclass. The multi_class parameter is deprecated, so use OneVsRestClassifier(LogisticRegression(...)) for explicit one-vs-rest modeling.

48

How can logistic regression predict a patient’s disease probability?

Logistic regression estimates the probability that a patient belongs to the positive class, such as having a disease, from clinical features. It converts a linear score into a probability between 0 and 1 and is trained by minimizing binary cross-entropy. Its main trade-off is an interpretable model and linear decision boundary versus limited ability to represent nonlinear relationships without feature engineering.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); the `multi_class` deprecation note applies to 1.5–1.6

How It Works

For binary outcomes, the model calculates:

z = w . x + b

where w contains the feature weights, x is the patient’s feature vector, and b is the intercept. The predicted probability is:

P(y = 1 | x) = 1 / (1 + e^(-z))

A prediction can be produced by thresholding this probability, commonly at 0.5. Because z is linear, the decision boundary between classes is also linear in the original feature space.

Training maximizes the observed-data likelihood, equivalently minimizing log loss:

LogLoss = -(1/n) * sum(y * log(p) + (1 - y) * log(1 - p))

Here, n is the number of patients, y is the true binary label, and p is the predicted probability of the positive class. The regularized binary log-loss objective is convex, so optimization does not require the same non-convex search used by many neural networks.

Data Preparation and Training

  • Split the data into development and test sets before fitting any transformation.

  • Use StandardScaler inside a Pipeline so it learns only from each training fold and avoids data leakage.

  • The default l2 penalty controls overfitting. In scikit-learn, C is the inverse of regularization strength: larger C means weaker regularization.

  • l1 requires liblinear or saga; elasticnet requires saga.

  • For severely imbalanced outcomes, consider class_weight="balanced" and evaluate recall, precision, F1, ROC-AUC, and PR-AUC rather than accuracy alone.

Interpreting Coefficients

A coefficient β represents the change in the log-odds for a one-unit increase in that feature, holding other features constant. Its odds ratio is exp(β): for example, exp(β) = 2 means the odds double.

When features are standardized, a coefficient describes the effect of a one-standard-deviation increase. Before standardization, it describes the effect in the feature’s original units.

Multiclass Extension

For more than two mutually exclusive diagnoses, multinomial logistic regression uses the softmax function. One-vs-rest can also train a separate binary model per class. From scikit-learn 1.5 onward, solvers that support multinomial optimization, including lbfgs, use multinomial regression by default. The multi_class parameter was deprecated in scikit-learn 1.5–1.6 and removed in 1.7+, so current code should not set it.

Code Example

This example uses load_breast_cancer, remaps malignant cases to 1, and treats benign cases as 0. It selects an F1-based threshold on a validation set and evaluates the final model once on untouched test data.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    f1_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Load bundled medical data
data = load_breast_cancer()
X = data.data

# load_breast_cancer uses 0 for malignant and 1 for benign.
# Remap so that 1 means the positive disease class.
malignant_label = list(data.target_names).index("malignant")
y = (data.target == malignant_label).astype(int)

# Keep the test set untouched during model development
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

# Reserve validation data for threshold selection
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.20, stratify=y_dev, random_state=42
)

def build_model():
    return Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", LogisticRegression(max_iter=1000)),
    ])

# Estimate generalization performance using cross-validation
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_auc = cross_val_score(
    build_model(), X_dev, y_dev, cv=cv, scoring="roc_auc"
)
print(f"Mean cross-validated ROC-AUC: {cv_auc.mean():.3f}")

# Select a threshold using validation predictions only
validation_model = build_model()
validation_model.fit(X_train, y_train)
validation_prob = validation_model.predict_proba(y_val)[:, 1] if False else (
    validation_model.predict_proba(X_val)[:, 1]
)

thresholds = np.linspace(0.10, 0.90, 161)
f1_scores = [
    f1_score(y_val, validation_prob >= threshold)
    for threshold in thresholds
]
best_threshold = thresholds[int(np.argmax(f1_scores))]

# Refit on all development data, then evaluate once on the test set
final_model = build_model()
final_model.fit(X_dev, y_dev)
test_prob = final_model.predict_proba(X_test)[:, 1]
test_prediction = (test_prob >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.3f}")
print(f"Test accuracy: {accuracy_score(y_test, test_prediction):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, test_prob):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, test_prediction))
print("First five predicted disease probabilities:")
print(test_prob[:5])

Common Pitfalls

  • Leakage: Fitting a scaler or imputer on the full dataset exposes test information. Use Pipeline and fold-local transformations.

  • Wrong metrics: Accuracy can be misleading with imbalanced classes. In screening, false negatives may matter more, so emphasize recall and PR-AUC.

  • Threshold leakage: Do not tune and evaluate the threshold on the same test set.

  • Perfect separation: Completely separable data can produce very large coefficients; regularization and suitable features help.

  • Nonlinearity: Logistic regression cannot learn curved boundaries directly. Add meaningful nonlinear features or use a more flexible model.

  • Calibration: Ranking quality measured by ROC-AUC does not guarantee accurate probabilities. Validate calibration on held-out data and consider CalibratedClassifierCV when precise probabilities are important.

49

How would logistic regression power an e-commerce recommender system?

Logistic regression can act as a fast, interpretable purchase-propensity model that ranks e-commerce candidates by predicted probability. It models a linear relationship between engineered user, item, and context features and the purchase log-odds, so complex behavior must be added through feature engineering. Its main trade-off is transparent, cheap scoring versus the flexibility of nonlinear recommenders.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

How It Works

For each user-item pair, logistic regression computes z = w · x + b and p(y=1|x) = 1 / (1 + e^(-z)). Here, x contains user, item, and contextual features, w contains learned weights, b is the intercept, and p is the estimated purchase probability.

Training maximizes the likelihood of observed binary labels, equivalently minimizing binary cross-entropy: Log Loss = -(1/n) * sum(i=1..n) [y_i*log(p_i) + (1-y_i)*log(1-p_i)]. Here, n is the number of examples, y_i is an observed binary label, p_i is its predicted probability, and i indexes the examples. The regularized objective is convex.

Each coefficient changes the predicted log-odds; exp(coefficient) is the odds ratio for a one-unit increase. For a standardized numerical feature, one unit normally means one standard deviation.

Recommender Design

  1. Generate candidates: Exclude unavailable, unwanted, and already-purchased items; begin with popularity, category, or collaborative-filtering candidates.

  2. Build training rows: Represent each candidate with user features, item features, context, and meaningful interactions such as category preference or discount × price.

  3. Fit one model: Train a single classifier across all labeled user-item rows. Use chronological splits in production so future interactions cannot leak into training.

  4. Score and rank: Apply predict_proba(X)[:, 1], sort by probability, and then apply constraints or reranking for diversity, novelty, and business value.

Evaluate ranking with Precision@K, Recall@K, MAP, and NDCG@K. For binary purchase prediction, emphasize precision, recall, F1, ROC-AUC, and PR-AUC or average precision; accuracy can hide poor performance on rare purchases. Validate important changes with an online A/B test.

Training Considerations

  • Preprocessing: Impute missing values and standardize numerical features inside a pipeline so these steps are fitted only on training data. Scaling makes regularization penalties comparable and improves optimization.

  • Regularization: C is the inverse of regularization strength. The default penalty is l2; l1 requires liblinear or saga, and elasticnet requires saga.

  • Imbalance: For rare purchases, test class_weight="balanced" and evaluate PR-AUC and recall. Balanced weighting can alter probability scale, so recalibrate when probabilities matter.

  • Complexity: Each solver iteration is approximately O(n*d) for binary logistic regression; prediction is approximately O(d) per user-item row.

For a mutually exclusive multiclass target:

Variant How it works Strength Weakness Best fit and scale
One-vs-rest One binary classifier per class Simple and parallelizable Class probabilities may be incoherent Independent binary tasks; roughly k model fits
Multinomial One joint softmax model Coherent class probabilities Classes are coupled Mutually exclusive classes; approximately O(n*d*k) per iteration

With the lbfgs solver, scikit-learn uses multinomial logistic regression for multiclass classification by default. In scikit-learn 1.5-1.6, the multi_class parameter was deprecated; it was removed in 1.7.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, average_precision_score, confusion_matrix,
    f1_score, precision_recall_curve, precision_score,
    recall_score, roc_auc_score
)
from sklearn.model_selection import (
    StratifiedKFold, cross_validate, train_test_split
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train_val, X_test, y_train_val, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train_val, y_train_val, test_size=0.25,
    stratify=y_train_val, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42))
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_validate(
    model, X_train, y_train, cv=cv,
    scoring=("accuracy", "roc_auc", "average_precision")
)
for name, values in cv_scores.items():
    print(f"CV {name}: {values.mean():.3f}")

model.fit(X_train, y_train)
val_probability = model.predict_proba(X_val)[:, 1]
precision_curve, recall_curve, thresholds = precision_recall_curve(
    y_val, val_probability
)
f1 = 2 * precision_curve[:-1] * recall_curve[:-1] / np.maximum(
    precision_curve[:-1] + recall_curve[:-1], 1e-12
)
threshold = float(thresholds[np.argmax(f1)])

test_probability = model.predict_proba(X_test)[:, 1]
test_prediction = (test_probability >= threshold).astype(int)

print("Threshold:", threshold)
print("Accuracy:", accuracy_score(y_test, test_prediction))
print("ROC-AUC:", roc_auc_score(y_test, test_probability))
print("Average precision:", average_precision_score(y_test, test_probability))
print("Precision:", precision_score(y_test, test_prediction))
print("Recall:", recall_score(y_test, test_prediction))
print("F1:", f1_score(y_test, test_prediction))
print("Confusion matrix:\n", confusion_matrix(y_test, test_prediction))
print("Example probabilities:", test_probability[:5])

Common Pitfalls

  • Data leakage: Fit preprocessing only on training folds and derive features only from information available at prediction time.

  • False negatives: An unclicked item is not necessarily a true negative; use impressions or propensity weighting when possible.

  • Perfect separation: Coefficients may become extremely large and probabilities unstable; inspect convergence and use regularization.

  • Nonlinearity and calibration: Add meaningful crosses or splines when relationships are nonlinear, and validate probability calibration separately from ranking quality.

  • Wrong evaluation: Emphasize PR-AUC and ranking metrics for rare purchases, and do not repeatedly tune thresholds against the test set.

50

How can logistic regression analyze employee attrition factors?

Logistic regression estimates an employee’s probability of leaving from factors such as tenure, compensation, workload, and manager support, producing interpretable coefficients and odds ratios. Its main trade-off is a linear decision boundary that requires feature engineering for nonlinear effects, and its results identify associations rather than proving causation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For employee i, logistic regression calculates a linear score and converts it to a probability:

z_i = b + sum(w_j * x_ij)

p_i = 1 / (1 + e^(-z_i))

Here, i identifies an employee, x_ij is factor j, w_j is its coefficient, b is the intercept, z_i is the linear score, e is Euler’s number, and p_i is the predicted attrition probability.

Without regularization, maximum-likelihood training minimizes binary cross-entropy:

L = -1/n * sum(y_i * log(p_i) + (1 - y_i) * log(1 - p_i))

Here, n is the number of employees, y_i is the observed binary attrition outcome, p_i is the predicted probability, and log is the natural logarithm. Scikit-learn’s default penalty="l2" instead minimizes this loss plus an L2 penalty, equivalent to maximizing a penalized likelihood. The regularized objective is convex, so any converged optimum is global; without regularization, perfect separation can prevent a finite optimum from existing.

Interpreting Factors

The model can also be expressed in log-odds:

log(p_i / (1 - p_i)) = b + sum(w_j * x_ij)

  • Holding other predictors fixed, a one-unit increase changes log-odds by the coefficient.

  • exp(w_j) is the corresponding odds ratio; 2 means the odds double, while 0.5 means they are halved.

  • For a binary predictor, the coefficient describes the change from 0 to 1.

  • With StandardScaler, coefficients represent one-standard-deviation increases. Divide each coefficient by scaler.scale_ before exponentiating to obtain the odds ratio per original unit.

A sound workflow is:

  1. Define observed attrition and select plausible predictors.

  2. Split the data before fitting scalers, imputers, encoders, or feature selectors.

  3. Use a Pipeline so each cross-validation fold learns preprocessing only from its training portion.

  4. Tune the classification threshold using out-of-fold predictions, then evaluate once on untouched test data.

Code Example

import numpy as np
import pandas as pd

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 1000

employees = pd.DataFrame({
    "tenure_years": rng.uniform(0.1, 10, n),
    "overtime": rng.binomial(1, 0.30, n),
    "manager_support": rng.normal(0, 1, n),
    "commute_minutes": rng.normal(30, 12, n),
})

score = (
    -1.2
    + 0.90 * employees["overtime"]
    - 0.75 * employees["manager_support"]
    - 0.025 * employees["commute_minutes"]
    - 0.12 * employees["tenure_years"]
)
probability = 1 / (1 + np.exp(-score.to_numpy()))
employees["attrition"] = rng.binomial(1, probability)

X = employees.drop(columns="attrition")
y = employees["attrition"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

model = Pipeline([
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(
        penalty="l2",
        C=1.0,
        solver="lbfgs",
        max_iter=1000,
    )),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

oof_probability = cross_val_predict(
    model,
    X_train,
    y_train,
    cv=cv,
    method="predict_proba",
)[:, 1]

thresholds = np.linspace(0.10, 0.90, 161)
f1_values = [
    f1_score(y_train, oof_probability >= threshold, zero_division=0)
    for threshold in thresholds
]
best_threshold = float(thresholds[np.argmax(f1_values)])

model.fit(X_train, y_train)
test_probability = model.predict_proba(X_test)[:, 1]
prediction = (test_probability >= best_threshold).astype(int)

print(f"Selected threshold: {best_threshold:.3f}")
print(f"Accuracy: {accuracy_score(y_test, prediction):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, test_probability):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, test_probability):.3f}")
print(f"Precision: {precision_score(y_test, prediction):.3f}")
print(f"Recall: {recall_score(y_test, prediction):.3f}")
print(f"F1: {f1_score(y_test, prediction):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, prediction))

scaler = model.named_steps["scaler"]
classifier = model.named_steps["classifier"]
coefficients = classifier.coef_[0]

coefficient_table = pd.DataFrame({
    "feature": X_train.columns,
    "coefficient_per_sd": coefficients,
    "odds_ratio_per_sd": np.exp(coefficients),
    "odds_ratio_per_raw_unit": np.exp(coefficients / scaler.scale_),
})
print("\nCoefficient interpretation:")
print(
    coefficient_table.sort_values(
        "odds_ratio_per_raw_unit",
        ascending=False,
    )
)

examples = X_test.head(5).copy()
examples["attrition_probability"] = test_probability[:5]
print("\nExample probabilities:")
print(examples)

Practical Considerations

  • Scaling and regularization: Scale predictors for regularized logistic regression. C is the inverse of regularization strength, so smaller C means stronger regularization. L2 is the default; L1 requires liblinear or saga, and elastic net requires saga.

  • Class imbalance: Accuracy can be misleading when most employees stay. Inspect precision, recall, F1, ROC-AUC, and PR-AUC. class_weight="balanced" emphasizes the minority class but estimates a reweighted objective.

  • Missing and categorical data: Place SimpleImputer and encoders inside a Pipeline or ColumnTransformer, ensuring they are fitted only on training folds.

  • Nonlinearity and overfitting: Add justified transformations, splines, or interactions. Keep the test set untouched and tune preprocessing, regularization, and thresholds within training data.

  • Perfect separation: A predictor that perfectly determines attrition can produce extreme unregularized coefficients. Investigate data quality and validate the model.

  • Calibration: Good ranking metrics do not guarantee accurate probabilities. Inspect a calibration curve when probability estimates will drive HR decisions.

  • Multiclass: Multinomial logistic regression uses softmax, while one-vs-rest fits one binary model per class. Scikit-learn’s lbfgs solver uses multinomial by default for multiclass; multi_class was deprecated in 1.5 and should not be set in new code.

Common Pitfalls

  • Fitting preprocessing before splitting or cross-validation causes leakage.

  • Selecting a threshold on the test set invalidates the reported performance.

  • Treating odds ratios as causal effects ignores confounding and selection bias.

  • Correlated predictors can make individual coefficients unstable.

  • Reporting only accuracy can conceal poor minority-class recall.

Common follow-up questions include how missing values and categorical factors are handled, how calibration is tested, how perfect separation is diagnosed, and how predictive importance is distinguished from causal impact.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Logistic Regression answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects