Skip to content
Machine Learning & Data Science

Scikit-Learn interview questions and answers

50 questions, 50 with code. About 2 h 39 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

2

What design principles underlie scikit-learn's API?

Scikit-learn’s API follows a few deliberate design principles, described in the project’s 2013 API design paper: consistency (every object shares a small set of methods: fit, predict, transform, score), inspection (hyperparameters are public attributes set in __init__, learned parameters end in _), limited object hierarchy (data is plain NumPy arrays, SciPy sparse matrices or DataFrames, not custom classes; hyperparameters are strings and numbers), composition (complex models are built from simple blocks with Pipeline, ColumnTransformer and meta-estimators), and sensible defaults (models work reasonably without tuning). Estimators are cloneable via get_params/set_params, which is what makes grid search and cross-validation work generically.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Principles

Principle In practice
Consistency Same fit/predict/transform interface everywhere
Inspection model.get_params(), fitted attributes like coef_, feature_importances_
Non-proliferation of classes Inputs and outputs are arrays or DataFrames
Composition Pipeline, ColumnTransformer, FeatureUnion, GridSearchCV
Sensible defaults RandomForestClassifier() works out of the box
Separation of hyperparameters and learned state __init__ only stores arguments; learning happens in fit
Duck typing Any object with the right methods works in pipelines

Code Example: Cloning, Parameters and Duck Typing

import numpy as np
from sklearn.base import BaseEstimator, ClassifierMixin, clone
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.utils.multiclass import unique_labels

model = LogisticRegression(C=0.5, max_iter=1000)
print("hyperparameters:", {k: v for k, v in model.get_params().items() if k in ("C", "max_iter")})

copy = clone(model)                      # same hyperparameters, no learned state
copy.set_params(C=2.0)
print("clone C:", copy.C, "| original C:", model.C)


class MajorityClassifier(ClassifierMixin, BaseEstimator):
    """Minimal custom estimator following the API conventions."""

    def __init__(self, strategy="most_frequent"):
        self.strategy = strategy            # store hyperparameters only

    def fit(self, X, y):
        self.classes_ = unique_labels(y)    # learned attributes end with _
        values, counts = np.unique(y, return_counts=True)
        self.majority_ = values[np.argmax(counts)]
        return self

    def predict(self, X):
        return np.full(len(X), self.majority_)


X, y = load_iris(return_X_y=True)
print("custom estimator works with cross_val_score:", cross_val_score(MajorityClassifier(), X, y, cv=5).mean().round(3))
print("real model:", cross_val_score(model, X, y, cv=5).mean().round(3))

The custom class works with cross_val_score without any registration, because it follows the conventions: hyperparameters in __init__, learning in fit, a predict method and the ClassifierMixin default score.

Interview Tips

  • Mention cloning: model selection tools rely on get_params to create fresh copies per fold.

  • Relate the principles to benefits: easy experimentation, leak-free pipelines and reusable tooling.

3

How do you handle missing values with scikit-learn?

Scikit-learn handles missing values with imputers, indicators and estimators that accept NaN natively. SimpleImputer fills each column with the mean, median, most frequent value or a constant learned from the training data; KNNImputer uses the average of the nearest neighbours; IterativeImputer (experimental) predicts each feature from the others in rounds. MissingIndicator or add_indicator=True adds binary flags so the model can learn from missingness itself. HistGradientBoostingClassifier/Regressor, and decision trees and random forests in recent versions, can train directly on data with NaN. Wrap imputation in a Pipeline so it is fitted only on training folds, and choose the strategy per column type with ColumnTransformer.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Options

Approach Tool Notes
Simple statistics SimpleImputer(strategy="median") Fast, robust default for numbers
Categorical SimpleImputer(strategy="most_frequent") or constant “missing” as its own category
Neighbour-based KNNImputer Uses similar rows; slower
Model-based IterativeImputer Needs enable_iterative_imputer import
Flags add_indicator=True, MissingIndicator Keep missingness information
Native NaN support HistGradientBoosting* No imputation required

Code Example: Comparing Strategies

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.experimental import enable_iterative_imputer  # noqa: F401  (enables IterativeImputer)
from sklearn.impute import IterativeImputer, KNNImputer, SimpleImputer
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
rng = np.random.default_rng(42)
X_missing = X.copy()
X_missing[rng.random(X.shape) < 0.2] = np.nan            # remove 20% of values at random

pipelines = {
    "mean": make_pipeline(SimpleImputer(strategy="mean"), StandardScaler(), Ridge()),
    "median + indicator": make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler(), Ridge()),
    "KNN": make_pipeline(StandardScaler(), KNNImputer(n_neighbors=5), Ridge()),
    "iterative": make_pipeline(IterativeImputer(random_state=42, max_iter=10), StandardScaler(), Ridge()),
    "HistGB native NaN": HistGradientBoostingRegressor(random_state=42),
}
print("complete data Ridge:", cross_val_score(make_pipeline(StandardScaler(), Ridge()), X, y, cv=5, scoring="r2").mean().round(3))
for name, model in pipelines.items():
    print(f"{name:>20}: CV R² {cross_val_score(model, X_missing, y, cv=5, scoring='r2').mean():.3f}")

The # noqa comment marks an import that is needed only for its side effect of enabling IterativeImputer. The “complete data” score shows how much information was lost by removing values.

Common Pitfalls

  • df.fillna(df.mean()) before splitting, which leaks test statistics.

  • Treating values missing for a reason (e.g. no loan → no loan amount) as random gaps.

  • Forgetting that most estimators raise errors on NaN unless you impute.

4

What roles do transformers and estimators play in scikit-learn?

In scikit-learn, an estimator is any object that learns from data with fit, storing what it learns in attributes ending with an underscore. Transformers are estimators that also implement transform (and fit_transform), producing a modified version of the data: scalers, imputers, encoders, PCA, text vectorizers, feature selectors. Predictors are estimators that implement predict (classifiers, regressors, some clusterers) and usually score. A Pipeline chains transformers and ends with any estimator, and meta-estimators such as GridSearchCV, Pipeline, ColumnTransformer and VotingClassifier wrap other estimators while exposing the same interface. (These “transformers” are unrelated to the Transformer neural network architecture.)

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Roles

Role Methods Examples
Estimator (all) fit, get_params, set_params Every scikit-learn model or preprocessor
Transformer transform, fit_transform, get_feature_names_out StandardScaler, OneHotEncoder, PCA
Predictor predict, score (+ predict_proba) LogisticRegression, RandomForestRegressor
Clusterer fit_predict (some predict) KMeans, DBSCAN
Meta-estimator Wraps estimators Pipeline, GridSearchCV, StackingClassifier

Code Example

from sklearn.base import is_classifier, is_regressor
from sklearn.datasets import load_diabetes
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)

scaler = StandardScaler().fit(X)                       # transformer: learns mean_ and scale_
pca = PCA(n_components=3).fit(scaler.transform(X))     # transformer: learns components_
reg = LinearRegression().fit(pca.transform(scaler.transform(X)), y)   # predictor: learns coef_

for obj in (scaler, pca, reg, LogisticRegression()):
    print(f"{type(obj).__name__:>18}: transform={hasattr(obj, 'transform')}, predict={hasattr(obj, 'predict')}, "
          f"classifier={is_classifier(obj)}, regressor={is_regressor(obj)}")

pipe = make_pipeline(StandardScaler(), PCA(), LinearRegression())        # meta-estimator
search = GridSearchCV(pipe, {"pca__n_components": [2, 5, 8]}, cv=5).fit(X, y)
print("pipeline is itself an estimator; best:", search.best_params_)
print("parameters exposed:", [p for p in pipe.get_params() if p.startswith("pca__")][:4])

Because every object shares fit and get_params, meta-estimators can treat a whole pipeline exactly like a single model.

Common Pitfalls

  • Calling fit_transform on test data instead of transform.

  • Assuming every estimator has predict_proba (e.g. SVC needs probability=True, LinearSVC has none).

  • Confusing scikit-learn transformers with neural Transformer models.

5

What is the typical workflow for building a model with scikit-learn?

The typical scikit-learn workflow has a clear order: (1) define the problem and the metric that reflects the business goal; (2) load and explore the data; (3) split off a test set (stratified, grouped or time-based as needed); (4) build a preprocessing ColumnTransformer and wrap it with a model in a Pipeline; (5) establish a Dummy baseline; (6) compare a few model families with cross-validation; (7) tune the best candidates with GridSearchCV/RandomizedSearchCV; (8) evaluate the final pipeline once on the test set and analyse errors; (9) save it with joblib and deploy; (10) monitor and retrain. The consistent estimator API (fit, predict, transform, score) makes each step interchangeable.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Workflow

Stage scikit-learn
Problem and metric scoring="roc_auc", "neg_mean_absolute_error"
Split train_test_split(stratify=y)
Preprocess ColumnTransformer, imputers, encoders, scalers
Baseline DummyClassifier, DummyRegressor
Compare cross_validate
Tune GridSearchCV, RandomizedSearchCV
Evaluate classification_report, ConfusionMatrixDisplay, permutation_importance
Persist joblib.dump
Monitor Metrics on new labelled data, drift checks

Code Example

from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.datasets import fetch_openml
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import RandomizedSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

titanic = fetch_openml("titanic", version=1, as_frame=True)          # downloads the dataset once
X = titanic.data[["pclass", "sex", "age", "sibsp", "parch", "fare", "embarked"]]
y = titanic.target.astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
     make_column_selector(dtype_include="number")),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore")),
     make_column_selector(dtype_include="category")),
])
models = {
    "baseline": DummyClassifier(strategy="most_frequent"),
    "logistic": make_pipeline(preprocess, LogisticRegression(max_iter=1000)),
    "boosting": make_pipeline(preprocess, HistGradientBoostingClassifier(random_state=42)),
}
for name, model in models.items():
    print(f"{name:>9}: CV ROC AUC {cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc').mean():.3f}")

search = RandomizedSearchCV(models["boosting"],
                            {"histgradientboostingclassifier__learning_rate": [0.03, 0.1, 0.3],
                             "histgradientboostingclassifier__max_leaf_nodes": [8, 16, 31]},
                            n_iter=6, cv=5, scoring="roc_auc", random_state=42).fit(X_train, y_train)
best = search.best_estimator_
print(classification_report(y_test, best.predict(X_test), digits=3))
imp = permutation_importance(best, X_test, y_test, scoring="roc_auc", n_repeats=5, random_state=42)
print(dict(zip(X.columns, imp.importances_mean.round(3))))

fetch_openml downloads data on first use; with no network, swap in a bundled dataset such as load_breast_cancer.

Common Pitfalls

  • Skipping the baseline.

  • Tuning before fixing data issues and leakage.

  • Touching the test set during model selection.

6

How can you scale features with scikit-learn?

Feature scaling in scikit-learn uses transformers from sklearn.preprocessing that learn statistics on training data with fit and apply them with transform. The common choices are StandardScaler (zero mean, unit variance), MinMaxScaler (a fixed range, default [0, 1]), RobustScaler (median and IQR, resistant to outliers) and MaxAbsScaler (for sparse data). Scaling matters for models that use distances or gradients, such as k-NN, SVMs, k-means, PCA, regularized linear models and neural networks, but not for tree-based models. Apply scalers to numeric columns only with ColumnTransformer, inside a Pipeline, so scaling is fitted on training folds only.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

When Scaling Matters

Model Needs scaling? Why
k-NN, k-means, SVM (RBF) Yes Distances dominated by large-scale features
Logistic/linear with regularization Yes Penalty treats all coefficients equally
PCA Yes Variance depends on units
Neural networks (MLP) Yes Faster, stable optimization
Trees, random forests, gradient boosting No Splits are invariant to monotone scaling

Code Example: Effect of Scaling on k-NN

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler

X, y = load_wine(return_X_y=True)
print("feature ranges (min, max) of first 3 features:", list(zip(X.min(0)[:3].round(1), X.max(0)[:3].round(1))))

candidates = {
    "k-NN, no scaling": KNeighborsClassifier(),
    "k-NN + StandardScaler": make_pipeline(StandardScaler(), KNeighborsClassifier()),
    "k-NN + MinMaxScaler": make_pipeline(MinMaxScaler(), KNeighborsClassifier()),
    "k-NN + RobustScaler": make_pipeline(RobustScaler(), KNeighborsClassifier()),
    "forest, no scaling": RandomForestClassifier(random_state=42),
    "forest + StandardScaler": make_pipeline(StandardScaler(), RandomForestClassifier(random_state=42)),
}
for name, model in candidates.items():
    print(f"{name:>24}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

The wine data’s “proline” feature is in the hundreds to thousands while others are below 10, so unscaled k-NN effectively uses only proline; scaling improves k-NN substantially, while the random forest’s score is the same with or without scaling.

Common Pitfalls

  • Fitting the scaler on all data, including the test set.

  • Scaling binary or one-hot columns unnecessarily (harmless for most models, but it complicates interpretation).

  • Forgetting to apply the same fitted scaler at prediction time (a pipeline solves this).

7

What is a pipeline in scikit-learn?

A scikit-learn Pipeline is a sequence of steps, all transformers except the last, which can be any estimator, packaged as a single estimator. Calling fit fits each transformer on the training data and passes its output to the next step, then fits the final model; predict, predict_proba and score push new data through the already-fitted transformers. Because the whole pipeline behaves like one model, cross-validation and grid search refit preprocessing inside each fold (preventing leakage), hyperparameters of any step can be tuned with step__parameter names, and one saved object contains everything needed for deployment. make_pipeline creates one with automatic step names.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Why Use Pipelines

Benefit Explanation
No leakage Preprocessing learns only from training folds
Less code One fit / predict call
Joint tuning GridSearchCV over preprocessing and model parameters
Reproducibility Same steps in training and production
Deployment Save one object with joblib
Inspection pipe.named_steps["model"], slicing pipe[:-1]

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# Leaky: feature selection on the full data before CV
X_selected = SelectKBest(f_classif, k=5).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(max_iter=5000), X_selected, y, cv=5).mean()

# Correct: selection inside the pipeline, refit in every fold
pipe = Pipeline([
    ("scale", StandardScaler()),
    ("select", SelectKBest(f_classif, k=5)),
    ("model", LogisticRegression(max_iter=5000)),
])
correct = cross_val_score(pipe, X, y, cv=5).mean()
print(f"selection outside CV {leaky:.4f} vs inside pipeline {correct:.4f}")

search = GridSearchCV(pipe, {"select__k": [5, 10, 20], "model__C": [0.1, 1, 10]}, cv=5)
search.fit(X, y)
print("best:", search.best_params_)

auto = make_pipeline(StandardScaler(), PCA(n_components=10), LogisticRegression(max_iter=5000))
print("step names:", list(auto.named_steps))
print("transformed shape:", auto[:-1].fit_transform(X).shape)

With real signal the leak may be small, but on noisy data with many features, selecting features before cross-validation can make a useless model look good.

Common Pitfalls

  • Fitting transformers outside the pipeline “just once”.

  • Misspelled parameter names in grids (use pipe.get_params().keys() to check).

  • Putting a non-transformer (a model) in the middle of the pipeline.

8

What are the main categories of algorithms in scikit-learn?

Scikit-learn organizes its algorithms into a few families. Supervised learning covers classification and regression: linear models (LinearRegression, LogisticRegression, Ridge, Lasso, SGDClassifier), support vector machines, nearest neighbours, decision trees, ensembles (random forests, gradient boosting, bagging, stacking), naive Bayes, Gaussian processes, discriminant analysis and simple neural networks (MLPClassifier). Unsupervised learning covers clustering (KMeans, DBSCAN, HDBSCAN, AgglomerativeClustering, GaussianMixture), dimensionality reduction (PCA, TruncatedSVD, NMF, TSNE) and outlier detection (IsolationForest, LocalOutlierFactor). Around them sit preprocessing, feature selection, model selection and metrics modules.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Families and Modules

Category Module Examples
Linear models sklearn.linear_model LinearRegression, LogisticRegression, Ridge, Lasso, SGD
SVM sklearn.svm SVC, SVR, LinearSVC
Neighbours sklearn.neighbors KNeighborsClassifier, NearestNeighbors
Trees sklearn.tree DecisionTreeClassifier/Regressor
Ensembles sklearn.ensemble RandomForest, HistGradientBoosting, Stacking, IsolationForest
Naive Bayes sklearn.naive_bayes GaussianNB, MultinomialNB
Neural networks sklearn.neural_network MLPClassifier, MLPRegressor
Clustering sklearn.cluster KMeans, DBSCAN, HDBSCAN
Mixture models sklearn.mixture GaussianMixture
Decomposition sklearn.decomposition, sklearn.manifold PCA, NMF, TSNE
Preprocessing / features sklearn.preprocessing, sklearn.feature_selection StandardScaler, OneHotEncoder, SelectKBest
Model selection / metrics sklearn.model_selection, sklearn.metrics GridSearchCV, roc_auc_score

Code Example: The Same API Across Families

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.ensemble import IsolationForest, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import adjusted_rand_score
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_iris(return_X_y=True)

for name, model in [("linear", LogisticRegression(max_iter=1000)), ("SVM", SVC()),
                    ("k-NN", KNeighborsClassifier()), ("forest", RandomForestClassifier(random_state=42))]:
    pipe = make_pipeline(StandardScaler(), model)
    print(f"supervised {name:>6}: CV accuracy {cross_val_score(pipe, X, y, cv=5).mean():.3f}")

labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(StandardScaler().fit_transform(X))
print("clustering ARI vs species:", round(adjusted_rand_score(y, labels), 3))
print("PCA 2D shape:", PCA(n_components=2).fit_transform(X).shape)
print("outliers flagged:", int((IsolationForest(random_state=42).fit_predict(X) == -1).sum()))

Every model is used the same way (fit, then predict, transform or fit_predict), which is why swapping algorithms takes one line.

Interview Tips

  • Mention that scikit-learn focuses on classical ML for tabular data; deep learning belongs to PyTorch or TensorFlow.

  • Group answers by learning type rather than listing classes at random.

9

How do you encode categorical variables with scikit-learn?

Scikit-learn offers several encoders for categorical features, chosen by the type of category and the model. OneHotEncoder creates one binary column per category: the default for nominal categories with linear models, SVMs and neural networks, with handle_unknown="ignore" for unseen values and min_frequency/max_categories to group rare ones. OrdinalEncoder maps categories to integers, suitable for genuinely ordered categories (with categories=[...] in the right order) and for tree models. TargetEncoder (1.3+) replaces each category by a smoothed, cross-fitted mean of the target, which works well for high-cardinality features. LabelEncoder is only for target labels, not features. HistGradientBoosting* can also use categories natively.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Choosing an Encoder

Encoder Output Best for
OneHotEncoder One 0/1 column per category Nominal, low-to-medium cardinality, linear models
OrdinalEncoder Integer codes Ordered categories; tree models
TargetEncoder Smoothed target mean (cross-fitted) High cardinality (ZIP codes, product IDs)
HistGradientBoosting(categorical_features="from_dtype") Native splits Gradient boosting with ≤255 categories per feature
LabelEncoder Integer codes The target y only

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder

rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
    "color": rng.choice(["red", "green", "blue"], n),
    "size": rng.choice(["S", "M", "L", "XL"], n),
    "store_id": rng.choice([f"store_{i}" for i in range(300)], n),   # high cardinality
})
store_effect = dict(zip([f"store_{i}" for i in range(300)], rng.normal(0, 1, 300)))
logit = (df["color"] == "red") * 0.8 + df["size"].map({"S": -0.5, "M": 0, "L": 0.3, "XL": 0.6}) \
        + df["store_id"].map(store_effect)
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

print(OneHotEncoder(sparse_output=False).fit(df[["color"]]).get_feature_names_out())
print(OrdinalEncoder(categories=[["S", "M", "L", "XL"]]).fit_transform(df[["size"]].head(4)).ravel())

encoders = {
    "one-hot everything": ColumnTransformer([
        ("oh", OneHotEncoder(handle_unknown="ignore", min_frequency=5), ["color", "size", "store_id"])]),
    "one-hot + ordinal + target": ColumnTransformer([
        ("oh", OneHotEncoder(handle_unknown="ignore"), ["color"]),
        ("ord", OrdinalEncoder(categories=[["S", "M", "L", "XL"]]), ["size"]),
        ("te", TargetEncoder(random_state=42), ["store_id"]),
    ]),
}
for name, prep in encoders.items():
    model = make_pipeline(prep, LogisticRegression(max_iter=2000))
    print(f"{name:>28}: CV ROC AUC {cross_val_score(model, df, y, cv=5, scoring='roc_auc').mean():.3f}")

TargetEncoder uses internal cross-fitting during fit_transform, so a row’s encoding never uses its own label, which limits leakage compared with a naive target mean.

Common Pitfalls

  • Using LabelEncoder on features (it is for y and ignores order).

  • Ordinal-encoding nominal categories for linear models, which invents an order.

  • Computing target means on the full dataset outside cross-validation.

10

What strategies does scikit-learn offer for imbalanced datasets?

Scikit-learn handles class imbalance mainly through weighting, threshold control, splitting and metrics. Many classifiers accept class_weight="balanced" (or a dict), which increases the loss for minority-class errors; sample_weight in fit gives finer control. Stratified splitting (stratify=y, StratifiedKFold) keeps class proportions stable across folds. Threshold tuning with TunedThresholdClassifierCV (1.5+) picks the decision threshold that optimizes a chosen metric or cost, and FixedThresholdClassifier applies a known one. Imbalance-aware metrics include balanced_accuracy_score, f1_score, average_precision_score and classification_report. Resampling methods such as SMOTE are in the separate imbalanced-learn package, whose pipeline resamples only training folds.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Strategies

Strategy Tool Notes
Class weights class_weight="balanced" Logistic regression, SVM, trees, forests, HistGB
Sample weights fit(X, y, sample_weight=w) Custom costs
Stratification stratify=y, StratifiedKFold Stable evaluation
Threshold tuning TunedThresholdClassifierCV, FixedThresholdClassifier Uses CV to choose the cut-off
Metrics average_precision_score, balanced_accuracy_score, F1 Avoid plain accuracy
Resampling imbalanced-learn (SMOTE, RandomUnderSampler) Use imblearn.pipeline.Pipeline
Calibration CalibratedClassifierCV Weighting distorts probabilities

Code Example

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, balanced_accuracy_score, classification_report, f1_score
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=10_000, n_features=20, n_informative=6, weights=[0.97, 0.03],
                           flip_y=0.01, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25, random_state=42)

models = {
    "plain": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "balanced weights": make_pipeline(StandardScaler(), LogisticRegression(class_weight="balanced", max_iter=1000)),
    "tuned threshold (F1)": TunedThresholdClassifierCV(
        make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)), scoring="f1", cv=5),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    proba = model.predict_proba(X_test)[:, 1]
    print(f"{name:>20}: F1 {f1_score(y_test, pred):.3f}, balanced acc {balanced_accuracy_score(y_test, pred):.3f}, "
          f"PR AUC {average_precision_score(y_test, proba):.3f}")

print("chosen threshold:", round(models["tuned threshold (F1)"].best_threshold_, 3))
print(classification_report(y_test, models["tuned threshold (F1)"].predict(X_test), digits=3))

Class weights and threshold tuning change which errors the model makes, not how well it ranks cases, so PR AUC stays similar across the three, while F1 and recall on the minority class change.

Common Pitfalls

  • Oversampling before splitting, which copies minority samples into the test set.

  • Optimizing accuracy, which rewards predicting the majority class.

  • Using weighted-model probabilities as real risks without recalibration.

11

How do you split data into training and test sets with scikit-learn?

train_test_split from sklearn.model_selection randomly divides arrays or DataFrames into training and test subsets: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42). For classification, pass stratify=y so both sets keep the same class proportions; random_state makes the split reproducible; shuffle=False keeps the original order. When rows are related, use group-aware splitters (GroupShuffleSplit) so the same patient or user is not in both sets; for time series, split by time (TimeSeriesSplit or a date cut-off). Split before any preprocessing that learns from data, and add a separate validation set or cross-validation for tuning.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Options

Need Tool
Random split train_test_split(X, y, test_size=0.2, random_state=42)
Keep class balance stratify=y
Train/validation/test Call train_test_split twice
Grouped rows GroupShuffleSplit(test_size=0.2).split(X, y, groups)
Time order Date cut-off, or TimeSeriesSplit for CV
Reproducibility Fixed random_state

Code Example

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GroupShuffleSplit, train_test_split

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
print("sizes:", len(X_train), len(X_test))
print("class share train/test:", round(y_train.mean(), 3), round(y_test.mean(), 3))

# Train / validation / test (60 / 20 / 20)
X_tmp, X_test2, y_tmp, y_test2 = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.25, stratify=y_tmp, random_state=42)
print("train/val/test:", len(X_tr), len(X_val), len(X_test2))

# Group split: several visits per patient must stay together
rng = np.random.default_rng(42)
visits = pd.DataFrame({"patient_id": rng.integers(0, 100, 500), "value": rng.normal(size=500)})
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(visits, groups=visits["patient_id"]))
overlap = set(visits.iloc[train_idx]["patient_id"]) & set(visits.iloc[test_idx]["patient_id"])
print("patients in both sets:", len(overlap))

# Time-based split
events = pd.DataFrame({"date": pd.date_range("2025-01-01", periods=365, freq="D"), "y": rng.normal(size=365)})
cutoff = pd.Timestamp("2025-10-01")
train_time, test_time = events[events["date"] < cutoff], events[events["date"] >= cutoff]
print("time split sizes:", len(train_time), len(test_time))

The second test_size=0.25 takes 25% of the remaining 80%, which is 20% of the original data.

Common Pitfalls

  • Scaling or encoding the full dataset before splitting (leakage).

  • Random splits for time series or grouped data, which give overly optimistic scores.

  • Tuning on the test set; keep it for the final evaluation only.

12

What is ColumnTransformer used for in scikit-learn?

ColumnTransformer applies different preprocessing to different columns and concatenates the results into one feature matrix. Real datasets mix numeric, categorical and text columns, each needing its own treatment (impute and scale numbers, one-hot encode categories, vectorize text); ColumnTransformer takes a list of (name, transformer, columns) tuples, where columns can be names, positions, boolean masks or a selector such as make_column_selector(dtype_include="number"). The remainder argument decides whether unlisted columns are dropped (default) or passed through. Placed inside a Pipeline, it is fitted only on training folds, its parameters can be tuned (prep__num__impute__strategy), and get_feature_names_out() gives readable output names.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Key Options

Option Meaning
transformers=[(name, transformer, columns), ...] What to apply where
remainder="drop" / "passthrough" / a transformer Columns not listed
make_column_selector(dtype_include=..., pattern=...) Select columns by dtype or name pattern
"drop" / "passthrough" as transformer Remove or keep columns unchanged
sparse_threshold When to return a sparse matrix
verbose_feature_names_out Prefix output names with the step name
.set_output(transform="pandas") Return a DataFrame

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 800
df = pd.DataFrame({
    "size_m2": rng.normal(90, 25, n),
    "rooms": rng.integers(1, 6, n).astype(float),
    "city": rng.choice(["Pune", "Delhi", "Mumbai"], n),
    "description": rng.choice(["sea view balcony", "near metro", "quiet garden", "new building"], n),
    "listing_id": np.arange(n),
})
df.loc[rng.random(n) < 0.1, "size_m2"] = np.nan
price = (2000 * df["size_m2"].fillna(90) + 50000 * (df["city"] == "Mumbai")
         + 30000 * df["description"].str.contains("sea") + rng.normal(0, 20000, n))

preprocess = ColumnTransformer(
    [
        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()),
         make_column_selector(dtype_include="number", pattern="^(size_m2|rooms)$")),
        ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["city"]),
        ("text", TfidfVectorizer(), "description"),                  # a string, not a list, for text
        ("id", "drop", ["listing_id"]),
    ],
    remainder="drop",
    sparse_threshold=0,
)
model = make_pipeline(preprocess, HistGradientBoostingRegressor(random_state=42))
print("CV R²:", cross_val_score(model, df, price, cv=5, scoring="r2").mean().round(3))

preprocess.fit(df)
print(preprocess.get_feature_names_out())

Text vectorizers need a single column passed as a string, because they expect a 1-D sequence of documents; other transformers receive a 2-D selection from a list of names.

Common Pitfalls

  • Passing ["description"] (a list) to a text vectorizer, which then sees a 2-D frame.

  • Forgetting handle_unknown="ignore" for categories that appear only in new data.

  • Accidentally dropping useful columns with the default remainder="drop".

13

What preprocessing steps should you apply before training an ML model?

Preprocessing turns raw data into clean, model-ready features without leaking information. Typical steps are: understand and clean the data (types, duplicates, impossible values, consistent categories), split into train and test sets early (by time or group when needed), handle missing values (imputation, indicators), encode categorical variables (one-hot, ordinal, target encoding), scale numeric features for distance- and gradient-based models, transform skewed features (log, power transforms), engineer features (ratios, dates, aggregates), select or reduce features, and address class imbalance if needed. In scikit-learn, put all learnable steps into a Pipeline with ColumnTransformer so they are fitted only on training folds.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Checklist

Step scikit-learn tool Notes
Clean pandas Fix types, duplicates, typos
Split train_test_split, GroupShuffleSplit, TimeSeriesSplit Before learning any statistics
Missing values SimpleImputer, KNNImputer, add_indicator=True Statistics from training data
Categorical OneHotEncoder(handle_unknown="ignore"), OrdinalEncoder, TargetEncoder Rare categories: min_frequency
Scaling StandardScaler, RobustScaler Not needed for trees
Skew PowerTransformer, FunctionTransformer(np.log1p) For linear models
Features PolynomialFeatures, custom transformers Domain knowledge
Selection SelectKBest, SelectFromModel, VarianceThreshold Inside CV
Imbalance class_weight="balanced", threshold tuning Evaluate with suitable metrics

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 2000
df = pd.DataFrame({
    "age": rng.integers(18, 90, n).astype(float),
    "income": rng.lognormal(10.5, 0.8, n),
    "plan": rng.choice(["Basic", "basic ", "Pro", "Team"], n),          # inconsistent labels
    "constant_col": 1.0,
})
df.loc[rng.random(n) < 0.08, "income"] = np.nan
df["plan"] = df["plan"].str.strip().str.lower()                        # cleaning before modelling
y = (rng.random(n) < 1 / (1 + np.exp(-(0.03 * (df["age"] - 50) + (df["plan"] == "pro"))))).astype(int)

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, stratify=y, random_state=42)

numeric = make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
                        FunctionTransformer(np.log1p), StandardScaler())
preprocess = ColumnTransformer([
    ("age", StandardScaler(), ["age"]),
    ("income", numeric, ["income"]),
    ("plan", OneHotEncoder(handle_unknown="ignore", min_frequency=20), ["plan"]),
    ("const", "drop", ["constant_col"]),                               # zero variance: carries no information
])
model = make_pipeline(preprocess, LogisticRegression(class_weight="balanced", max_iter=1000))
print("CV ROC AUC:", cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc").mean().round(3))
model.fit(X_train, y_train)
print("features after preprocessing:", model[0].get_feature_names_out())

Cleaning that needs no learned statistics (fixing label spelling) can happen before the split; anything that learns from data (medians, categories, scales) lives in the pipeline.

Common Pitfalls

  • Fitting preprocessing on all data before splitting.

  • One-hot encoding high-cardinality IDs, which creates huge sparse matrices and overfitting.

  • Scaling features for tree models, which wastes effort without changing results.

14

How does SimpleImputer handle missing data in scikit-learn?

Scikit-learn’s imputers fill missing values with values learned from the training data. The old Imputer class was removed long ago; today the main tool is SimpleImputer in sklearn.impute, which replaces missing entries (by default np.nan) in each column with the column’s mean, median, most frequent value, or a constant, all learned in fit and applied in transform. add_indicator=True appends binary “was missing” columns. More advanced options are KNNImputer (average of the nearest neighbours’ values) and IterativeImputer (experimental; models each feature from the others). Some estimators, such as HistGradientBoosting* and decision trees (1.3+), handle NaN natively.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Imputers

Imputer Strategy Notes
SimpleImputer(strategy="mean") Column mean Numeric, roughly symmetric
SimpleImputer(strategy="median") Column median Skewed numeric
SimpleImputer(strategy="most_frequent") Mode Categorical
SimpleImputer(strategy="constant", fill_value="missing") Fixed value Makes missingness a category
KNNImputer(n_neighbors=5) Neighbour average Uses relationships between rows
IterativeImputer Model-based, round-robin from sklearn.experimental import enable_iterative_imputer
MissingIndicator / add_indicator=True Binary flags Missingness can be predictive

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X = pd.DataFrame({"age": [25, np.nan, 40, 35, np.nan],
                  "income": [30000, 42000, np.nan, 52000, 61000],
                  "city": ["A", "B", np.nan, "A", "A"]})

num_imputer = SimpleImputer(strategy="median", add_indicator=True).set_output(transform="pandas")
print(num_imputer.fit_transform(X[["age", "income"]]))
print("learned medians:", num_imputer.statistics_)

cat_imputer = SimpleImputer(strategy="most_frequent")
print("city filled:", cat_imputer.fit_transform(X[["city"]]).ravel())

print("KNN imputed:\n", KNNImputer(n_neighbors=2).fit_transform(X[["age", "income"]]).round(1))

# In a full pipeline with a model
rng = np.random.default_rng(42)
big = pd.DataFrame({"age": rng.normal(40, 10, 400), "income": rng.normal(50000, 10000, 400),
                    "city": rng.choice(["A", "B", "C"], 400)})
y = (big["age"] + big["income"] / 2000 + rng.normal(0, 5, 400) > 65).astype(int)
big.loc[rng.random(400) < 0.15, "age"] = np.nan
big.loc[rng.random(400) < 0.10, "city"] = np.nan
prep = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), ["age", "income"]),
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore")), ["city"]),
])
model = make_pipeline(prep, LogisticRegression(max_iter=1000))
print("CV accuracy:", cross_val_score(model, big, y, cv=5).mean().round(3))

The medians are learned from the training data and reused for new data, and inside cross-validation they are recomputed for each fold automatically.

Common Pitfalls

  • Imputing the full dataset before splitting.

  • Using mean imputation for skewed or categorical data.

  • Dropping the information that a value was missing when missingness is informative.

15

How do you normalize or standardize data with scikit-learn?

Scikit-learn’s sklearn.preprocessing module provides scalers that learn statistics from training data and apply them consistently. StandardScaler standardizes each feature to mean 0 and standard deviation 1; MinMaxScaler rescales to a range, by default [0, 1]; RobustScaler uses the median and IQR, so outliers have less influence; MaxAbsScaler divides by the maximum absolute value (keeps sparse data sparse). Note that Normalizer is different: it scales each row (sample) to unit length, as used for text vectors and cosine similarity. PowerTransformer and QuantileTransformer also change the distribution shape. Put the scaler in a Pipeline, fit it on training data only, and remember tree models do not need scaling.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Scalers

Scaler Result Use when
StandardScaler Mean 0, SD 1 per feature Linear models, SVM, k-NN, PCA, neural nets
MinMaxScaler Range [0, 1] Bounded inputs, image pixels
RobustScaler Median 0, IQR-based scale Outliers
MaxAbsScaler Range [−1, 1] Sparse data
Normalizer Each row has unit norm Text/TF-IDF, cosine similarity
PowerTransformer More Gaussian-like Skewed features

Code Example

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, Normalizer, RobustScaler, StandardScaler

rng = np.random.default_rng(42)
df = pd.DataFrame({
    "age": rng.normal(40, 12, 500),
    "income": np.r_[rng.lognormal(10.5, 0.4, 495), [5e6, 6e6, 7e6, 8e6, 9e6]],   # a few extreme incomes
})
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)

for scaler in (StandardScaler(), MinMaxScaler(), RobustScaler()):
    scaler.set_output(transform="pandas")
    train_scaled = scaler.fit_transform(X_train)            # fit on training data only
    test_scaled = scaler.transform(X_test)
    print(f"{type(scaler).__name__:>14}: income median {train_scaled['income'].median():8.3f}, "
          f"IQR {train_scaled['income'].quantile(0.75) - train_scaled['income'].quantile(0.25):8.3f}, "
          f"test age mean {test_scaled['age'].mean():.3f}")

rows = Normalizer().fit_transform(X_train.iloc[:3])
print("Normalizer row norms:", np.linalg.norm(rows, axis=1).round(3))   # each sample scaled to length 1

With extreme incomes, StandardScaler and MinMaxScaler squeeze most values into a tiny range (small IQR), while RobustScaler keeps typical incomes spread out. set_output(transform="pandas") keeps column names.

Common Pitfalls

  • Fitting the scaler on the full dataset before splitting.

  • Confusing Normalizer (rows) with MinMaxScaler (columns).

  • Scaling one-hot or binary columns unnecessarily; apply scalers only to numeric columns with ColumnTransformer.

16

What is the process of training a supervised model with scikit-learn?

Training a supervised model in scikit-learn follows a consistent process: load data into a feature matrix X and target y; split off a test set with train_test_split(stratify=y); build a Pipeline of preprocessing (imputation, scaling, encoding via ColumnTransformer) and an estimator; compare against a Dummy baseline and a few candidate models with cross-validation on the training set; tune hyperparameters with GridSearchCV or RandomizedSearchCV; refit the best pipeline on all training data; evaluate once on the test set with suitable metrics; then save the pipeline with joblib. Every estimator uses the same fit/predict API, which makes these steps interchangeable across algorithms.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Steps

Step Tool
1. Data load_*, pandas, X and y
2. Split train_test_split(..., stratify=y, random_state=42)
3. Preprocess ColumnTransformer, SimpleImputer, StandardScaler, OneHotEncoder
4. Baseline DummyClassifier / DummyRegressor
5. Compare models cross_val_score
6. Tune GridSearchCV with model__param names
7. Evaluate classification_report, roc_auc_score on the test set
8. Save joblib.dump(best_pipeline, ...)

Code Example

import tempfile
from pathlib import Path

import joblib
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import GridSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

candidates = {
    "dummy": DummyClassifier(strategy="most_frequent"),
    "logistic": Pipeline([("scale", StandardScaler()), ("model", LogisticRegression(max_iter=5000))]),
    "forest": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in candidates.items():
    print(f"{name:>8}: CV ROC AUC {cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc').mean():.4f}")

search = GridSearchCV(candidates["logistic"], {"model__C": [0.01, 0.1, 1, 10]}, cv=5, scoring="roc_auc")
search.fit(X_train, y_train)
best = search.best_estimator_
print("best C:", search.best_params_["model__C"])

proba = best.predict_proba(X_test)[:, 1]
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))
print(classification_report(y_test, best.predict(X_test), digits=3))

path = Path(tempfile.mkdtemp()) / "breast_cancer_model.joblib"
joblib.dump(best, path)
print("saved to", path)

The dummy baseline (ROC AUC 0.5) shows how much the real models add; the test set is used only once, at the end.

Common Pitfalls

  • Looking at the test set repeatedly while choosing models.

  • Preprocessing before splitting.

  • Skipping the baseline, which hides whether the model learned anything useful.

17

How do you perform cross-validation with scikit-learn?

Cross-validation in scikit-learn estimates how a model will perform on unseen data by repeatedly training on part of the data and scoring on the rest. The quickest tool is cross_val_score(model, X, y, cv=5, scoring=...), which returns one score per fold; cross_validate supports several metrics, training scores and fit times; cross_val_predict returns out-of-fold predictions for every sample. The cv argument takes a number of folds or a splitter: KFold, StratifiedKFold (default for classifiers), GroupKFold/StratifiedGroupKFold for grouped data, TimeSeriesSplit for temporal data, RepeatedStratifiedKFold for more stable estimates. Always pass a Pipeline so preprocessing is refit in each fold.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Splitters

Splitter Use when
KFold(shuffle=True) Regression, independent rows
StratifiedKFold Classification; keeps class proportions
GroupKFold / StratifiedGroupKFold Several rows per patient, user or session
TimeSeriesSplit Ordered data; always trains on the past
RepeatedStratifiedKFold Small datasets, smoother estimates
LeaveOneOut Very small datasets (high variance, slow)

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import (GroupKFold, RepeatedStratifiedKFold, TimeSeriesSplit, cross_val_predict,
                                     cross_val_score, cross_validate)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print("5-fold ROC AUC:", scores.round(4), "mean", scores.mean().round(4))

results = cross_validate(model, X, y, cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=42),
                         scoring=["accuracy", "f1"], return_train_score=True)
print("repeated CV accuracy:", results["test_accuracy"].mean().round(4), "±", results["test_accuracy"].std().round(4))
print("train accuracy:", results["train_accuracy"].mean().round(4))

oof = cross_val_predict(model, X, y, cv=5)
print("out-of-fold confusion matrix:\n", confusion_matrix(y, oof))

groups = np.arange(len(y)) // 5                     # pretend every 5 rows come from one patient
print("GroupKFold accuracy:", cross_val_score(model, X, y, cv=GroupKFold(n_splits=5), groups=groups).mean().round(4))

for i, (train_idx, test_idx) in enumerate(TimeSeriesSplit(n_splits=3).split(X)):
    print(f"time split {i}: train rows 0-{train_idx[-1]}, test rows {test_idx[0]}-{test_idx[-1]}")

Out-of-fold predictions give every sample a prediction from a model that did not see it, useful for error analysis and stacking (but not as a substitute for a final test set).

Common Pitfalls

  • Scaling or selecting features before cross-validation (leakage).

  • Shuffled folds on time series or grouped data, which inflates scores.

  • Using cross_val_predict output to compute a single “CV score” when folds differ in size; average per-fold scores instead.

18

Which scikit-learn metrics suit regression versus classification models?

Regression metrics measure how far numeric predictions are from true values; classification metrics measure how well labels or probabilities match true classes. For regression, scikit-learn provides mean_absolute_error (MAE), mean_squared_error and root_mean_squared_error (RMSE, 1.4+), r2_score, mean_absolute_percentage_error, median_absolute_error and explained_variance_score. For classification: accuracy_score, precision_score, recall_score, f1_score, balanced_accuracy_score, confusion_matrix, classification_report for label quality, and roc_auc_score, average_precision_score, log_loss and brier_score_loss for probability quality. In model selection, use the scoring strings ("neg_mean_absolute_error", "roc_auc"), where error metrics are negated so higher is always better.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Regression Metrics

Metric Function Notes
MAE mean_absolute_error Average error in target units; robust to outliers
RMSE root_mean_squared_error Penalizes large errors
R² r2_score Share of variance explained; can be negative
MAPE mean_absolute_percentage_error Relative error; unstable near zero
Median AE median_absolute_error Very robust

Classification Metrics

Metric Function Notes
Accuracy accuracy_score Misleading with imbalance
Precision / recall / F1 precision_score, recall_score, f1_score Choose average= for multiclass
Balanced accuracy balanced_accuracy_score Mean recall per class
ROC AUC roc_auc_score Ranking quality, threshold-free
PR AUC average_precision_score Better for rare positives
Log loss / Brier log_loss, brier_score_loss Probability quality and calibration

Code Example

from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (average_precision_score, balanced_accuracy_score, classification_report, f1_score,
                             log_loss, mean_absolute_error, r2_score, roc_auc_score, root_mean_squared_error)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xr, yr = load_diabetes(return_X_y=True)
Xr_tr, Xr_te, yr_tr, yr_te = train_test_split(Xr, yr, random_state=42)
pred = RandomForestRegressor(n_estimators=300, random_state=42).fit(Xr_tr, yr_tr).predict(Xr_te)
print(f"regression: MAE {mean_absolute_error(yr_te, pred):.2f}, RMSE {root_mean_squared_error(yr_te, pred):.2f}, "
      f"R² {r2_score(yr_te, pred):.3f}")

Xc, yc = load_breast_cancer(return_X_y=True)
Xc_tr, Xc_te, yc_tr, yc_te = train_test_split(Xc, yc, stratify=yc, random_state=42)
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000)).fit(Xc_tr, yc_tr)
proba = clf.predict_proba(Xc_te)[:, 1]
labels = clf.predict(Xc_te)
print(f"classification: F1 {f1_score(yc_te, labels):.3f}, balanced acc {balanced_accuracy_score(yc_te, labels):.3f}, "
      f"ROC AUC {roc_auc_score(yc_te, proba):.3f}, PR AUC {average_precision_score(yc_te, proba):.3f}, "
      f"log loss {log_loss(yc_te, proba):.3f}")
print(classification_report(yc_te, labels, digits=3))

Regression metrics are in the target’s units (here, disease progression score), which makes MAE easy to explain; classification probability metrics such as ROC AUC and log loss judge the scores before any threshold is applied.

Common Pitfalls

  • Reporting accuracy alone for imbalanced classes.

  • Using R² to compare models on different datasets or targets.

  • Passing labels instead of probabilities to roc_auc_score.

19

What is GridSearchCV, and what is it used for?

GridSearchCV finds good hyperparameters by exhaustively trying every combination in a parameter grid, evaluating each with cross-validation, and selecting the one with the best mean score. With refit=True (default), it then retrains the best configuration on all the data passed to fit, so the search object itself works like the best model (predict, score). Results are in best_params_, best_score_, best_estimator_ and the detailed cv_results_. It accepts any estimator or Pipeline (parameters named step__param), a scoring string or several metrics, a CV splitter, and n_jobs for parallelism.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Key Arguments

Argument Purpose
estimator Model or pipeline
param_grid Dict (or list of dicts) of parameter values
scoring Metric, e.g. "roc_auc", "f1_macro", or a list/dict
cv Number of folds or a splitter (StratifiedKFold, GroupKFold)
refit Retrain best model; with several metrics, name the one to use
n_jobs Parallel fits
return_train_score Diagnose overfitting

Code Example

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25, random_state=42)

pipe = Pipeline([("scale", StandardScaler()), ("svc", SVC())])
param_grid = [
    {"svc__kernel": ["linear"], "svc__C": [0.1, 1, 10]},
    {"svc__kernel": ["rbf"], "svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.01, 0.1]},
]
search = GridSearchCV(pipe, param_grid, scoring={"auc": "roc_auc", "acc": "accuracy"}, refit="auc",
                      cv=StratifiedKFold(5, shuffle=True, random_state=42), n_jobs=-1, return_train_score=True)
search.fit(X_train, y_train)

print("best parameters:", search.best_params_)
print("best CV ROC AUC:", round(search.best_score_, 4))
print("test ROC AUC of refitted best model:", round(search.score(X_test, y_test), 4))

results = pd.DataFrame(search.cv_results_)
cols = ["param_svc__kernel", "param_svc__C", "param_svc__gamma", "mean_test_auc", "mean_test_acc", "rank_test_auc"]
print(results[cols].sort_values("rank_test_auc").head(5).to_string(index=False))

A list of grids avoids meaningless combinations (gamma does not apply to the linear kernel). With several metrics, refit names the one that chooses the winner, and search.score uses it.

Common Pitfalls

  • Grids that grow combinatorially; use RandomizedSearchCV for many parameters.

  • Reporting best_score_ as the final performance (it is optimistically biased).

  • Preprocessing outside the pipeline, which leaks information into each fold.

20

What is the difference between fit(), predict() and transform()?

fit(X, y) learns from data, predict(X) uses what was learned to output predictions, and transform(X) uses what was learned to output a changed version of the data. In scikit-learn, every estimator has fit, which stores learned parameters in attributes ending in an underscore (coef_, mean_) and returns self. Predictors (classifiers, regressors, clusterers) add predict, and classifiers also offer predict_proba or decision_function. Transformers (scalers, encoders, PCA, vectorizers) add transform, and fit_transform does both in one step, often more efficiently. The key rule: call fit (or fit_transform) only on training data, and only transform/predict on validation, test and production data.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Methods

Method Available on Does Returns
fit(X, y=None) All estimators Learns parameters self
predict(X) Classifiers, regressors, some clusterers Uses parameters to predict Labels or values
predict_proba(X) Most classifiers Class probabilities Array (n_samples, n_classes)
transform(X) Transformers Applies a learned transformation Transformed data
fit_transform(X) Transformers Fit then transform the same data Transformed data
fit_predict(X) Clusterers Fit and return labels Labels
score(X, y) Predictors Default metric (accuracy, R²) Float

Code Example

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

scaler = StandardScaler()
scaler.fit(X_train)                                  # learns mean_ and scale_ from training data
print("learned means:", scaler.mean_.round(2))
X_train_s = scaler.transform(X_train)               # applies them
X_test_s = scaler.transform(X_test)                 # same means and scales, no refitting
print("test feature means after scaling (not exactly 0):", X_test_s.mean(axis=0).round(2))

model = LogisticRegression(max_iter=1000).fit(X_train_s, y_train)
print("predictions:", model.predict(X_test_s[:5]))
print("probabilities:", model.predict_proba(X_test_s[:2]).round(3))
print("accuracy:", round(model.score(X_test_s, y_test), 3))

# A pipeline calls the right methods automatically
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
print("pipeline accuracy:", round(pipe.score(X_test, y_test), 3))

The test set’s scaled means are close to, but not exactly, zero because the scaler reused the training statistics, which is the correct behaviour.

Common Pitfalls

  • Calling fit_transform on test data, which leaks test statistics and makes features inconsistent.

  • Expecting transform on a classifier or predict on a scaler.

  • Forgetting that fit returns self, which allows chaining such as model.fit(X, y).predict(X_new).

21

What is overfitting, and how do you detect it with scikit-learn?

Overfitting happens when a model learns noise and quirks of the training data rather than the general pattern, so it performs much better on training data than on new data. In scikit-learn you detect it by comparing training and validation scores: cross_validate(..., return_train_score=True) shows the gap, validation_curve shows how the gap changes with a complexity parameter (tree depth, C, number of neighbours), and learning_curve shows whether more data closes it. Remedies include regularization, simpler models, early stopping, more data, feature selection and ensembling. A model that scores poorly on both sets is underfitting instead.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Signs and Fixes

Diagnostic Overfitting sign
Train vs CV score Large gap (e.g. 1.00 vs 0.85)
Validation curve CV score falls as complexity rises while train score keeps rising
Learning curve Big gap that narrows slowly as data grows
Test vs CV Test much worse than CV (possible leakage or over-tuning)
Fix scikit-learn tool
Regularization Ridge, Lasso, C in LogisticRegression/SVC
Limit complexity max_depth, min_samples_leaf, ccp_alpha
Early stopping HistGradientBoosting(early_stopping=True), MLPClassifier(early_stopping=True)
Feature selection SelectKBest, SelectFromModel
Ensembles RandomForestClassifier, bagging

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(n_samples=1500, n_features=20, n_informative=5, flip_y=0.1, random_state=42)

depths = [1, 2, 4, 6, 8, 12, 16, None]
param_range = [d if d is not None else 50 for d in depths]
train_scores, val_scores = validation_curve(
    DecisionTreeClassifier(random_state=42), X, y,
    param_name="max_depth", param_range=param_range, cv=5, scoring="accuracy")
for d, tr, va in zip(param_range, train_scores.mean(1), val_scores.mean(1)):
    print(f"max_depth={d:>2}: train {tr:.3f}, validation {va:.3f}, gap {tr - va:.3f}")

best = param_range[int(np.argmax(val_scores.mean(1)))]
sizes, tr_lc, va_lc = learning_curve(DecisionTreeClassifier(max_depth=best, random_state=42), X, y,
                                     cv=5, train_sizes=np.linspace(0.1, 1.0, 5))
print("learning curve gaps:", (tr_lc.mean(1) - va_lc.mean(1)).round(3))

As depth grows, training accuracy approaches 1.0 while validation accuracy peaks and then declines: the region after the peak is overfitting. The 10% label noise (flip_y=0.1) means no honest model can reach 100% on validation data.

Common Pitfalls

  • Judging only on training accuracy.

  • “Fixing” overfitting by tuning repeatedly on the same validation set, which overfits the validation set.

  • Mistaking leakage (great validation, poor production) for a well-generalizing model.

22

How do you build ensemble models with scikit-learn?

Scikit-learn offers ensembles at three levels. Ready-made tree ensembles: RandomForest* and ExtraTrees* (bagging with random features), GradientBoosting* and HistGradientBoosting* (boosting), and AdaBoost*. Generic wrappers: BaggingClassifier/Regressor (bootstrap any base model). And model combiners: VotingClassifier/Regressor (average predictions or probabilities from different models) and StackingClassifier/Regressor (a final estimator learns how to combine base models’ out-of-fold predictions). Ensembles work best when the members are accurate but make different errors, for example a linear model, a tree ensemble and a k-NN.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Ensemble Types

Type scikit-learn class Idea
Bagging BaggingClassifier, RandomForestClassifier, ExtraTreesClassifier Average many models on resampled data
Boosting GradientBoostingClassifier, HistGradientBoostingClassifier, AdaBoostClassifier Sequentially correct errors
Voting VotingClassifier(voting="soft") Average probabilities of different models
Stacking StackingClassifier(final_estimator=...) Meta-model on out-of-fold predictions

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import (BaggingClassifier, HistGradientBoostingClassifier, RandomForestClassifier,
                              StackingClassifier, VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)

base = [
    ("logreg", make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))),
    ("knn", make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15))),
    ("hgb", HistGradientBoostingClassifier(random_state=42)),
]
models = dict(base)
models["bagged trees"] = BaggingClassifier(DecisionTreeClassifier(), n_estimators=200, random_state=42)
models["random forest"] = RandomForestClassifier(n_estimators=300, random_state=42)
models["soft voting"] = VotingClassifier(base, voting="soft")
models["stacking"] = StackingClassifier(base, final_estimator=LogisticRegression(max_iter=1000), cv=5)

for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
    print(f"{name:>14}: ROC AUC {scores.mean():.4f} ± {scores.std():.4f}")

Compare ensembles with their best single member: gains are often small on clean tabular data, and the extra complexity must be worth it. Stacking uses internal cross-validation (cv=5) so the meta-model never trains on in-sample predictions.

Common Pitfalls

  • Combining near-identical models, which adds cost without diversity.

  • Hard voting when good probabilities are available (soft voting usually works better).

  • Stacking without proper cross-validation, which leaks training predictions into the meta-model.

23

How does scikit-learn construct a decision tree?

Scikit-learn builds decision trees with an optimized CART algorithm: starting from all training samples at the root, it searches every feature and candidate threshold for the binary split that most reduces impurity, Gini or entropy (log-loss) for classification, squared error (or others) for regression, then recursively repeats the process on each child node. Splitting stops when a node is pure or a limit is reached: max_depth, min_samples_split, min_samples_leaf, max_leaf_nodes or min_impurity_decrease. Leaves predict the majority class (with class proportions as probabilities) or the mean target. Cost-complexity pruning with ccp_alpha trims the grown tree, and max_features adds randomness, which random forests use.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Key Parameters

Parameter Effect
criterion "gini", "entropy", "log_loss" (classification); "squared_error", "absolute_error", "poisson" (regression)
max_depth Maximum depth (main overfitting control)
min_samples_leaf Minimum samples per leaf
min_samples_split Minimum samples to split a node
max_features Features considered per split
ccp_alpha Post-pruning strength
class_weight Reweights classes in the impurity calculation

Trees need no feature scaling. Since scikit-learn 1.3, trees also support missing values (NaN) for some splitters.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, stratify=data.target, random_state=42)

full = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
print(f"unpruned: depth {full.get_depth()}, leaves {full.get_n_leaves()}, "
      f"train {full.score(X_train, y_train):.3f}, test {full.score(X_test, y_test):.3f}")

# Cost-complexity pruning path: choose ccp_alpha by cross-validation
path = full.cost_complexity_pruning_path(X_train, y_train)
alphas = path.ccp_alphas[::max(1, len(path.ccp_alphas) // 10)]
cv_scores = [cross_val_score(DecisionTreeClassifier(ccp_alpha=a, random_state=42), X_train, y_train, cv=5).mean()
             for a in alphas]
best_alpha = alphas[int(np.argmax(cv_scores))]
pruned = DecisionTreeClassifier(ccp_alpha=best_alpha, random_state=42).fit(X_train, y_train)
print(f"pruned (alpha={best_alpha:.4f}): depth {pruned.get_depth()}, leaves {pruned.get_n_leaves()}, "
      f"test {pruned.score(X_test, y_test):.3f}")

shallow = DecisionTreeClassifier(max_depth=2, random_state=42).fit(X_train, y_train)
print(export_text(shallow, feature_names=list(data.feature_names)))

export_text prints the learned rules; each split is the threshold on one feature that reduced Gini impurity the most at that node.

Common Pitfalls

  • Unlimited depth memorizing the training data.

  • Treating the tree as stable: small data changes can produce very different trees.

  • Trusting impurity-based importances for high-cardinality features.

24

How do RandomForestClassifier and GradientBoostingClassifier differ?

Both are tree ensembles, but they build trees differently. RandomForestClassifier uses bagging: it grows many deep, independent trees in parallel, each on a bootstrap sample with a random subset of features at each split, and averages their predictions, which mainly reduces variance. GradientBoostingClassifier uses boosting: it builds shallow trees sequentially, each fitted to the gradient of the loss (the errors) of the current ensemble and added with a learning_rate, which mainly reduces bias. Random forests are robust with little tuning and parallelize easily; gradient boosting is often more accurate when tuned but is sequential, more sensitive to hyperparameters and can overfit. For speed on larger data, prefer HistGradientBoostingClassifier.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Aspect RandomForestClassifier GradientBoostingClassifier
Ensemble method Bagging + random features Boosting (sequential)
Trees Deep, independent Shallow (max_depth=3 default), dependent
Main effect Lower variance Lower bias
Training Parallel (n_jobs) Sequential
Key parameters n_estimators, max_features, min_samples_leaf n_estimators, learning_rate, max_depth, subsample
Overfitting with more trees Rarely Yes, without early stopping
Early stopping No (use OOB score for estimates) n_iter_no_change, validation_fraction
Tuning effort Low Higher
Faster alternative ExtraTreesClassifier HistGradientBoostingClassifier

Code Example

import time

from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.model_selection import cross_validate

X, y = make_classification(n_samples=4000, n_features=25, n_informative=10, flip_y=0.03, random_state=42)

models = {
    "RandomForest": RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=42),
    "GradientBoosting": GradientBoostingClassifier(n_estimators=300, learning_rate=0.1, max_depth=3,
                                                   n_iter_no_change=10, random_state=42),
    "HistGradientBoosting": HistGradientBoostingClassifier(random_state=42),
}
for name, model in models.items():
    start = time.perf_counter()
    cv = cross_validate(model, X, y, cv=5, scoring="roc_auc", return_train_score=True)
    elapsed = time.perf_counter() - start
    print(f"{name:>20}: train AUC {cv['train_score'].mean():.3f}, CV AUC {cv['test_score'].mean():.3f}, {elapsed:.1f}s")

The random forest usually reaches a near-perfect training score (deep trees) with a decent CV score, while boosting’s train and CV scores are closer together. Timings show the cost of sequential boosting versus the histogram-based version.

Common Pitfalls

  • Using a high learning rate with many boosting rounds and no early stopping.

  • Expecting random forests to extrapolate beyond the training range (no tree ensemble does).

  • Reading impurity-based feature importances as reliable; prefer permutation importance.

25

How does scikit-learn's SVM handle non-linear data?

Scikit-learn’s SVMs handle non-linear data with the kernel trick. A kernel function computes similarity between points as if they had been mapped into a higher-dimensional feature space, where a linear separating hyperplane may exist, without ever computing that mapping explicitly. SVC and SVR support kernel="rbf" (the default, Gaussian), "poly", "sigmoid", "linear", a callable, or "precomputed" Gram matrices. The RBF kernel’s gamma sets how far each training point’s influence reaches (large gamma = very local, wiggly boundaries), and C trades margin width against training errors; both are tuned together with cross-validation. For large datasets, approximate kernel maps (Nystroem, RBFSampler) with a linear model scale better.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Kernels

Kernel Formula (simplified) Key parameters
Linear x · z C
RBF exp(−γ ‖x − z‖²) C, gamma
Polynomial (γ x · z + r)^d C, degree, gamma, coef0
Sigmoid tanh(γ x · z + r) C, gamma, coef0

gamma="scale" (default) uses 1 / (n_features × X.var()).

Code Example

from sklearn.datasets import make_moons
from sklearn.kernel_approximation import Nystroem
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)

models = {
    "linear SVC": make_pipeline(StandardScaler(), SVC(kernel="linear")),
    "RBF SVC": make_pipeline(StandardScaler(), SVC(kernel="rbf")),
    "poly SVC (degree 3)": make_pipeline(StandardScaler(), SVC(kernel="poly", degree=3, coef0=1)),
    "explicit poly features + linear": make_pipeline(StandardScaler(), PolynomialFeatures(3),
                                                     SVC(kernel="linear")),
    "Nystroem + SGD (scalable)": make_pipeline(StandardScaler(), Nystroem(gamma=1.0, n_components=200, random_state=42),
                                               SGDClassifier(random_state=42)),
}
for name, model in models.items():
    print(f"{name:>32}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

search = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
                      {"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.01, 0.1, 1, 10]}, cv=5)
search.fit(X, y)
print("best RBF parameters:", search.best_params_, "| CV accuracy:", round(search.best_score_, 3))

The linear SVC cannot separate the interleaving moons; the RBF and polynomial kernels can. Explicit polynomial features reach a similar result by building the feature space directly, which the kernel trick avoids for high degrees.

Common Pitfalls

  • Tuning C and gamma separately instead of jointly.

  • Very large gamma memorizing the training data.

  • Forgetting scaling, which distorts RBF distances.

26

How does scikit-learn's KMeans implement k-means clustering?

KMeans partitions data into k clusters by minimizing inertia, the sum of squared distances from points to their cluster centre. Scikit-learn initializes centres with k-means++ (spread-out seeds that speed convergence), then repeats two steps until centres stop moving (within tol) or max_iter is reached: assign each point to its nearest centre, and update each centre to the mean of its points. Because results depend on initialization, it runs n_init times and keeps the lowest inertia (n_init="auto" uses 1 run with k-means++). The default algorithm="lloyd" is the classic method; MiniBatchKMeans uses small random batches for large data. After fitting: labels_, cluster_centers_, inertia_, and predict for new points.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Process

Step Detail
1. Scale features k-means uses Euclidean distance
2. Initialize k-means++ seeding (init="k-means++")
3. Assign Each point → nearest centre
4. Update Centre = mean of assigned points
5. Repeat Until convergence or max_iter
6. Restarts n_init runs; keep the lowest inertia
7. Choose k Elbow on inertia, silhouette score

Code Example

from sklearn.cluster import KMeans, MiniBatchKMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.preprocessing import StandardScaler

X, y_true = make_blobs(n_samples=1500, centers=4, cluster_std=[1.0, 1.5, 0.8, 1.2], random_state=42)
X = StandardScaler().fit_transform(X)

for k in range(2, 7):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    print(f"k={k}: inertia {km.inertia_:8.1f}, silhouette {silhouette_score(X, km.labels_):.3f}, iterations {km.n_iter_}")

km = KMeans(n_clusters=4, n_init=10, random_state=42).fit(X)
print("ARI vs true groups:", round(adjusted_rand_score(y_true, km.labels_), 3))
print("cluster centres:\n", km.cluster_centers_.round(2))
print("new points assigned to:", km.predict(X[:3]))

mbk = MiniBatchKMeans(n_clusters=4, batch_size=256, n_init=3, random_state=42).fit(X)
print("MiniBatchKMeans inertia:", round(mbk.inertia_, 1), "vs KMeans:", round(km.inertia_, 1))

Inertia always decreases as k grows, so look for the “elbow” and the highest silhouette score (here at the true k = 4). MiniBatchKMeans gives slightly higher inertia in exchange for speed.

Common Pitfalls

  • Unscaled features dominating the distance.

  • Expecting k-means to find non-spherical or very unequal-sized clusters (use DBSCAN, HDBSCAN or Gaussian mixtures).

  • Treating cluster label numbers as meaningful or stable between runs.

27

How does scikit-learn's logistic regression differ from linear regression?

LinearRegression and LogisticRegression share a linear combination of features but differ in purpose, output, loss and defaults. LinearRegression is a regressor: it predicts continuous values by ordinary least squares, has no regularization (use Ridge, Lasso or ElasticNet for that), and its score is R². LogisticRegression is a classifier: it passes the linear score through the sigmoid (or softmax for multiclass), minimizes log-loss, applies L2 regularization by default (C=1.0), is fitted with iterative solvers (lbfgs, liblinear, saga), provides predict_proba and decision_function, and its score is accuracy.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison in scikit-learn

Aspect LinearRegression LogisticRegression
Task Regression Classification (binary, multiclass)
Output predict → real numbers predict → labels; predict_proba → probabilities
Loss Squared error Log-loss (cross-entropy)
Solution Closed form (least squares via scipy.linalg.lstsq) Iterative solvers
Regularization None (use Ridge/Lasso) L2 by default; C is the inverse strength
score() R² Accuracy
Coefficients Change in y per unit x Change in log-odds per unit x
Scaling Not needed for the fit Recommended (regularization and convergence)

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

Xr, yr = load_diabetes(return_X_y=True)
Xr_train, Xr_test, yr_train, yr_test = train_test_split(Xr, yr, random_state=42)
lin = LinearRegression().fit(Xr_train, yr_train)
print("LinearRegression score (R²):", round(lin.score(Xr_test, yr_test), 3))
print("first predictions:", lin.predict(Xr_test[:3]).round(1))

Xc, yc = load_breast_cancer(return_X_y=True)
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, stratify=yc, random_state=42)
for C in (0.01, 1.0, 100.0):
    log = make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=5000)).fit(Xc_train, yc_train)
    print(f"LogisticRegression C={C:>6}: accuracy {log.score(Xc_test, yc_test):.3f}, "
          f"sum |coef| {np.abs(log[-1].coef_).sum():.1f}")
print("probabilities:", log.predict_proba(Xc_test[:3]).round(3))

# Linear regression on 0/1 labels gives values outside [0, 1]
lpm = LinearRegression().fit(Xc_train, yc_train)
preds = lpm.predict(Xc_test)
print("linear model on labels, min/max prediction:", round(preds.min(), 2), round(preds.max(), 2))

Smaller C shrinks logistic coefficients more strongly, while LinearRegression has no such knob. Fitting LinearRegression to 0/1 labels produces “probabilities” outside [0, 1], which is why classification needs the logistic model.

Common Pitfalls

  • Assuming LogisticRegression is unregularized (pass penalty=None for that).

  • Comparing coefficients across differently scaled features.

  • Using score without remembering it means R² for one and accuracy for the other.

28

What is an SVM, and how is it used for classification and regression?

A support vector machine (SVM) for classification finds the boundary (hyperplane) that separates classes with the largest margin, the distance to the nearest training points, called support vectors; a soft margin controlled by C allows some violations, and kernels (RBF, polynomial) let it draw non-linear boundaries. For regression, support vector regression (SVR) fits a function that keeps most points within an ε-insensitive tube: errors smaller than ε cost nothing, larger ones are penalized linearly, and again only support vectors shape the model. In scikit-learn these are SVC/LinearSVC and SVR/LinearSVR; always scale features first.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Classification vs Regression

Aspect SVC SVR
Goal Maximum-margin class separation Function within an ε-tube
Loss Hinge loss ε-insensitive loss
Key parameters C, kernel, gamma C, epsilon, kernel, gamma
Support vectors Points on or inside the margin Points on or outside the tube
Output Class (predict), score (decision_function), probabilities with probability=True Continuous value
Linear version LinearSVC (large data) LinearSVR

Code Example

from sklearn.datasets import load_breast_cancer, make_regression
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, SVR

# Classification
X, y = load_breast_cancer(return_X_y=True)
for kernel in ("linear", "rbf"):
    clf = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=1.0))
    print(f"SVC {kernel:>6}: CV accuracy {cross_val_score(clf, X, y, cv=5).mean():.3f}")

clf = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0)).fit(X, y)
print("support vectors per class:", clf[-1].n_support_)

# Regression
Xr, yr = make_regression(n_samples=400, n_features=5, noise=15, random_state=42)
for name, reg in [("linear regression", LinearRegression()),
                  ("SVR rbf", make_pipeline(StandardScaler(), SVR(kernel="rbf", C=100, epsilon=5))),
                  ("SVR linear", make_pipeline(StandardScaler(), SVR(kernel="linear", C=10, epsilon=5)))]:
    print(f"{name:>17}: CV R² {cross_val_score(reg, Xr, yr, cv=5, scoring='r2').mean():.3f}")

svr = make_pipeline(StandardScaler(), SVR(kernel="linear", C=10, epsilon=5)).fit(Xr, yr)
print("share of points that are support vectors:", round(len(svr[-1].support_) / len(Xr), 3))

Increasing epsilon widens the tube, so fewer points become support vectors and the model gets simpler. For this linear synthetic data, linear models do well; RBF kernels pay off when the relationship is non-linear.

Common Pitfalls

  • Unscaled features (SVMs are distance-based).

  • Kernel SVMs on hundreds of thousands of rows (training is roughly quadratic or worse); use LinearSVC or SGDClassifier.

  • Using probability=True without realising it adds internal cross-validation and may not match predict.

29

How are hyperparameters tuned in scikit-learn?

Hyperparameters are settings chosen before training (regularization strength, tree depth, number of neighbours), and scikit-learn tunes them by searching over candidates with cross-validation. GridSearchCV tries every combination in a grid; RandomizedSearchCV samples a fixed number of combinations from distributions (usually more efficient for many parameters); HalvingGridSearchCV and HalvingRandomSearchCV (experimental) use successive halving to discard weak candidates early. Some estimators have built-in CV versions (LogisticRegressionCV, RidgeCV, LassoCV). Tune a whole Pipeline with names like model__C, pick a suitable scoring, and evaluate the refitted best model on a held-out test set. For large searches, external libraries such as Optuna add Bayesian optimization.

Read full answer

Applies toscikit-learn 1.5+, SciPy 1.13+ (Python 3.12)

Search Methods

Method How When
GridSearchCV All combinations Few parameters, small grids
RandomizedSearchCV n_iter samples from distributions Many parameters, continuous ranges
HalvingGridSearchCV / HalvingRandomSearchCV Successive halving Large candidate sets (from sklearn.experimental import enable_halving_search_cv)
*CV estimators Efficient path-based search Linear models
Nested CV Outer loop around the search Unbiased performance estimate

Code Example

from scipy.stats import loguniform, randint
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

pipe = Pipeline([("scale", StandardScaler()), ("model", LogisticRegression(max_iter=5000))])
grid = GridSearchCV(pipe, {"model__C": [0.01, 0.1, 1, 10], "model__penalty": ["l2"]},
                    cv=5, scoring="roc_auc", n_jobs=-1).fit(X_train, y_train)
print("grid best:", grid.best_params_, round(grid.best_score_, 4))

random_search = RandomizedSearchCV(
    HistGradientBoostingClassifier(random_state=42),
    {"learning_rate": loguniform(0.01, 0.3), "max_leaf_nodes": randint(8, 64),
     "min_samples_leaf": randint(5, 50), "l2_regularization": loguniform(1e-4, 1.0)},
    n_iter=25, cv=5, scoring="roc_auc", random_state=42, n_jobs=-1).fit(X_train, y_train)
print("random best:", {k: round(v, 4) if isinstance(v, float) else v for k, v in random_search.best_params_.items()})
print("test ROC AUC (grid / random):", round(grid.score(X_test, y_test), 4), round(random_search.score(X_test, y_test), 4))

# Nested CV for an unbiased estimate of the whole tuning procedure
nested = cross_val_score(GridSearchCV(pipe, {"model__C": [0.01, 0.1, 1, 10]}, cv=3, scoring="roc_auc"),
                         X, y, cv=5, scoring="roc_auc")
print("nested CV ROC AUC:", nested.mean().round(4))

best_score_ is optimistic because it is the maximum over many candidates; the test set or nested CV gives an honest estimate.

Common Pitfalls

  • Tuning on the test set.

  • Linear grids for scale parameters like C or learning rate; use log-spaced values.

  • Ignoring cv_results_; several settings may perform equally well, and the simplest is often best.

30

How do you train and evaluate a logistic regression model with scikit-learn?

Training a logistic regression in scikit-learn takes a few steps: split the data with train_test_split(stratify=y), build a pipeline of StandardScaler and LogisticRegression (scaling helps the solver and makes regularization fair across features), tune the regularization strength C with cross-validation (LogisticRegressionCV or GridSearchCV), and evaluate on the test set with a confusion matrix, precision, recall, F1, ROC AUC and a calibration check. Coefficients, exponentiated, give odds ratios per one standard deviation of each feature.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Key Parameters

Parameter Meaning
C Inverse regularization strength (smaller = stronger)
penalty "l2" (default), "l1", "elasticnet" (solver-dependent)
solver "lbfgs" (default), "liblinear", "saga"
class_weight "balanced" for imbalanced classes
max_iter Raise it if you see convergence warnings

Code Example

import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.25, stratify=data.target, random_state=42)

model = make_pipeline(StandardScaler(),
                      LogisticRegressionCV(Cs=10, cv=5, scoring="roc_auc", max_iter=5000))
model.fit(X_train, y_train)
clf = model[-1]
print("chosen C:", round(float(clf.C_[0]), 4))

proba = model.predict_proba(X_test)[:, 1]
pred = model.predict(X_test)
print("confusion matrix:\n", confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=data.target_names, digits=3))
print("ROC AUC:", round(roc_auc_score(y_test, proba), 4))

frac_pos, mean_pred = calibration_curve(y_test, proba, n_bins=5)
print("calibration (predicted → observed):", list(zip(mean_pred.round(2), frac_pos.round(2))))

coef = clf.coef_[0]
top = np.argsort(-np.abs(coef))[:5]
for i in top:
    print(f"{data.feature_names[i]:>25}: odds ratio per SD = {np.exp(coef[i]):.3f}")

In this dataset class 1 is “benign”, so an odds ratio below 1 means larger values of that feature make a benign diagnosis less likely.

Common Pitfalls

  • Ignoring convergence warnings (increase max_iter, scale features).

  • Interpreting coefficients of unscaled or highly correlated features.

  • Using the default 0.5 threshold when error costs differ; tune it with TunedThresholdClassifierCV (1.5+).

31

How do you write a Python function for k-fold cross-validation with scikit-learn?

A k-fold cross-validation function splits the data into k folds, trains on k − 1 of them and validates on the remaining one, repeating k times so every sample is used for validation once, then summarizes the scores. Scikit-learn’s cross_validate does this in one call, supports several metrics at once and can return training scores; choose the splitter to match the data: StratifiedKFold for classification, KFold for regression, GroupKFold when rows belong to groups (patients, users), TimeSeriesSplit for temporal data. Passing a Pipeline keeps preprocessing inside each fold.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Code Example

import numpy as np
from sklearn.base import is_classifier
from sklearn.datasets import load_diabetes, load_wine
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


def k_fold_cv(model, X, y, k=5, scoring=None, random_state=42):
    """Run k-fold CV and return mean and std for each metric."""
    if is_classifier(model):
        splitter = StratifiedKFold(n_splits=k, shuffle=True, random_state=random_state)
        scoring = scoring or ["accuracy", "f1_macro"]
    else:
        splitter = KFold(n_splits=k, shuffle=True, random_state=random_state)
        scoring = scoring or ["r2", "neg_mean_absolute_error"]
    results = cross_validate(model, X, y, cv=splitter, scoring=scoring, return_train_score=True)
    summary = {}
    for metric in scoring:
        test, train = results[f"test_{metric}"], results[f"train_{metric}"]
        summary[metric] = {"test_mean": test.mean(), "test_std": test.std(), "train_mean": train.mean()}
    return summary


X_cls, y_cls = load_wine(return_X_y=True)
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
for metric, s in k_fold_cv(clf, X_cls, y_cls).items():
    print(f"classification {metric:>10}: {s['test_mean']:.3f} ± {s['test_std']:.3f} (train {s['train_mean']:.3f})")

X_reg, y_reg = load_diabetes(return_X_y=True)
reg = RandomForestRegressor(n_estimators=200, random_state=42)
for metric, s in k_fold_cv(reg, X_reg, y_reg, k=10).items():
    print(f"regression {metric:>24}: {s['test_mean']:.3f} ± {s['test_std']:.3f} (train {s['train_mean']:.3f})")

# Manual version showing what happens inside
scores = []
for train_idx, val_idx in StratifiedKFold(5, shuffle=True, random_state=42).split(X_cls, y_cls):
    fitted = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_cls[train_idx], y_cls[train_idx])
    scores.append(fitted.score(X_cls[val_idx], y_cls[val_idx]))
print("manual 5-fold accuracy:", np.round(scores, 3), "mean", round(float(np.mean(scores)), 3))

Error metrics such as MAE appear as negative numbers (neg_mean_absolute_error) because scikit-learn always treats higher scores as better.

Common Pitfalls

  • Preprocessing outside the pipeline, which leaks validation data into training.

  • Shuffled k-fold on time series or grouped data.

  • Reporting only the mean; the standard deviation shows how stable the estimate is.

32

How do you extract text features with CountVectorizer or TfidfVectorizer?

CountVectorizer turns a collection of documents into a sparse matrix of token counts (a bag of words): it tokenizes, builds a vocabulary from the training texts, and counts each term per document. TfidfVectorizer does the same and then reweights counts by inverse document frequency, so words common to every document (like “the”) get small weights and distinctive words get large ones; rows are L2-normalized by default. Key options are ngram_range (phrases), min_df/max_df (drop rare or ubiquitous terms), stop_words, max_features, sublinear_tf, and analyzer="char_wb" for character n-grams. Fit the vectorizer on training text only, inside a Pipeline.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Important Parameters

Parameter Effect
ngram_range=(1, 2) Words and two-word phrases
min_df=2 / max_df=0.9 Drop terms in fewer than 2 docs / more than 90% of docs
stop_words="english" Remove common English words (careful with “not”)
max_features Keep the most frequent terms
sublinear_tf=True Use 1 + log(tf) to dampen repeated words
analyzer="char_wb" Character n-grams, robust to typos
binary=True Presence instead of counts

Code Example

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

docs = ["the cat sat on the mat", "the dog sat on the log", "cats and dogs are pets"]

counts = CountVectorizer()
X_counts = counts.fit_transform(docs)
print("vocabulary:", counts.get_feature_names_out())
print("count matrix:\n", X_counts.toarray())

tfidf = TfidfVectorizer()
X_tfidf = tfidf.fit_transform(docs)
row0 = dict(zip(tfidf.get_feature_names_out(), X_tfidf[0].toarray()[0].round(3)))
print("TF-IDF weights, doc 0:", {k: v for k, v in row0.items() if v > 0})

# In a model: TF-IDF with bigrams inside a pipeline
texts = ["great product, works well", "terrible, broke after a day", "love it", "not good at all",
         "excellent value", "waste of money", "very happy with it", "do not buy"] * 25
labels = [1, 0, 1, 0, 1, 0, 1, 0] * 25
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True), LogisticRegression(max_iter=1000))
print("CV accuracy:", cross_val_score(model, texts, labels, cv=5).mean().round(3))

In document 0, “the” appears twice but also in document 1, so its TF-IDF weight is lower relative to its count, while “cat” and “mat”, which appear only in document 0, get higher weights. The repeated toy texts make the CV score trivially high; real data needs a proper evaluation.

Common Pitfalls

  • Fitting the vectorizer on all text before splitting.

  • Removing negations with stop-word lists in sentiment tasks.

  • Calling .toarray() on large sparse matrices (memory explosion).

33

How do you scale data and train a Naive Bayes classifier with scikit-learn?

For Gaussian Naive Bayes, preprocessing is part of a simple pipeline: scale or transform the features, then fit GaussianNB. Note that GaussianNB estimates a separate mean and variance per feature and class, so linear scaling (StandardScaler, MinMaxScaler) does not change its predictions; what can help is a transformation that makes features closer to normal, such as PowerTransformer or QuantileTransformer(output_distribution="normal"). For count or TF-IDF data, use MultinomialNB (which needs non-negative inputs, so MinMaxScaler rather than StandardScaler), and BernoulliNB for binary features. Evaluate with a held-out test set or cross-validation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Choosing the Variant and Preprocessing

Variant Data Preprocessing
GaussianNB Continuous Optional PowerTransformer toward normality
MultinomialNB Counts, TF-IDF Non-negative input (MinMaxScaler if needed)
ComplementNB Imbalanced text As multinomial
BernoulliNB Binary features binarize threshold
CategoricalNB Categories OrdinalEncoder

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, PowerTransformer, StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

candidates = {
    "GaussianNB (raw)": GaussianNB(),
    "StandardScaler + GaussianNB": make_pipeline(StandardScaler(), GaussianNB()),
    "PowerTransformer + GaussianNB": make_pipeline(PowerTransformer(), GaussianNB()),
    "MinMaxScaler + MultinomialNB": make_pipeline(MinMaxScaler(), MultinomialNB()),
}
for name, model in candidates.items():
    cv = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
    print(f"{name:>30}: CV accuracy {cv.mean():.3f} ± {cv.std():.3f}")

best = make_pipeline(PowerTransformer(), GaussianNB()).fit(X_train, y_train)
print(classification_report(y_test, best.predict(X_test), digits=3))

The raw and standardized GaussianNB scores should be identical, confirming that linear scaling has no effect on it; the power transform can change results because it reshapes skewed features.

Common Pitfalls

  • Passing standardized (negative) values to MultinomialNB, which raises an error.

  • Trusting Naive Bayes probabilities as calibrated; they are often extreme. Use CalibratedClassifierCV if probabilities matter.

  • Highly correlated features being “double counted” because of the independence assumption.

34

How do you combine preprocessing and model training with a Pipeline?

A Pipeline chains preprocessing steps and a final estimator into one object: calling fit runs fit_transform on each transformer in turn and then fits the model; predict passes new data through the same fitted transformers. Combined with ColumnTransformer for different column types, this guarantees that cross-validation refits preprocessing inside each fold (no leakage), that grid search can tune preprocessing and model parameters together (named step__param), and that the deployed artifact contains every step. make_pipeline names steps automatically.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 1500
df = pd.DataFrame({
    "age": rng.integers(18, 80, n).astype(float),
    "income": rng.lognormal(10.5, 0.5, n),
    "plan": rng.choice(["basic", "plus", "pro"], n),
    "region": rng.choice(["north", "south", "east", "west"], n),
})
df.loc[rng.random(n) < 0.1, "income"] = np.nan
logit = -1 + 0.03 * (df["age"] - 45) + 0.8 * (df["plan"] == "pro") - 0.5 * (df["region"] == "south")
y = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)

numeric = ["age", "income"]
categorical = ["plan", "region"]
preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
pipe = Pipeline([("prep", preprocess), ("model", LogisticRegression(max_iter=1000))])

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, stratify=y, random_state=42)
print("CV ROC AUC:", cross_val_score(pipe, X_train, y_train, cv=5, scoring="roc_auc").mean().round(3))

search = GridSearchCV(pipe, {
    "prep__num__impute__strategy": ["mean", "median"],
    "model": [LogisticRegression(max_iter=1000), HistGradientBoostingClassifier(random_state=42)],
}, cv=5, scoring="roc_auc")
search.fit(X_train, y_train)
print("best:", search.best_params_)
print("test ROC AUC:", round(search.score(X_test, y_test), 3))

Grid search can even swap the final model by treating the step itself as a parameter ("model": [...]). The fitted search.best_estimator_ is a complete pipeline ready to save.

Common Pitfalls

  • Scaling or imputing the whole dataset before splitting.

  • Forgetting handle_unknown="ignore", so new categories crash predictions.

  • Long pipelines refitting expensive steps repeatedly; use Pipeline(memory=...) to cache.

35

How do you grid-search a RandomForestClassifier's hyperparameters?

A grid search over a random forest tries every combination of chosen hyperparameter values with cross-validation and keeps the best one. The parameters that matter most are n_estimators (more trees are rarely worse, just slower), max_depth and min_samples_leaf (control tree complexity), max_features (how many features each split considers, controlling tree diversity), and class_weight for imbalanced data. Wrap the forest in a Pipeline if there is preprocessing, use StratifiedKFold, choose a scoring metric that fits the problem, set n_jobs=-1, and evaluate the refitted best model once on a held-out test set.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Code Example

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split


def tune_random_forest(X, y, scoring="roc_auc", cv_splits=5, random_state=42):
    """Grid-search a random forest and return the fitted search object."""
    param_grid = {
        "n_estimators": [200, 500],
        "max_depth": [None, 6, 12],
        "min_samples_leaf": [1, 3, 5],
        "max_features": ["sqrt", 0.5],
        "class_weight": [None, "balanced"],
    }
    search = GridSearchCV(
        RandomForestClassifier(random_state=random_state, n_jobs=1),
        param_grid,
        scoring=scoring,
        cv=StratifiedKFold(n_splits=cv_splits, shuffle=True, random_state=random_state),
        n_jobs=-1,
        refit=True,
        return_train_score=True,
    )
    return search.fit(X, y)


X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

search = tune_random_forest(X_train, y_train)
print("best parameters:", search.best_params_)
print("best CV ROC AUC:", round(search.best_score_, 4))

results = pd.DataFrame(search.cv_results_).sort_values("rank_test_score")
print(results[["params", "mean_train_score", "mean_test_score", "std_test_score"]].head(5).to_string(index=False))

print(classification_report(y_test, search.best_estimator_.predict(X_test), digits=3))

This grid has 2 × 3 × 3 × 2 × 2 = 72 combinations, each fitted 5 times. For larger spaces, RandomizedSearchCV or HalvingGridSearchCV finds good settings with far fewer fits. Setting the forest’s own n_jobs=1 avoids oversubscribing CPUs while the search runs in parallel.

Common Pitfalls

  • Tuning on the test set or reporting the CV score of the winner as unbiased (it is optimistic).

  • Huge grids that waste compute on parameters that barely matter.

  • Comparing mean_train_score and mean_test_score only for the best row; check the spread too.

36

How do you visualize a non-linear SVM decision boundary with scikit-learn?

To visualize an SVM decision boundary, train an SVC with a non-linear kernel (usually RBF) on two features, then evaluate the model on a fine grid covering the feature space and colour the regions by predicted class. Scikit-learn’s DecisionBoundaryDisplay.from_estimator does the grid work for you; plotting the decision_function contours at −1, 0 and +1 shows the boundary and the margins, and support_vectors_ marks the points that define them. Comparing several values of C and gamma shows how the boundary goes from smooth (underfitting) to wiggly (overfitting).

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.9+ (Python 3.12)

Code Example

import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_circles(n_samples=400, factor=0.4, noise=0.12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

settings = [(1.0, 0.5), (1.0, 5.0), (100.0, 50.0)]           # (C, gamma)
fig, axes = plt.subplots(1, 3, figsize=(15, 4.5))
for ax, (C, gamma) in zip(axes, settings):
    model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C, gamma=gamma)).fit(X_train, y_train)
    DecisionBoundaryDisplay.from_estimator(model, X, response_method="predict", alpha=0.3, ax=ax, cmap="coolwarm")
    DecisionBoundaryDisplay.from_estimator(model, X, response_method="decision_function", plot_method="contour",
                                           levels=[-1, 0, 1], linestyles=["--", "-", "--"], colors="k", ax=ax)
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap="coolwarm", edgecolor="k", s=20)
    svc = model[-1]
    sv = model[0].inverse_transform(svc.support_vectors_)       # back to original feature scale
    ax.scatter(sv[:, 0], sv[:, 1], s=80, facecolors="none", edgecolors="k", linewidths=1)
    ax.set_title(f"C={C}, gamma={gamma}\ntrain {model.score(X_train, y_train):.2f}, "
                 f"test {model.score(X_test, y_test):.2f}, SVs {len(svc.support_)}")
fig.tight_layout()
plt.show()

The solid line is the decision boundary (decision function = 0), the dashed lines are the margins, and circled points are support vectors. Very large C and gamma produce a convoluted boundary with a gap between train and test accuracy, a visual sign of overfitting.

Common Pitfalls

  • Visualizing with more than two features (use two chosen features or a 2D projection, and remember the boundary in the projection is approximate).

  • Forgetting to scale features before an RBF SVM.

  • Reading the plotted boundary as the full story when tuning; use cross-validation for C and gamma.

37

How do you apply PCA with scikit-learn and visualize the result?

PCA in scikit-learn is a transformer: standardize the features, then PCA(n_components=2).fit_transform(X) projects the data onto the two directions of greatest variance for a scatter plot. Check explained_variance_ratio_ to see how much information the 2D view keeps, plot the cumulative ratio to choose a number of components for modelling (or pass n_components=0.95), and read components_ (loadings) to see which original features drive each component. Put scaling and PCA in a Pipeline so the same transformation applies to new data.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.9+ (Python 3.12)

Code Example

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_wine()
X, y = data.data, data.target

pca_pipe = make_pipeline(StandardScaler(), PCA(n_components=2, random_state=42))
Z = pca_pipe.fit_transform(X)
pca = pca_pipe[-1]
print("explained variance ratio:", pca.explained_variance_ratio_.round(3),
      "| total:", round(pca.explained_variance_ratio_.sum(), 3))

loadings = pca.components_[0]
top = np.argsort(-np.abs(loadings))[:3]
print("features driving PC1:", [data.feature_names[i] for i in top])

full = PCA().fit(StandardScaler().fit_transform(X))
cumulative = np.cumsum(full.explained_variance_ratio_)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4))
for cls in np.unique(y):
    ax1.scatter(Z[y == cls, 0], Z[y == cls, 1], label=data.target_names[cls], alpha=0.7)
ax1.set_xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.0%})")
ax1.set_ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.0%})")
ax1.legend()
ax1.set_title("Wine data in the first two components")

ax2.plot(range(1, len(cumulative) + 1), cumulative, marker="o")
ax2.axhline(0.95, linestyle="--", color="grey")
ax2.set_xlabel("number of components")
ax2.set_ylabel("cumulative explained variance")
ax2.set_title("Choosing the number of components")
fig.tight_layout()
plt.show()

The classes separate well in two components even though PCA never saw the labels; the cumulative curve shows how many components reach 95% of the variance.

Common Pitfalls

  • Skipping standardization when features have different units.

  • Treating 2D plots as complete pictures when the first two components explain little variance.

  • Fitting PCA on train and test together before cross-validation.

38

How do you run a DBSCAN clustering analysis with scikit-learn?

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that lie in dense regions and labels isolated points as noise (label −1). It has two key parameters: eps, the neighbourhood radius, and min_samples, the number of points needed within eps to form a dense core. It finds clusters of arbitrary shape, does not need the number of clusters in advance, and handles outliers, but struggles when clusters have very different densities. Scale the features first, choose eps from a k-distance plot (the “elbow” of sorted distances to the k-th nearest neighbour), and evaluate with silhouette scores on non-noise points or adjusted Rand index when labels exist. HDBSCAN (1.3+) removes the need to pick eps.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.9+ (Python 3.12)

Code Example

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import DBSCAN, HDBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score
from sklearn.neighbors import NearestNeighbors
from sklearn.preprocessing import StandardScaler

X, y_true = make_moons(n_samples=600, noise=0.07, random_state=42)
rng = np.random.default_rng(42)
X = np.vstack([X, rng.uniform(-1.5, 2.5, size=(30, 2))])          # add scattered outliers
y_true = np.r_[y_true, np.full(30, -1)]
X_scaled = StandardScaler().fit_transform(X)

# k-distance plot to choose eps (k = min_samples)
min_samples = 8
distances, _ = NearestNeighbors(n_neighbors=min_samples).fit(X_scaled).kneighbors(X_scaled)
k_dist = np.sort(distances[:, -1])

db = DBSCAN(eps=0.25, min_samples=min_samples).fit(X_scaled)
labels = db.labels_
print("clusters found:", len(set(labels)) - (1 if -1 in labels else 0), "| noise points:", int((labels == -1).sum()))
print("ARI DBSCAN:", round(adjusted_rand_score(y_true, labels), 3))
print("ARI k-means (k=2):", round(adjusted_rand_score(y_true, KMeans(2, n_init=10, random_state=42).fit_predict(X_scaled)), 3))
print("ARI HDBSCAN:", round(adjusted_rand_score(y_true, HDBSCAN(min_cluster_size=15).fit_predict(X_scaled)), 3))

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4))
ax1.plot(k_dist)
ax1.set_xlabel("points sorted by distance")
ax1.set_ylabel(f"distance to {min_samples}th neighbour")
ax1.set_title("k-distance plot (pick eps at the elbow)")
ax2.scatter(X_scaled[:, 0], X_scaled[:, 1], c=labels, cmap="viridis", s=12)
ax2.set_title("DBSCAN clusters (noise = −1)")
plt.show()

DBSCAN follows the curved moon shapes and marks the scattered points as noise, while k-means splits the moons with a straight boundary. If the ARI is low, adjust eps using the k-distance plot.

Common Pitfalls

  • Using DBSCAN on unscaled features, which makes eps meaningless.

  • A single eps for clusters of very different densities (try HDBSCAN or OPTICS).

  • Forgetting that DBSCAN has no predict for new points; use a classifier trained on the cluster labels or HDBSCAN’s approximate methods.

39

How do you save and load a trained scikit-learn model?

The standard way to persist a scikit-learn model is joblib.dump(model, "model.joblib") and joblib.load("model.joblib"), which pickles the fitted object efficiently, including large NumPy arrays. Save the whole pipeline (preprocessing plus model) so new data is transformed exactly as during training. Two warnings matter: pickled models must be loaded with the same scikit-learn (and Python/NumPy) versions, and loading pickles can execute arbitrary code, so only load files from trusted sources. For safer sharing, skops (skops.io.dump/load) lets you inspect and allow types before loading; for cross-language serving, export to ONNX with skl2onnx.

Read full answer

Applies toscikit-learn 1.5+, joblib 1.4+, skops 0.10+ (optional) (Python 3.12)

Options

Format Pros Cons
joblib Standard, fast for arrays, compression Unsafe with untrusted files; version-bound
pickle Built in Same risks; slower for big arrays
skops Safer loading with a list of trusted types Extra dependency
ONNX (skl2onnx) Language-independent, fast runtimes Not every estimator converts

Code Example

import json
import platform
import tempfile
from pathlib import Path

import joblib
import numpy as np
import sklearn
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

out_dir = Path(tempfile.mkdtemp())
joblib.dump(pipe, out_dir / "model.joblib", compress=3)
metadata = {
    "sklearn_version": sklearn.__version__,
    "python_version": platform.python_version(),
    "test_accuracy": round(float(pipe.score(X_test, y_test)), 4),
}
(out_dir / "model_meta.json").write_text(json.dumps(metadata, indent=2))

# Later, in the serving environment (same library versions, trusted file)
loaded = joblib.load(out_dir / "model.joblib")
meta = json.loads((out_dir / "model_meta.json").read_text())
if meta["sklearn_version"] != sklearn.__version__:
    raise RuntimeError("model saved with a different scikit-learn version")
print("identical predictions:", np.array_equal(loaded.predict(X_test), pipe.predict(X_test)))
print(meta)

Storing metadata (versions, metrics, training date) next to the model file makes version mismatches easy to catch.

Safer Loading with skops

import tempfile
from pathlib import Path

import skops.io as sio
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X, y)
out_dir = Path(tempfile.mkdtemp())

sio.dump(pipe, out_dir / "model.skops")
unknown = sio.get_untrusted_types(file=out_dir / "model.skops")
print("types to review before loading:", unknown)
loaded_safe = sio.load(out_dir / "model.skops", trusted=unknown)   # only after reviewing them

Common Pitfalls

  • Saving only the estimator and re-implementing preprocessing elsewhere.

  • Loading a model saved with another scikit-learn version (warnings, wrong results or errors).

  • Accepting model files from users or the internet.

40

How do you deploy a scikit-learn model into production?

Deploying a scikit-learn model means packaging the fitted pipeline with a pinned environment and exposing it where predictions are needed. The usual steps are: train and validate the full pipeline; save it with joblib (plus metadata: versions, metrics, feature schema); build a serving layer, either a REST API (FastAPI or Flask) in a Docker container for real-time requests, or a batch job that scores tables on a schedule; validate inputs against the training schema; log predictions; and monitor drift and performance with a path to retraining and rollback. Managed platforms (SageMaker, Vertex AI, Azure ML) or ONNX runtimes are alternatives for scale and latency.

Read full answer

Applies toscikit-learn 1.5+, FastAPI 0.115+, pydantic 2.x (Python 3.12)

Deployment Checklist

Step Detail
Package Whole Pipeline with joblib; requirements.txt with exact versions
Interface Input schema (column names, types, ranges); output with model version
Serving FastAPI + Uvicorn in Docker; or batch scoring job
Validation Reject or flag malformed input; handle unseen categories (handle_unknown="ignore")
Testing Unit tests for the API; check saved vs reloaded predictions
Rollout Shadow or canary deployment; keep the previous model for rollback
Monitoring Latency, errors, input drift, prediction drift, labelled performance

Code Example: FastAPI Service

# serve.py — run with: uvicorn serve:app --host 0.0.0.0 --port 8000
import joblib
import pandas as pd
from fastapi import FastAPI
from pydantic import BaseModel, Field

MODEL_PATH = "model.joblib"            # a pipeline saved from training (trusted file)
MODEL_VERSION = "churn-v3"

app = FastAPI()
pipeline = joblib.load(MODEL_PATH)


class Customer(BaseModel):
    tenure_months: int = Field(ge=0)
    monthly_charges: float = Field(ge=0)
    contract: str


@app.get("/health")
def health():
    return {"status": "ok", "model_version": MODEL_VERSION}


@app.post("/predict")
def predict(customers: list[Customer]):
    frame = pd.DataFrame([c.model_dump() for c in customers])
    probabilities = pipeline.predict_proba(frame)[:, 1]
    return {"model_version": MODEL_VERSION,
            "churn_probability": [round(float(p), 4) for p in probabilities]}

Loading the model once at start-up avoids reloading it per request, and pydantic validates the input types and ranges before they reach the pipeline. The pipeline itself (a ColumnTransformer with OneHotEncoder(handle_unknown="ignore")) handles categorical encoding.

Common Pitfalls

  • Different library versions in training and serving.

  • Preprocessing rewritten in the application instead of saved in the pipeline.

  • No monitoring, so silent drift goes unnoticed.

41

How do you update a scikit-learn model with new data over time?

There are two ways to update a scikit-learn model with new data. The common one is periodic retraining: refit the whole pipeline on a recent window (or all data) on a schedule or when monitoring shows drift, validate it on the latest period against the current model, and promote it only if it is better. The second is incremental learning with estimators that implement partial_fit (SGDClassifier, SGDRegressor, MultinomialNB, MiniBatchKMeans, PassiveAggressiveClassifier, IncrementalPCA), which update on mini-batches without revisiting old data. Most estimators, including random forests and gradient boosting, have no partial_fit; warm_start=True can add trees or iterations but still trains on whatever data you pass.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Options

Approach Estimators Notes
Full retrain (sliding or expanding window) Any Simplest, most reliable
partial_fit SGD*, NB, MiniBatchKMeans, IncrementalPCA, Perceptron Pass classes= on the first call
warm_start=True Forests (add trees), GB, some linear models Not true online learning
Champion/challenger Any Compare on the latest labelled window before promotion

Code Example: Incremental Learning vs Periodic Retraining Under Drift

import numpy as np
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)

def batch(t, n=500):
    X = rng.normal(size=(n, 5))
    w = np.array([1.0, -1.0, 0.5, 0.0, 0.0]) + np.array([0.0, 0.0, 0.0, 1.0, -1.0]) * t / 10   # drifting weights
    y = (X @ w + rng.normal(0, 0.5, n) > 0).astype(int)
    return X, y

scaler = StandardScaler()
online = SGDClassifier(loss="log_loss", random_state=42)
static = SGDClassifier(loss="log_loss", random_state=42)

X0, y0 = batch(0)
scaler.partial_fit(X0)
online.partial_fit(scaler.transform(X0), y0, classes=np.array([0, 1]))
static.fit(scaler.transform(X0), y0)

for t in range(1, 11):
    X_new, y_new = batch(t)
    Xs = scaler.transform(X_new)
    if t % 5 == 0:
        print(f"t={t:>2}: static acc {accuracy_score(y_new, static.predict(Xs)):.3f}, "
              f"online acc {accuracy_score(y_new, online.predict(Xs)):.3f}")
    online.partial_fit(Xs, y_new)                     # update after evaluating on the new batch

The model updated with partial_fit keeps up with the drifting relationship, while the static model, trained only on the first batch, degrades. In practice, keep the scaler fixed or update it carefully, because changing it also changes the meaning of the learned weights.

Common Pitfalls

  • Updating on new data without evaluating first (each batch should be scored before it is learned from).

  • Forgetting classes= on the first partial_fit call.

  • Feedback loops: the model’s own decisions shaping the next training data.

42

How do you monitor a scikit-learn model's performance in production?

Monitoring a scikit-learn model in production tracks three things over time: data quality and drift in the inputs (missing values, out-of-range values, unseen categories, shifts in feature distributions measured with PSI or KS tests against the training data), prediction drift (changes in score distribution and positive rate), and actual performance once labels arrive (the same metrics used at validation, computed on recent windows and compared with the deployment baseline). Log every prediction with its model version and inputs, set alert thresholds, and use the results to trigger investigation or retraining. Scikit-learn’s metrics functions compute the numbers; tools such as Evidently, WhyLabs or custom dashboards display them.

Read full answer

Applies toscikit-learn 1.5+, SciPy 1.13+, pandas 2.2+ (Python 3.12)

What to Monitor

Layer Metric Alert example
Inputs Null rate, range checks, unseen categories Null rate > 5%
Feature drift PSI, KS statistic per feature PSI > 0.2
Predictions Mean score, positive rate, score histogram Positive rate doubles
Performance ROC AUC, precision/recall, MAE on labelled data AUC drops 0.03 below baseline
Calibration Calibration curve, Brier score Predicted vs observed rate gap
Operations Latency, error rate, throughput p95 latency > 200 ms

Code Example: Drift and Performance Checks

import numpy as np
from scipy import stats
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

def psi(expected, actual, bins=10):
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    e = np.histogram(np.clip(expected, edges[0], edges[-1]), edges)[0] / len(expected)
    a = np.histogram(np.clip(actual, edges[0], edges[-1]), edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

X, y = make_classification(n_samples=6000, n_features=8, random_state=42)
X_train, y_train, X_prod, y_prod = X[:4000], y[:4000], X[4000:].copy(), y[4000:]
X_prod[:, 0] += 1.0                                     # simulate drift in feature 0

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
baseline_auc = roc_auc_score(y_train, model.predict_proba(X_train)[:, 1])

for j in range(3):
    ks = stats.ks_2samp(X_train[:, j], X_prod[:, j])
    print(f"feature {j}: PSI {psi(X_train[:, j], X_prod[:, j]):.3f}, KS p-value {ks.pvalue:.2g}")

p_prod = model.predict_proba(X_prod)[:, 1]
print(f"positive rate: train {model.predict(X_train).mean():.3f}, production {(p_prod >= 0.5).mean():.3f}")
print(f"AUC: baseline {baseline_auc:.3f}, production {roc_auc_score(y_prod, p_prod):.3f}")

Feature 0’s PSI should be clearly above the others, flagging the shift; the performance check shows whether the drift actually hurts the model.

Common Pitfalls

  • Alerting on every statistically significant drift; with large volumes tiny shifts are significant. Use effect sizes like PSI.

  • Waiting for labels before monitoring anything; input and prediction drift give earlier warnings.

  • Comparing production metrics with training-set metrics instead of a held-out baseline.

43

What are scikit-learn's limitations with very large datasets?

Scikit-learn is designed for data that fits in memory on one machine. Most estimators load the full dataset as an in-memory NumPy array or SciPy sparse matrix, so datasets larger than RAM need sampling or other tools. Training runs on CPU only (GPU support is limited to experimental array API dispatch for a few estimators), parallelism with n_jobs stays on a single machine, and some algorithms scale poorly: kernel SVMs are roughly quadratic to cubic in the number of samples, Gaussian processes cubic, and exact nearest neighbours and hierarchical clustering become very slow. Workarounds include partial_fit estimators, mini-batch variants, histogram gradient boosting, sparse matrices, approximate kernels and distributed libraries.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Limitations and Workarounds

Limitation Workaround
Data must fit in memory Sample; partial_fit on chunks; Dask-ML, Spark MLlib
CPU only cuML (RAPIDS), XGBoost/LightGBM GPU
Single machine Dask/Ray backends for joblib; distributed frameworks
Kernel SVM scaling LinearSVC, SGDClassifier, Nystroem + linear model
Slow gradient boosting on big data HistGradientBoostingClassifier (binned, multithreaded)
Huge categorical/text spaces HashingVectorizer, FeatureHasher, sparse matrices
float64 memory use Cast to float32 where estimators support it
Exact KNN / DBSCAN on millions of points Approximate nearest neighbours (FAISS, Annoy), HDBSCAN on samples

Code Example: Out-of-Core Learning on Chunks

import numpy as np
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.linear_model import SGDClassifier

rng = np.random.default_rng(42)
vectorizer = HashingVectorizer(n_features=2 ** 18, alternate_sign=False)   # stateless: no fit needed
model = SGDClassifier(loss="log_loss", random_state=42)

good = ["fast delivery", "great quality", "works perfectly", "very happy"]
bad = ["arrived broken", "poor quality", "stopped working", "very disappointed"]

def stream_chunks(n_chunks=20, size=1000):
    for _ in range(n_chunks):
        labels = rng.integers(0, 2, size)
        texts = [rng.choice(good if lab else bad) + " " + rng.choice(["item", "product", "order"]) for lab in labels]
        yield texts, labels

for i, (texts, labels) in enumerate(stream_chunks()):
    X = vectorizer.transform(texts)                   # sparse matrix, constant memory
    if i == 0:
        model.partial_fit(X, labels, classes=np.array([0, 1]))
        continue
    if i % 5 == 0:
        print(f"chunk {i:>2} accuracy before update: {model.score(X, labels):.3f}")
    model.partial_fit(X, labels)

HashingVectorizer needs no vocabulary, so it works on streams that never fit in memory at once, and SGDClassifier.partial_fit learns chunk by chunk.

Common Pitfalls

  • Densifying sparse matrices (.toarray()) on large text data.

  • Running GridSearchCV with n_jobs=-1 on large data, which copies data into each worker; use pre_dispatch or fewer jobs.

  • Choosing algorithms with poor scaling (kernel SVM, GP) for millions of rows.

44

How do you implement custom transformers in scikit-learn?

A custom transformer is a class that inherits from BaseEstimator and TransformerMixin and implements fit(X, y=None) (learn any parameters from training data, store them in attributes ending with an underscore, and return self) and transform(X) (apply the learned transformation). TransformerMixin adds fit_transform, and BaseEstimator adds get_params/set_params as long as __init__ only stores its arguments unchanged. Implementing get_feature_names_out makes it work with set_output(transform="pandas"). For stateless functions, FunctionTransformer is simpler. Custom transformers then plug into Pipeline, ColumnTransformer and grid search.

Read full answer

Applies toscikit-learn 1.6+, pandas 2.2+ (Python 3.12)

Rules

Rule Why
__init__ stores arguments only, no validation or logic get_params, clone and grid search rely on it
Learn from data only in fit Prevents leakage across CV folds
Fitted attributes end with _ Convention; check_is_fitted uses it
fit returns self Enables chaining and pipelines
Set n_features_in_ / feature_names_in_ Consistency checks
Implement get_feature_names_out Feature names and pandas output

Code Example: Winsorizing Transformer

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.utils.validation import check_is_fitted, validate_data

class Winsorizer(TransformerMixin, BaseEstimator):
    """Clip each feature to percentiles learned on the training data."""

    def __init__(self, lower=1.0, upper=99.0):
        self.lower = lower
        self.upper = upper

    def fit(self, X, y=None):
        X = validate_data(self, X)                    # sets n_features_in_ (and feature_names_in_)
        self.low_, self.high_ = np.percentile(X, [self.lower, self.upper], axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, reset=False)
        return np.clip(X, self.low_, self.high_)

    def get_feature_names_out(self, input_features=None):
        if input_features is not None:
            return np.asarray(input_features, dtype=object)
        if hasattr(self, "feature_names_in_"):
            return self.feature_names_in_
        return np.array([f"x{i}" for i in range(self.n_features_in_)], dtype=object)

X, y = load_diabetes(return_X_y=True, as_frame=True)
pipe = make_pipeline(Winsorizer(), StandardScaler(), Ridge())
search = GridSearchCV(pipe, {"winsorizer__upper": [95.0, 99.0, 100.0], "ridge__alpha": [0.1, 1.0, 10.0]},
                      cv=5, scoring="r2")
search.fit(X, y)
print("best params:", search.best_params_, "| CV R²:", round(search.best_score_, 3))

pipe.set_output(transform="pandas")
print(pipe[:-1].fit_transform(X, y).columns[:3].tolist())

validate_data is the public helper in scikit-learn 1.6+; in 1.5 use self._validate_data(X) instead. The transformer’s parameters appear in the grid search as winsorizer__upper because make_pipeline names steps after their lowercase class name.

Common Pitfalls

  • Computing statistics in transform or __init__, which leaks test data or breaks cloning.

  • Modifying the input array in place.

  • Forgetting return self in fit.

45

Which recent ML advances are not fully supported in scikit-learn?

Scikit-learn deliberately focuses on well-established, general-purpose algorithms for tabular data with a stable API, so several recent areas are outside its scope or only partly covered. These include deep learning at scale (transformers, CNNs, large language models; its MLPClassifier is basic and CPU-only), GPU acceleration (only limited, experimental array API support), foundation models and embeddings, reinforcement learning, probabilistic programming and Bayesian deep learning, graph neural networks, advanced time series forecasting, large-scale recommenders, distributed training on clusters, and newer gradient boosting features found in XGBoost, LightGBM and CatBoost. Its inclusion criteria require algorithms to be mature and widely cited, which keeps the library reliable but means new methods arrive slowly or never.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Gaps and Common Alternatives

Area Status in scikit-learn Common alternatives
Deep learning, transformers Basic MLP only PyTorch, TensorFlow/Keras, Hugging Face
GPU training Experimental array API for some estimators cuML (RAPIDS), XGBoost/LightGBM GPU
Distributed / out-of-core partial_fit for some models Dask-ML, Spark MLlib, Ray
Gradient boosting extras HistGradientBoosting (fast, native categoricals) XGBoost, LightGBM, CatBoost
Time series forecasting Generic regressors, TimeSeriesSplit statsmodels, sktime, Darts, Prophet
Probabilistic models BayesianRidge, GaussianProcess* PyMC, NumPyro, Stan
Graph learning Not covered PyTorch Geometric, DGL
Reinforcement learning Not covered Gymnasium, Stable-Baselines3
AutoML / advanced tuning Grid and randomized search, successive halving Optuna, auto-sklearn, FLAML

What Scikit-Learn Has Added Recently

HDBSCAN and TargetEncoder (1.3), metadata routing, FixedThresholdClassifier and TunedThresholdClassifierCV (1.5), and broader set_output(transform="pandas") support.

Code Example: Checking the Installed Version and Experimental Array API

import sklearn
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

print("scikit-learn version:", sklearn.__version__)

# Array API dispatch is experimental and supported by only some estimators (e.g. LDA, PCA);
# with PyTorch or CuPy inputs and array-api-compat installed, work can run on the GPU.
sklearn.set_config(array_api_dispatch=False)       # default; set True only with a supported setup
lda = LinearDiscriminantAnalysis()
print("LDA estimator created:", lda)

Interview Tips

  • Frame scikit-learn as the standard for classical ML on tabular data and as the “glue” (preprocessing, CV, metrics) around other libraries.

  • Avoid claiming it can train large neural networks or run natively distributed.

46

How does scikit-learn integrate with TensorFlow and PyTorch?

Scikit-learn integrates with deep learning libraries mainly around the model, not inside it. Its preprocessing (ColumnTransformer, scalers, encoders), data splitting, cross-validation and metrics work with any NumPy arrays, so they are commonly used before and after training TensorFlow/Keras or PyTorch models. Wrappers make deep learning models behave like scikit-learn estimators: skorch for PyTorch and SciKeras for Keras, so they fit into Pipeline and GridSearchCV. You can also use a pretrained neural network as a feature extractor (embeddings) and train a scikit-learn model on top, and the array API support in recent scikit-learn lets some estimators run on PyTorch or CuPy arrays.

Read full answer

Applies toscikit-learn 1.5+, PyTorch 2.x, skorch 1.x (Python 3.12)

Integration Patterns

Pattern Example
Shared preprocessing ColumnTransformer → NumPy → torch.tensor
Splits and CV train_test_split, StratifiedKFold indices for a PyTorch loop
Metrics roc_auc_score, classification_report on network outputs
Estimator wrappers skorch NeuralNetClassifier, SciKeras KerasClassifier
Embeddings + classic model Sentence or image embeddings → LogisticRegression
Array API (experimental) sklearn.set_config(array_api_dispatch=True) with supported estimators
Model exchange Convert sklearn pipelines to ONNX (skl2onnx) for serving alongside neural nets

Code Example: Scikit-Learn Preprocessing and Metrics Around a PyTorch Model

import torch
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from torch import nn

torch.manual_seed(42)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.25, random_state=42)

scaler = StandardScaler().fit(X_train)                 # fit on training data only
Xt = torch.tensor(scaler.transform(X_train), dtype=torch.float32)
yt = torch.tensor(y_train, dtype=torch.float32)

model = nn.Sequential(nn.Linear(X.shape[1], 32), nn.ReLU(), nn.Linear(32, 1))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(300):
    optimizer.zero_grad()
    loss = nn.functional.binary_cross_entropy_with_logits(model(Xt).squeeze(1), yt)
    loss.backward()
    optimizer.step()

with torch.no_grad():
    proba = torch.sigmoid(model(torch.tensor(scaler.transform(X_test), dtype=torch.float32))).squeeze(1).numpy()
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 4))

With skorch, the same network becomes NeuralNetBinaryClassifier(module=..., max_epochs=...) and can sit in make_pipeline(StandardScaler(), net) inside GridSearchCV.

Common Pitfalls

  • Fitting scalers on all data before splitting.

  • dtype mismatches (scikit-learn returns float64; PyTorch layers default to float32).

  • Expensive deep models inside large grid searches; use targeted searches or dedicated tuners (Optuna).

47

What role does joblib play when working with scikit-learn?

joblib is a library that scikit-learn depends on for two jobs. First, parallelism: whenever an estimator or tool accepts n_jobs (random forests, GridSearchCV, cross_val_score, permutation_importance), joblib runs the work across CPU cores using processes or threads, and joblib.parallel_backend / parallel_config controls how. Second, persistence: joblib.dump and joblib.load save and load fitted models and pipelines efficiently, handling large NumPy arrays better than plain pickle and supporting compression. It also offers Memory for caching expensive function results, which Pipeline(memory=...) uses to cache fitted transformers.

Read full answer

Applies toscikit-learn 1.5+, joblib 1.4+ (Python 3.12)

Roles

Role How it is used
Parallel computation n_jobs=-1 in estimators and model selection
Backend control with joblib.parallel_config(backend="threading", n_jobs=4):
Model persistence joblib.dump(pipe, "model.joblib"), joblib.load(...)
Compression joblib.dump(obj, path, compress=3)
Caching Pipeline(steps, memory=joblib.Memory("cache_dir"))
Custom parallel loops Parallel(n_jobs=-1)(delayed(f)(x) for x in items)

Code Example

import tempfile
from pathlib import Path

import joblib
import numpy as np
from joblib import Parallel, delayed
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=2000, n_features=20, random_state=42)

# Parallel grid search: joblib spreads the fits over all cores
search = GridSearchCV(RandomForestClassifier(random_state=42),
                      {"n_estimators": [100, 300], "max_depth": [None, 8]},
                      cv=5, n_jobs=-1, scoring="roc_auc")
search.fit(X, y)
print("best params:", search.best_params_)

# Persistence
tmp = Path(tempfile.mkdtemp())
pipe = make_pipeline(StandardScaler(), search.best_estimator_).fit(X, y)
joblib.dump(pipe, tmp / "model.joblib", compress=3)
restored = joblib.load(tmp / "model.joblib")          # load only files you trust
print("same predictions:", np.array_equal(restored.predict(X[:50]), pipe.predict(X[:50])))

# Custom parallel loop
def bootstrap_mean(seed):
    rng = np.random.default_rng(seed)
    return rng.choice(X[:, 0], size=len(X)).mean()
means = Parallel(n_jobs=-1)(delayed(bootstrap_mean)(s) for s in range(100))
print("bootstrap SE of feature 0 mean:", round(float(np.std(means)), 4))

Common Pitfalls

  • Loading joblib/pickle files from untrusted sources (they can execute code); consider skops for safer sharing.

  • Loading a model with a different scikit-learn version than it was saved with.

  • Nested parallelism (n_jobs=-1 in both the model and the search) oversubscribing CPUs.

48

How would you build a recommendation system using scikit-learn?

Scikit-learn has no dedicated recommender module, but its building blocks cover the main approaches. Content-based recommendations use TfidfVectorizer on item descriptions and NearestNeighbors (cosine) to find similar items. Collaborative filtering factorizes a sparse user-item matrix with TruncatedSVD or NMF to get latent factors and score unseen items. Ranking models frame recommendation as supervised learning: predict the probability a user interacts with a candidate item from user, item and context features with a classifier such as HistGradientBoostingClassifier. Evaluate with ranking metrics (precision@k, recall@k, NDCG via ndcg_score) on a time-based holdout. For large-scale production, dedicated libraries (implicit, LightFM, TensorFlow Recommenders) are better suited.

Read full answer

Applies toscikit-learn 1.5+, SciPy 1.13+ (Python 3.12)

Approaches

Approach scikit-learn tools Good for
Content-based TfidfVectorizer, NearestNeighbors(metric="cosine") New items, explainability
Item-item similarity cosine_similarity on the user-item matrix Small catalogs
Matrix factorization TruncatedSVD, NMF Implicit feedback, latent tastes
Learning to rank (pointwise) Gradient boosting classifiers on candidate pairs Rich features, re-ranking
Evaluation ndcg_score, custom precision@k Ranking quality

Code Example: Matrix Factorization with TruncatedSVD

import numpy as np
from scipy.sparse import csr_matrix
from sklearn.decomposition import TruncatedSVD

rng = np.random.default_rng(42)
n_users, n_items, k_true = 300, 100, 4
user_taste = rng.normal(size=(n_users, k_true))
item_traits = rng.normal(size=(n_items, k_true))
affinity = user_taste @ item_traits.T
interactions = (affinity + rng.normal(0, 1, affinity.shape) > 2.5).astype(float)   # implicit clicks

# Hold out one interaction per user for evaluation
train = interactions.copy()
held_out = np.full(n_users, -1)
for u in range(n_users):
    liked = np.flatnonzero(train[u])
    if len(liked) > 1:
        held_out[u] = rng.choice(liked)
        train[u, held_out[u]] = 0

svd = TruncatedSVD(n_components=8, random_state=42)
user_factors = svd.fit_transform(csr_matrix(train))
scores = user_factors @ svd.components_
scores[train > 0] = -np.inf                        # do not recommend already-seen items

top10 = np.argsort(-scores, axis=1)[:, :10]
evaluated = held_out >= 0
hit_rate = np.mean([held_out[u] in top10[u] for u in np.flatnonzero(evaluated)])
popular_top10 = np.argsort(-train.sum(axis=0))[:10]
pop_rate = np.mean([held_out[u] in popular_top10 for u in np.flatnonzero(evaluated)])
print(f"hit rate@10: SVD {hit_rate:.3f} vs popularity baseline {pop_rate:.3f}")

Always compare with a popularity baseline: it is surprisingly strong, and a recommender that cannot beat it is not adding value.

Common Pitfalls

  • Random holdouts that let the model learn from interactions that happened after the test ones.

  • Treating missing interactions as dislikes in implicit-feedback data.

  • Ignoring cold-start users and items; combine with content features.

49

How do you diagnose and fix performance issues in a scikit-learn model?

Diagnosing a weak scikit-learn model follows a sequence: confirm the evaluation is sound (a proper split, the right metric, a baseline from DummyClassifier/DummyRegressor, no leakage), then compare training and validation scores with cross_validate(return_train_score=True) and learning curves to tell high bias (both scores low) from high variance (large gap). Then inspect errors with confusion matrices and residuals, check feature usefulness with permutation importance, and fix accordingly: more or better features and more flexible models for bias; regularization, simpler models or more data for variance; class weights and threshold tuning for imbalance. If the problem is speed, profile and use faster estimators.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Diagnosis Table

Symptom Likely cause Fixes
Barely beats the Dummy baseline Weak features, wrong problem framing Feature engineering, check labels
Train and validation both low High bias (underfitting) More flexible model, interactions, less regularization
Train high, validation low High variance (overfitting) Regularization, fewer features, more data, ensembles
Validation suspiciously high Leakage Pipelines, time/group splits, audit features
Good accuracy, poor minority recall Imbalance, threshold class_weight, TunedThresholdClassifierCV
Slow training Model or data size HistGradientBoosting, SGDClassifier, n_jobs=-1

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import cross_validate, learning_curve, train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5, random_state=42)

for name, model in [("dummy", DummyClassifier(strategy="most_frequent")),
                    ("deep forest", RandomForestClassifier(random_state=42)),
                    ("shallow forest", RandomForestClassifier(max_depth=6, random_state=42))]:
    cv = cross_validate(model, X, y, cv=5, scoring="roc_auc", return_train_score=True)
    print(f"{name:>14}: train AUC {cv['train_score'].mean():.3f}, validation AUC {cv['test_score'].mean():.3f}")

sizes, train_scores, val_scores = learning_curve(
    RandomForestClassifier(max_depth=6, random_state=42), X, y, cv=5,
    train_sizes=np.linspace(0.1, 1.0, 5), scoring="roc_auc")
for n, tr, va in zip(sizes, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"n={n:>5}: train {tr:.3f}, validation {va:.3f}")

X_train, X_val, y_train, y_val = train_test_split(X, y, stratify=y, random_state=42)
fitted = RandomForestClassifier(max_depth=6, random_state=42).fit(X_train, y_train)
imp = permutation_importance(fitted, X_val, y_val, scoring="roc_auc", n_repeats=5, random_state=42)
print("most useful features:", np.argsort(-imp.importances_mean)[:5])

A large train-validation gap for the unrestricted forest points to variance; if validation keeps rising along the learning curve, more data would help.

Common Pitfalls

  • Tuning hyperparameters before checking for leakage or label problems.

  • Judging on a single split; use cross-validation with confidence in the spread.

  • Using accuracy on imbalanced data.

50

How would you build a scikit-learn pipeline for social media text?

A scikit-learn pipeline for social media text combines text cleaning tailored to posts (URLs, mentions, hashtags, emojis, elongated words), vectorization with TF-IDF over word n-grams plus character n-grams (robust to misspellings and slang), optional numeric metadata (likes, length, hashtag count) through a ColumnTransformer, and a fast linear classifier such as LogisticRegression or LinearSVC. Wrapping everything in a Pipeline lets cross-validation and grid search tune cleaning and model together without leakage. For analysis beyond classification, the same TF-IDF features feed topic models (NMF) or clustering.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Pipeline Design

Stage Tool
Cleaning FunctionTransformer with regexes (URLs → URL, @user → USER, keep hashtag words)
Word features TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True)
Character features TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5))
Metadata StandardScaler on numeric columns
Combine ColumnTransformer
Model LogisticRegression(class_weight="balanced")
Tuning GridSearchCV over C and n-gram ranges, f1_macro scoring
Topics NMF on TF-IDF for exploratory analysis

Code Example

import re

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

def clean_posts(texts):
    out = []
    for t in texts:
        t = re.sub(r"https?://\S+", " URL ", t)
        t = re.sub(r"@\w+", " USER ", t)
        t = re.sub(r"#(\w+)", r" \1 ", t)                 # keep the hashtag word
        t = re.sub(r"(.)\1{2,}", r"\1\1", t)              # "soooo" -> "soo"
        out.append(t.lower())
    return out

rng = np.random.default_rng(42)
pos = ["love this", "sooo good", "best day ever", "amazing service #happy"]
neg = ["worst app", "totally broken", "hate the update", "refund now @support"]
rows = []
for _ in range(600):
    label = int(rng.integers(0, 2))
    text = rng.choice(pos if label else neg) + " " + rng.choice(["", "https://t.co/x", "@friend", "!!!"])
    rows.append({"text": text, "likes": int(rng.poisson(5 + 10 * label)), "label": label})
df = pd.DataFrame(rows)

text_features = make_pipeline(FunctionTransformer(clean_posts),
                              TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True))
char_features = make_pipeline(FunctionTransformer(clean_posts),
                              TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)))
preprocess = ColumnTransformer([
    ("words", text_features, "text"),
    ("chars", char_features, "text"),
    ("meta", StandardScaler(), ["likes"]),
])
model = make_pipeline(preprocess, LogisticRegression(max_iter=2000, class_weight="balanced"))

X_train, X_test, y_train, y_test = train_test_split(
    df[["text", "likes"]], df["label"], test_size=0.25, stratify=df["label"], random_state=42)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test), digits=3))

Passing the column name as a string ("text") gives the vectorizer a 1-D column of strings, which is what text vectorizers expect.

Common Pitfalls

  • Removing emojis, negations or punctuation that carry sentiment.

  • Random splits when the same user or near-duplicate posts (retweets) appear in train and test; split by user or deduplicate.

  • Training on one platform’s language and deploying on another without re-evaluation.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Scikit-Learn answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects