Skip to content
Machine Learning & Data Science

SVM interview questions and answers

70 questions, 70 with code. About 3 h 42 min to read them all.

By TestLab · Updated

Plan for an interview
70 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is a Support Vector Machine (SVM) in machine learning?

A Support Vector Machine (SVM) is a supervised learning algorithm that classifies data by finding the boundary (hyperplane) that separates the classes with the widest possible margin. Only the training points closest to the boundary, the support vectors, determine where it lies. With the kernel trick, SVMs can learn non-linear boundaries efficiently, and a variant called Support Vector Regression (SVR) handles numeric targets. SVMs work very well on small to medium, high-dimensional datasets such as text.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Ideas

Idea Meaning
Hyperplane The decision boundary w·x + b = 0; a line in 2D, a plane in 3D
Margin Distance between the boundary and the closest points of each class
Support vectors Points on or inside the margin; removing other points does not change the model
Soft margin © Allows some violations; large C = fewer violations, less regularization
Kernel Computes similarity in a higher-dimensional space without building it: linear, polynomial, RBF, sigmoid
Hinge loss Penalizes points on the wrong side of their margin

The soft-margin objective, in words: minimize (1/2)||w||^2 (which maximizes the margin) plus C times the total slack (how far points violate the margin).

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))       # scaling is essential for SVMs
grid = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.1]}, cv=5)
grid.fit(X_train, y_train)

print("best parameters:", grid.best_params_)
svc = grid.best_estimator_.named_steps["svc"]
print("support vectors per class:", svc.n_support_)
print(classification_report(y_test, grid.predict(X_test)))

Strengths and Weaknesses

Strengths Weaknesses
Effective in high dimensions (text, genomics) Kernel SVM training scales roughly O(n^2) to O(n^3): slow beyond tens of thousands of samples
Margin maximization gives good generalization Needs feature scaling and tuning of C and gamma
Flexible boundaries via kernels No native probabilities (Platt scaling is extra cost)
Model depends only on support vectors Less interpretable with non-linear kernels

Interview Tips

  • Explain the margin, support vectors, C and the kernel trick in plain words.

  • Mention LinearSVC or SGDClassifier(loss="hinge") for large datasets.

  • Note that decision_function values are distances to the boundary, not probabilities.

2

What is a hyperplane in SVM?

A hyperplane is a flat decision boundary with one dimension fewer than the feature space: a line in 2D, a plane in 3D, and a “hyperplane” in higher dimensions. In an SVM it is defined by w·x + b = 0, where w is a weight vector perpendicular to the boundary and b shifts it from the origin. Points with w·x + b > 0 are assigned to one class and points with a negative value to the other, and the SVM chooses the hyperplane that leaves the widest margin between the classes.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Geometry

Element Meaning
w Normal vector: points toward the positive class; its direction sets the boundary’s orientation
b Bias: shifts the hyperplane
w·x + b Signed score; the sign gives the class

| (w·x + b) / ||w|| | Signed distance of x from the hyperplane | | w·x + b = ±1 | The two margin hyperplanes passing through the support vectors |

With a non-linear kernel, the boundary is still a hyperplane, but in the implicit high-dimensional feature space; in the original space it appears curved.

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=200, n_features=2, n_redundant=0, n_clusters_per_class=1,
                           class_sep=2.0, random_state=42)
model = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0)).fit(X, y)
svc = model.named_steps["svc"]
w, b = svc.coef_[0], svc.intercept_[0]
print(f"hyperplane (scaled space): {w[0]:.3f}*x1 + {w[1]:.3f}*x2 + {b:.3f} = 0")

x_new = model.named_steps["standardscaler"].transform(X[:3])
scores = x_new @ w + b
print("scores:", scores.round(3))
print("same as decision_function:", np.allclose(scores, model.decision_function(X[:3])))
print("signed distances:", (scores / np.linalg.norm(w)).round(3))
print("predicted classes:", model.predict(X[:3]))

Common Pitfalls

  • Reading coef_ as feature importance without scaling the features first.

  • Expecting coef_ for non-linear kernels: it exists only for the linear kernel.

  • Thinking a hyperplane can only separate linearly separable data in the original space; kernels change the space.

3

What is the maximum margin classifier in the context of SVM?

The maximum margin classifier is the linear classifier that separates two perfectly separable classes with the largest possible gap, or margin, between the decision boundary and the nearest points of either class. Among the infinitely many lines that separate the data, it picks the one farthest from both classes, which tends to generalize best. It is the foundation of the SVM: the soft-margin SVM relaxes it for overlapping data, and kernels extend it to non-linear boundaries.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

The Optimization Problem

  • Boundary: w·x + b = 0. Labels are y = +1 or -1.

  • Require every point to be on the correct side with a functional margin of at least 1: y_i (w·x_i + b) >= 1.

  • The geometric margin width is 2 / ||w||. Maximizing it is the same as minimizing (1/2)||w||^2.

  • So the hard-margin SVM solves: minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1 for all points.

In words: find the smallest weight vector that still puts every point on the right side, at least one unit of “score” away from the boundary; the smaller the weights, the wider the gap.

Hard Versus Soft Margin

Aspect Hard margin (maximum margin classifier) Soft margin SVM
Data requirement Perfectly linearly separable Any data
Violations allowed None Yes, penalized by C × slack
Sensitivity to outliers Very high: one point can move the boundary Controlled by C
Practical use Mostly theoretical Standard SVM

Code Example

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y = make_blobs(n_samples=60, centers=2, cluster_std=0.8, random_state=42)   # separable data

clf = SVC(kernel="linear", C=1e6)            # very large C approximates a hard margin
clf.fit(X, y)

w, b = clf.coef_[0], clf.intercept_[0]
print("w =", w.round(3), "b =", round(b, 3))
print("margin width =", round(2 / np.linalg.norm(w), 3))
print("support vectors:\n", clf.support_vectors_)
print("functional margins of support vectors:",
      (np.where(y[clf.support_] == 1, 1, -1) * clf.decision_function(clf.support_vectors_)).round(3))   # about 1

Support vectors sit exactly on the margin, so their functional margin is 1 (up to solver tolerance).

Common Pitfalls

  • Assuming real data is separable; use a soft margin with tuned C.

  • Forgetting feature scaling: the margin is measured in feature units, so unscaled features distort it.

  • Confusing the functional margin (score) with the geometric margin (distance).

4

What are support vectors, and why are they important in SVM?

Support vectors are the training points that lie on the margin boundaries or inside the margin (including misclassified points). They are the only points that define the SVM’s decision boundary: if you delete any other point and retrain, the model does not change. This makes SVM solutions sparse, which matters for interpretation, memory and prediction speed, since a kernel SVM’s prediction is a weighted sum of kernel values between the new point and each support vector.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Why They Matter

Aspect Role of support vectors
Decision boundary Determined entirely by them
Dual coefficients Only support vectors have non-zero Lagrange multipliers (alpha_i > 0)
Prediction f(x) = sum over SVs of alpha_i y_i K(x_i, x) + b; more SVs = slower predictions
Model complexity Many SVs (a large share of the data) can signal overlap, a small C or an overly flexible kernel
Generalization The leave-one-out error is bounded by the fraction of support vectors

Types

  • On the margin (0 < alpha_i < C): exactly at functional margin 1.

  • Margin violators (alpha_i = C): inside the margin or misclassified.

Code Example

import numpy as np
from sklearn.datasets import make_moons
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.2, random_state=42)

for C in (0.1, 1, 100):
    model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C)).fit(X, y)
    svc = model.named_steps["svc"]
    at_bound = np.isclose(np.abs(svc.dual_coef_[0]), C).sum()
    print(f"C={C:>5}: {len(svc.support_)} support vectors "
          f"({len(svc.support_) / len(X):.0%} of data), {at_bound} with alpha = C")

# Removing non-support vectors does not change the model
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1)).fit(X, y)
sv_idx = model.named_steps["svc"].support_
model_sv = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1))
model_sv.named_steps["standardscaler"].fit(X)                     # keep the same scaling
model_sv.named_steps["svc"].fit(model_sv.named_steps["standardscaler"].transform(X[sv_idx]), y[sv_idx])
print("same predictions:", (model.predict(X) == model_sv.predict(X)).mean())

The last line should print 1.0 or very close to it (tiny differences can come from solver tolerance).

Common Pitfalls

  • Assuming fewer support vectors is always better: too few can mean overfitting with a very large C on separable data.

  • Ignoring prediction cost: an RBF SVM with 50,000 support vectors is slow at inference.

  • Forgetting that support_vectors_ are in the scaled feature space when a scaler is used.

5

What is the difference between linear and non-linear SVM?

A linear SVM separates classes with a straight hyperplane in the original feature space, while a non-linear SVM uses a kernel (RBF, polynomial) to fit a hyperplane in an implicit higher-dimensional space, which appears as a curved boundary in the original space. Linear SVMs are fast, scale to millions of samples and are interpretable through their weights; non-linear SVMs capture complex patterns but train much more slowly and need careful tuning of kernel parameters.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Aspect Linear SVM Non-linear (kernel) SVM
Boundary Hyperplane w·x + b = 0 Curved: sum alpha_i y_i K(x_i, x) + b = 0
scikit-learn LinearSVC, SGDClassifier(loss="hinge"), SVC(kernel="linear") SVC(kernel="rbf" / "poly")
Training cost Roughly linear in n Between O(n^2) and O(n^3)
Prediction cost One dot product One kernel evaluation per support vector
Hyperparameters C C plus gamma, degree, coef0
Interpretability Weights per feature Hard to interpret
Best for High-dimensional data (text, many features), large n Low/medium dimensions with complex boundaries, n up to tens of thousands

Code Example

import time

from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = make_moons(n_samples=2000, noise=0.2, random_state=42)

models = {
    "LinearSVC": make_pipeline(StandardScaler(), LinearSVC(C=1.0)),
    "LinearSVC + poly features": make_pipeline(PolynomialFeatures(3), StandardScaler(), LinearSVC(C=1.0, max_iter=5000)),
    "SVC rbf": make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0)),
}
for name, model in models.items():
    start = time.perf_counter()
    acc = cross_val_score(model, X, y, cv=5).mean()
    print(f"{name:28s} accuracy {acc:.3f}  ({time.perf_counter() - start:.2f}s)")

The moons are not linearly separable, so the plain linear SVM should score lowest; adding polynomial features or using the RBF kernel lets it follow the curves.

Choosing

  • Try a linear model first; if it underfits and n is moderate, try an RBF kernel.

  • For large non-linear problems, use kernel approximations (Nystroem, RBFSampler) followed by a linear SVM.

Common Pitfalls

  • Using SVC(kernel="linear") on large datasets instead of the much faster LinearSVC.

  • Assuming an RBF kernel is always better; on high-dimensional data it often adds cost without accuracy.

6

How does the kernel trick work in SVM?

The kernel trick lets an SVM behave as if the data had been mapped into a high- (even infinite-) dimensional feature space, without ever computing that mapping. It works because the SVM’s dual optimization and its predictions use the data only through inner products between pairs of points; replacing every inner product x·z with a kernel function K(x, z) = φ(x)·φ(z) gives the same result as working in the mapped space, at the cost of computing K in the original space.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Why It Works

  • Dual problem: maximize sum(alpha_i) - (1/2) sum_i sum_j alpha_i alpha_j y_i y_j (x_i·x_j) subject to 0 <= alpha_i <= C and sum(alpha_i y_i) = 0.

  • Prediction: f(x) = sum_i alpha_i y_i (x_i·x) + b.

  • Both contain only inner products, so x_i·x_j can be replaced by K(x_i, x_j).

  • A function is a valid kernel when its kernel matrix is always positive semi-definite (Mercer’s condition).

Concrete Example

For 2D points, the polynomial kernel K(x, z) = (x·z)^2 equals the inner product of the explicit features φ(x) = (x1^2, sqrt(2) x1 x2, x2^2). Computing K takes one dot product and a square; computing φ grows quickly with the degree and dimension. The RBF kernel corresponds to an infinite-dimensional φ, which could never be computed explicitly.

import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

rng = np.random.default_rng(42)
x, z = rng.normal(size=2), rng.normal(size=2)

def phi(v):
    return np.array([v[0] ** 2, np.sqrt(2) * v[0] * v[1], v[1] ** 2])

print("kernel in original space:", round(float(np.dot(x, z) ** 2), 6))
print("dot product after mapping:", round(float(np.dot(phi(x), phi(z))), 6))     # identical

# Circles are not linearly separable in 2D, but are with the degree-2 kernel
X, y = make_circles(n_samples=400, factor=0.4, noise=0.08, random_state=42)
for kernel in ("linear", "poly", "rbf"):
    clf = SVC(kernel=kernel, degree=2, coef0=0) if kernel == "poly" else SVC(kernel=kernel)
    print(kernel, round(cross_val_score(clf, X, y, cv=5).mean(), 3))

Trade-Offs

Benefit Cost
Non-linear boundaries with a linear algorithm Kernel matrix is n × n: memory and time grow quickly with n
Works on non-vector data (strings, graphs) via custom kernels Kernel and its parameters must be chosen and tuned
Infinite-dimensional features (RBF) Less interpretable; predictions depend on all support vectors

Common Pitfalls

  • Thinking the data is actually transformed: only pairwise similarities are computed.

  • Applying kernels to unscaled features, which distorts distances.

  • Using exact kernels on very large data; use Nystroem or RBFSampler approximations with a linear model.

7

Which kernels can be used in SVM? Give examples of each.

A kernel is a function K(x, z) that returns the inner product of two points in some feature space, which lets an SVM learn non-linear boundaries without computing that space explicitly. The standard kernels are linear, polynomial, RBF (Gaussian) and sigmoid; specialized kernels exist for strings, graphs, histograms and other structured data, and you can define custom kernels as long as they are valid (positive semi-definite). Choosing a kernel encodes your assumption about what makes two examples similar.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Kernel Overview

Kernel Formula Key parameters Example use
Linear x·z C Text classification with TF-IDF, high-dimensional sparse data
Polynomial (gamma * x·z + coef0)^degree degree, gamma, coef0 Feature interactions, normalized image pixels

| RBF (Gaussian) | exp(-gamma * ||x - z||^2) | gamma, C | Default for non-linear tabular data (moons, sensor data) | | Sigmoid | tanh(gamma * x·z + coef0) | gamma, coef0 | Rarely; resembles a two-layer neural network; not always valid | | Laplacian | exp(-gamma * ||x - z||_1) | gamma | Data with sharp changes | | Chi-squared | Based on histogram differences | gamma | Bag-of-visual-words, histograms | | String / graph kernels | Count shared substrings or substructures | Problem-specific | Protein sequences, molecules |

The RBF kernel in words: similarity is 1 for identical points and decays toward 0 as distance grows; gamma sets how fast it decays.

Code Example

from sklearn.datasets import make_moons
from sklearn.metrics.pairwise import laplacian_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=400, noise=0.25, random_state=42)

configs = {
    "linear": SVC(kernel="linear"),
    "poly (degree 3)": SVC(kernel="poly", degree=3, coef0=1),
    "rbf": SVC(kernel="rbf", gamma="scale"),
    "sigmoid": SVC(kernel="sigmoid", coef0=0.0),
    "laplacian (custom)": SVC(kernel=laplacian_kernel),
}
for name, svc in configs.items():
    scores = cross_val_score(make_pipeline(StandardScaler(), svc), X, y, cv=5)
    print(f"{name:20s} accuracy {scores.mean():.3f} ± {scores.std():.3f}")

Any callable that returns the kernel matrix between two sets of rows can be passed as kernel, as shown with laplacian_kernel.

Choosing a Kernel

  • Start with linear when features are many (text) or n is large; it is fast and hard to beat there.

  • Use RBF as the default for low- to medium-dimensional non-linear data, tuning C and gamma together.

  • Use polynomial when interactions of a known degree matter.

  • Use domain kernels when data is not a fixed-length vector.

Common Pitfalls

  • Using RBF on very high-dimensional sparse text, where linear is usually as good and much faster.

  • High polynomial degrees, which overfit and become numerically unstable.

  • Custom kernels that are not positive semi-definite, which can make the optimization fail.

8

What is a soft margin in SVM, and why is it used?

A soft margin lets an SVM accept some points inside the margin or on the wrong side of the boundary, paying a penalty for each violation. Real data is rarely perfectly separable and often has noise or outliers, so a hard margin either has no solution or produces a narrow, overfitted boundary. The soft margin adds slack variables that measure each violation and a parameter C that sets how expensive violations are, trading margin width against training errors.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

The Objective

Minimize (1/2)||w||^2 + C * sum(xi_i) subject to y_i (w·x_i + b) >= 1 - xi_i and xi_i >= 0.

  • xi_i (slack) is 0 for points outside the margin on the correct side, between 0 and 1 for points inside the margin but correctly classified, and greater than 1 for misclassified points.

  • In words: keep the weights small (wide margin) while keeping the total amount of violation small; C decides which matters more.

  • Equivalent form: minimize (1/2)||w||^2 + C * sum(max(0, 1 - y_i f(x_i))), which is hinge loss with L2 regularization.

Effect of C

C Margin Violations Risk
Small (e.g. 0.01) Wide Many tolerated Underfitting
Moderate (e.g. 1) Balanced Some Usually good
Large (e.g. 1000) Narrow Few Overfitting to noise and outliers

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=300, n_features=2, n_redundant=0, n_clusters_per_class=1,
                           flip_y=0.1, class_sep=1.0, random_state=42)    # overlapping, noisy labels

for C in (0.01, 0.1, 1, 10, 1000):
    model = make_pipeline(StandardScaler(), SVC(kernel="linear", C=C))
    cv = cross_val_score(model, X, y, cv=5).mean()
    model.fit(X, y)
    svc = model.named_steps["svc"]
    margin = 2 / np.linalg.norm(svc.coef_)
    print(f"C={C:>7}: margin {margin:.3f}, support vectors {len(svc.support_)}, CV accuracy {cv:.3f}")

Expect the margin and the number of support vectors to shrink as C grows; the best C is the one with the highest cross-validated score.

Common Pitfalls

  • Choosing C by training accuracy, which always favors large C.

  • Tuning C without scaling features; the right C depends on feature scale.

  • Using a hard margin (huge C) on noisy data.

9

How does SVM handle multi-class classification?

SVMs are binary classifiers by design, so multi-class problems are handled by combining several binary SVMs. The two standard strategies are one-vs-rest (OvR), which trains one classifier per class against all others, and one-vs-one (OvO), which trains one classifier for every pair of classes and predicts by voting. There are also direct multi-class formulations (Crammer-Singer). In scikit-learn, SVC uses one-vs-one internally, while LinearSVC uses one-vs-rest.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Strategies

Strategy Classifiers for K classes Each trained on Prediction Used by
One-vs-rest K All data (one class vs the rest) Highest decision score LinearSVC (default), OneVsRestClassifier
One-vs-one K(K-1)/2 Only the two classes’ data Majority vote SVC, NuSVC, OneVsOneClassifier
Crammer-Singer One joint problem All data Highest score LinearSVC(multi_class="crammer_singer")

Trade-offs: OvO trains many more models but each on less data, which suits kernel SVMs whose cost grows faster than linearly with n. OvR trains fewer models but creates imbalanced binary problems.

Code Example

from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.multiclass import OneVsRestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = load_digits(return_X_y=True)          # 10 classes

ovo = make_pipeline(StandardScaler(), SVC(kernel="rbf", decision_function_shape="ovo"))
ovr = make_pipeline(StandardScaler(), OneVsRestClassifier(SVC(kernel="rbf")))
linear = make_pipeline(StandardScaler(), LinearSVC(C=0.1))       # one-vs-rest by default

for name, model in [("SVC (one-vs-one)", ovo), ("OneVsRest(SVC)", ovr), ("LinearSVC (OvR)", linear)]:
    print(f"{name:18s} accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

ovo.fit(X, y)
print("OvO decision_function shape:", ovo.decision_function(X[:1]).shape)   # (1, 45) = 10*9/2 pairs

SVC‘s default decision_function_shape="ovr" only reshapes the one-vs-one scores into one column per class; training is still one-vs-one.

Common Pitfalls

  • Assuming decision_function_shape="ovr" changes the training strategy.

  • Using probability=True without need; it adds an internal cross-validation and slows training.

  • Ignoring class imbalance in OvR; use class_weight="balanced".

10

What are the limitations of SVMs?

SVMs have several practical limitations: kernel SVMs scale poorly to large datasets, they are sensitive to feature scaling and to the choice of C and kernel parameters, they do not output calibrated probabilities natively, and non-linear models are hard to interpret. They also handle noisy, heavily overlapping or imbalanced data less gracefully without tuning, and they need numeric inputs. Knowing these helps you decide when gradient-boosted trees, logistic regression or neural networks are better choices.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Limitations and Workarounds

Limitation Why Workaround
Training time and memory Kernel SVM training is roughly O(n^2) to O(n^3); kernel matrix is n × n LinearSVC, SGDClassifier(loss="hinge"), Nystroem + linear SVM, subsampling
Slow prediction with many support vectors Each prediction evaluates the kernel against every support vector Fewer SVs (tune C), linear models, kernel approximation
Hyperparameter sensitivity C, gamma, degree interact strongly Grid or Bayesian search with cross-validation
Needs scaling Margins and RBF distances depend on feature scale StandardScaler in a Pipeline
No native probabilities Outputs distances to the boundary Platt scaling (probability=True or CalibratedClassifierCV)
Interpretability Non-linear decision functions Linear kernel weights, SHAP, permutation importance
Noisy, overlapping classes Many support vectors, unstable boundary Smaller C, better features
Imbalanced data Margin dominated by the majority class class_weight="balanced", threshold tuning
Categorical and missing values Requires numeric, complete inputs One-hot encoding, imputation
Multi-class Binary by design OvO/OvR combinations (built in)
Online learning Batch solvers SGDClassifier with partial_fit

Code Example: Training Time Growth

import time

from sklearn.datasets import make_classification
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

for n in (2000, 4000, 8000, 16000):
    X, y = make_classification(n_samples=n, n_features=20, flip_y=0.05, random_state=42)
    timings = {}
    for name, clf in [("SVC rbf", SVC()), ("LinearSVC", LinearSVC())]:
        start = time.perf_counter()
        make_pipeline(StandardScaler(), clf).fit(X, y)
        timings[name] = time.perf_counter() - start
    print(n, {k: f"{v:.2f}s" for k, v in timings.items()})

Each doubling of n should make the RBF SVC noticeably more than twice as slow, while LinearSVC grows roughly linearly; exact times depend on the machine.

When to Prefer Other Models

  • Large tabular datasets: gradient-boosted trees (XGBoost, LightGBM).

  • Probabilities needed for decisions: logistic regression or calibrated boosting.

  • Images, audio, raw text: neural networks.

11

What is the objective function of the SVM?

The SVM objective function balances two goals: a wide margin, achieved by keeping the weight vector small, and few margin violations, measured with slack variables or, equivalently, the hinge loss. For the soft-margin SVM the primal objective is to minimize (1/2)||w||^2 + C * sum(xi_i) subject to each point being on the correct side of its margin up to its slack. The same problem can be written as regularized hinge-loss minimization, and it has a dual form that enables kernels.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Three Equivalent Views

| Form | Objective | Notes | |—|—|—| | Primal (constrained) | Minimize (1/2)||w||^2 + C * sum(xi_i) s.t. y_i (w·x_i + b) >= 1 - xi_i, xi_i >= 0 | Quadratic program | | Unconstrained (hinge loss) | Minimize (1/2)||w||^2 + C * sum(max(0, 1 - y_i (w·x_i + b))) | Same as L2-regularized hinge loss; used by SGD and LinearSVC | | Dual | Maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j K(x_i, x_j) s.t. 0 <= alpha_i <= C, sum(alpha_i y_i) = 0 | Depends only on kernel values; solved by SMO in SVC |

In words: - (1/2)||w||^2: the margin width is 2/||w||, so a smaller w means a wider margin (regularization). - Hinge loss: zero for points beyond their margin on the correct side, growing linearly as a point moves into the margin or across the boundary. - C: how much a unit of violation costs compared with margin width; large C = less regularization.

Code Example: Evaluating the Objective

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=200, n_features=4, random_state=42)
X = StandardScaler().fit_transform(X)
y_pm = np.where(y == 1, 1, -1)                    # labels as +1 / -1
C = 1.0

svc = SVC(kernel="linear", C=C).fit(X, y)
w, b = svc.coef_[0], svc.intercept_[0]
margins = y_pm * (X @ w + b)
hinge = np.maximum(0, 1 - margins)

print("regularization term:", round(0.5 * w @ w, 4))
print("hinge-loss term:", round(C * hinge.sum(), 4))
print("primal objective:", round(0.5 * w @ w + C * hinge.sum(), 4))
print("points with non-zero slack:", int((hinge > 1e-6).sum()), "| support vectors:", len(svc.support_))

Points with non-zero slack are support vectors; support vectors exactly on the margin have zero slack, so the second count is at least as large as the first.

Variants

  • Squared hinge (LinearSVC default): penalizes violations quadratically.

  • L1 penalty on w (LinearSVC(penalty="l1")): sparse weights for feature selection.

  • SVR: epsilon-insensitive loss, ignoring errors smaller than epsilon.

Common Pitfalls

  • Forgetting that labels are encoded as ±1 in the formulas.

  • Comparing C values across differently scaled datasets.

12

What role do Lagrange multipliers play in SVM?

Lagrange multipliers turn the SVM’s constrained optimization problem (maximize the margin while keeping points on the correct side) into its dual problem. Each training point gets a multiplier alpha_i >= 0 that measures how strongly its constraint pushes on the solution. Points with alpha_i = 0 have no influence, and points with alpha_i > 0 are the support vectors. The dual depends on the data only through inner products, which is what makes the kernel trick possible.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

From Primal to Dual

  1. Primal (soft margin): minimize (1/2)||w||^2 + C sum(xi_i) s.t. y_i (w·x_i + b) >= 1 - xi_i, xi_i >= 0.

  2. Lagrangian: add each constraint times a multiplier, alpha_i for the margin constraints and mu_i for xi_i >= 0.

  3. Set the derivatives with respect to w, b and xi to zero:

  4. w = sum_i alpha_i y_i x_i: the weight vector is a weighted sum of training points.

  5. sum_i alpha_i y_i = 0.

  6. alpha_i = C - mu_i, so 0 <= alpha_i <= C.

  7. Substitute back to get the dual: maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j (x_i·x_j).

What the Multipliers Tell You (KKT Conditions)

Value of alpha_i Point’s position Role
0 Beyond the margin, correctly classified Not a support vector
0 < alpha_i < C Exactly on the margin Support vector; used to compute b
C Inside the margin or misclassified Bounded support vector (margin violator)

The complementary slackness condition, alpha_i * (y_i f(x_i) - 1 + xi_i) = 0, says that a multiplier can be non-zero only when its constraint is exactly active.

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=200, n_features=2, n_redundant=0, n_clusters_per_class=1,
                           flip_y=0.05, random_state=42)
X = StandardScaler().fit_transform(X)
C = 1.0
svc = SVC(kernel="linear", C=C).fit(X, y)

alpha_y = svc.dual_coef_[0]                   # alpha_i * y_i for each support vector
alphas = np.abs(alpha_y)
w_from_dual = alpha_y @ svc.support_vectors_  # w = sum alpha_i y_i x_i
print("w (coef_):    ", svc.coef_[0].round(4))
print("w from alphas:", w_from_dual.round(4))
print("sum alpha_i y_i =", round(float(alpha_y.sum()), 6))         # about 0
print("on margin (0 < alpha < C):", int(((alphas > 1e-8) & (alphas < C - 1e-8)).sum()),
      "| at bound (alpha = C):", int(np.isclose(alphas, C).sum()))

Interview Tips

  • Explain that the multipliers make the solution sparse: only support vectors have non-zero alpha.

  • Connect w = sum alpha_i y_i x_i to the kernel trick: predictions become sum alpha_i y_i K(x_i, x) + b.

  • Mention the box constraint 0 <= alpha_i <= C as where C enters the dual.

13

How is the dual problem solved in SVM optimization?

The SVM dual problem is a convex quadratic program over one Lagrange multiplier alpha_i per training point: maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j K(x_i, x_j) subject to 0 <= alpha_i <= C and sum(alpha_i y_i) = 0. Because it is convex, any local optimum is global. It is solved with general QP solvers for small problems, or with specialized decomposition methods, mainly Sequential Minimal Optimization (SMO) in libsvm (used by scikit-learn’s SVC), which update a small working set of multipliers at a time.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x, SciPy (Python 3.12)

Solution Methods

Method Idea Used in
General QP / interior point Solve the full QP with the n × n kernel matrix Small n; CVXOPT, educational code
Chunking / decomposition Optimize a subset of variables while fixing the rest Early SVM solvers (SVMlight)
SMO Optimize two multipliers at a time analytically (two are needed to keep sum alpha_i y_i = 0) libsvm → SVC, SVR, NuSVC
Dual coordinate descent Update one multiplier at a time (linear kernel without bias constraint) liblinear → LinearSVC

Steps After Solving

  1. Support vectors are the points with alpha_i > 0.

  2. For a linear kernel: w = sum alpha_i y_i x_i.

  3. Bias b: average y_k - sum_i alpha_i y_i K(x_i, x_k) over margin support vectors (0 < alpha_k < C).

  4. Predict: sign(sum_i alpha_i y_i K(x_i, x) + b).

Code Example: Solving the Dual with a Generic Optimizer

import numpy as np
from scipy.optimize import minimize
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y01 = make_blobs(n_samples=40, centers=2, cluster_std=1.2, random_state=42)
y = np.where(y01 == 1, 1.0, -1.0)
C = 1.0
K = X @ X.T                                   # linear kernel matrix
Q = (y[:, None] * y[None, :]) * K

def neg_dual(a):                              # minimize the negative of the dual objective
    return 0.5 * a @ Q @ a - a.sum()

def neg_dual_grad(a):
    return Q @ a - np.ones_like(a)

res = minimize(neg_dual, x0=np.zeros(len(y)), jac=neg_dual_grad, method="SLSQP",
               bounds=[(0, C)] * len(y),
               constraints={"type": "eq", "fun": lambda a: a @ y, "jac": lambda a: y})
alpha = res.x
sv = alpha > 1e-5
w = (alpha * y) @ X
on_margin = sv & (alpha < C - 1e-5)
b = np.mean(y[on_margin] - X[on_margin] @ w)

ref = SVC(kernel="linear", C=C).fit(X, y)
print("support vectors (ours vs sklearn):", sv.sum(), len(ref.support_))
print("w ours:", w.round(3), "| sklearn:", ref.coef_[0].round(3))
print("b ours:", round(b, 3), "| sklearn:", round(ref.intercept_[0], 3))

The two solutions should agree closely; small differences come from solver tolerances.

Why the Dual Is Used

  • It needs only kernel values, enabling non-linear kernels.

  • The solution is sparse (most alpha_i = 0).

  • Its size depends on n, not on the (possibly infinite) feature dimension.

Common Pitfalls

  • Computing b from bounded support vectors (alpha = C), which gives wrong values.

  • Building the full n × n kernel matrix for large n; decomposition solvers avoid this with caching.

14

How do you choose the regularization parameter C in SVM?

The regularization parameter C sets how heavily the SVM penalizes margin violations: a small C favors a wide margin and tolerates misclassified training points (stronger regularization), while a large C tries hard to classify every training point correctly (weaker regularization, risk of overfitting). C should be chosen with cross-validation, searching values on a logarithmic scale, and tuned jointly with gamma when using an RBF kernel, because the two interact.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How C Affects the Model

C Margin Support vectors Bias / variance
Very small (1e-3) Very wide Many High bias (underfitting)
Moderate (1) Balanced Fewer Often a good starting point
Very large (1e3+) Narrow Few (on separable data) High variance (overfitting)

Procedure

  1. Put scaling and the SVM in a Pipeline so each fold scales with its own training data.

  2. Search C on a log scale, e.g. 10^-3 ... 10^3.

  3. For RBF, search C and gamma together.

  4. Use a metric that matches the goal (F1 or ROC AUC for imbalanced data).

  5. Refine around the best value, then evaluate once on a held-out test set.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split, validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))

# 1. Validation curve for C alone (gamma fixed at "scale")
C_range = np.logspace(-3, 3, 7)
train_scores, val_scores = validation_curve(pipe, X_train, y_train, param_name="svc__C",
                                            param_range=C_range, cv=5, scoring="roc_auc")
for C, tr, va in zip(C_range, train_scores.mean(axis=1), val_scores.mean(axis=1)):
    print(f"C={C:>8.3f}  train AUC {tr:.3f}  val AUC {va:.3f}")

# 2. Joint search over C and gamma
grid = GridSearchCV(pipe, {"svc__C": np.logspace(-2, 3, 6), "svc__gamma": np.logspace(-4, 0, 5)},
                    cv=5, scoring="roc_auc", n_jobs=-1)
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV AUC:", round(grid.best_score_, 3))
print("test AUC:", round(grid.score(X_test, y_test), 3))

A widening gap between training and validation scores as C grows is the sign of overfitting.

Tips

  • If the best value is at the edge of the grid, extend the grid.

  • Noisy labels usually favor smaller C.

  • In LinearSVC and SVC, C multiplies the loss; in SGDClassifier, alpha multiplies the regularization, so roughly alpha ≈ 1 / (C * n_samples).

Common Pitfalls

  • Choosing C by training accuracy.

  • Scaling the data before splitting, which leaks information into cross-validation.

  • Tuning C on the test set.

15

What is the hinge loss function?

Hinge loss is the loss function behind SVMs: for a label y in {-1, +1} and a raw score f(x) = w·x + b, it is max(0, 1 - y f(x)). It is zero when a point is on the correct side of the boundary with a margin of at least 1, and it grows linearly as the point moves into the margin or onto the wrong side. Minimizing hinge loss plus an L2 penalty on w is exactly the soft-margin SVM, which connects SVMs to other regularized linear models.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Behaviour

y f(x) (margin) Hinge loss Meaning
≥ 1 0 Correct and confident: ignored
between 0 and 1 1 - y f(x) (between 0 and 1) Correct but inside the margin
< 0 > 1 Misclassified

Comparison with Other Losses

Loss Formula Properties
Hinge max(0, 1 - m) Sparse solutions (only margin points matter); not differentiable at m = 1
Squared hinge max(0, 1 - m)^2 Smooth; penalizes large violations more (LinearSVC default)
Logistic (log loss) log(1 + exp(-m)) Never exactly zero; gives probabilities (logistic regression)
0-1 loss 1 if m < 0 else 0 What we care about, but not convex

Here m = y f(x). Hinge loss is a convex upper bound on the 0-1 loss, which makes it tractable.

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import hinge_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

def hinge(y_pm, scores):
    return np.maximum(0, 1 - y_pm * scores)

margins = np.array([2.0, 1.0, 0.5, 0.0, -1.0])
print("losses:", hinge(np.ones(5), margins))        # [0.  0.  0.5 1.  2. ]

# A linear SVM trained with SGD on hinge loss
X, y = make_classification(n_samples=2000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = make_pipeline(StandardScaler(),
                      SGDClassifier(loss="hinge", alpha=1e-4, max_iter=1000, random_state=42))
model.fit(X_train, y_train)

scores = model.decision_function(X_test)
y_pm = np.where(y_test == 1, 1, -1)
print("mean hinge loss (manual):", round(hinge(y_pm, scores).mean(), 4))
print("mean hinge loss (sklearn):", round(hinge_loss(y_test, scores), 4))
print("test accuracy:", round(model.score(X_test, y_test), 3))

Subgradient

Hinge loss is not differentiable at y f(x) = 1, so gradient methods use a subgradient: -y x when y f(x) < 1, and 0 otherwise. Only points that violate the margin update the weights.

Common Pitfalls

  • Using labels 0/1 instead of -1/+1 in a manual implementation.

  • Expecting probabilities from a hinge-loss model; use log loss or calibration.

  • Omitting regularization, which lets the weights grow to make every margin large.

16

Why do kernel parameters like sigma in the RBF kernel matter?

The RBF (Gaussian) kernel measures similarity as K(x, z) = exp(-||x - z||^2 / (2 sigma^2)), which scikit-learn writes as exp(-gamma * ||x - z||^2) with gamma = 1 / (2 sigma^2). Sigma (or gamma) sets the kernel’s reach: how far a training point’s influence extends. A small sigma (large gamma) makes each point influence only its immediate neighbourhood, producing very wiggly boundaries that overfit; a large sigma (small gamma) makes the boundary smooth, approaching a linear model, and can underfit.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Effect of Gamma

gamma (sigma) Influence of each point Boundary Support vectors Risk
Very small (large sigma) Very broad Almost linear Many Underfitting
Moderate Local but smooth Follows the true shape Moderate Good fit
Very large (small sigma) Tiny Islands around points Nearly all points Overfitting

scikit-learn’s default gamma="scale" uses 1 / (n_features * X.var()), a sensible starting point after standardization.

Interaction with C

  • Large gamma + large C: memorizes the training data.

  • Small gamma + small C: too simple.

  • Good models lie along a diagonal band in the (C, gamma) grid, so tune them together.

Code Example

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=400, noise=0.3, random_state=42)

for gamma in (0.01, 0.1, 1, 10, 100):
    sigma = np.sqrt(1 / (2 * gamma))
    model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, gamma=gamma))
    res = cross_validate(model, X, y, cv=5, return_train_score=True)
    n_sv = len(model.fit(X, y).named_steps["svc"].support_)
    print(f"gamma={gamma:>6} (sigma={sigma:.2f}): train {res['train_score'].mean():.3f}, "
          f"val {res['test_score'].mean():.3f}, support vectors {n_sv}")

Look for the gamma where validation accuracy peaks; past it, training accuracy keeps rising while validation accuracy falls.

Practical Tips

  • Always standardize features first; gamma is meaningless if features have very different scales.

  • Search gamma on a log scale, for example 10^-4 to 10^1.

  • A useful heuristic for sigma is the median pairwise distance between training points.

Common Pitfalls

  • Mixing up the two parameterizations: larger gamma means smaller sigma.

  • Tuning gamma with C fixed at an extreme value.

  • Reusing a gamma tuned on one feature set after adding or removing features.

17

What is the computational complexity of training an SVM?

Training a kernel SVM with standard solvers such as libsvm’s SMO takes roughly between O(n^2) and O(n^3) time in the number of samples n, plus up to O(n^2) memory for kernel values, so it becomes impractical beyond tens of thousands of samples. Linear SVMs trained with liblinear or stochastic gradient descent scale roughly linearly with the number of samples and features, and handle millions of examples. Prediction costs O(number of support vectors × d) for kernel SVMs and O(d) for linear ones.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Complexity Summary

Model / solver Training time Memory Prediction per sample
SVC / SVR (libsvm, SMO) ~O(n^2 d) to O(n^3 d) Kernel cache up to O(n^2) O(n_SV d)
LinearSVC (liblinear, coordinate descent) ~O(n d) per pass O(n d) O(d)
SGDClassifier(loss="hinge") O(n d) per epoch O(d) for the model O(d)
Nystroem(k) + linear SVM O(n k d + n k^2) O(n k) O(k d)

The exponent depends on C, the kernel, how separable the data is and how many support vectors there are: noisy data with many support vectors is the slow case.

Practical Speed-Ups

  • Use LinearSVC or SGDClassifier for large or high-dimensional data (text).

  • Approximate kernels with Nystroem or RBFSampler and train a linear SVM.

  • Increase cache_size (MB) in SVC when memory allows.

  • Subsample, or train on cluster centroids, for model selection; retrain the final model on more data.

  • Use GPU implementations (e.g. cuML’s SVC) for large kernel SVMs.

Code Example: Measuring Scaling

import time

import numpy as np
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

def fit_time(model, X, y):
    start = time.perf_counter()
    model.fit(X, y)
    return time.perf_counter() - start

sizes = [2000, 4000, 8000]
times = {"SVC rbf": [], "Nystroem + LinearSVC": [], "LinearSVC": []}
for n in sizes:
    X, y = make_classification(n_samples=n, n_features=30, flip_y=0.1, random_state=42)
    times["SVC rbf"].append(fit_time(make_pipeline(StandardScaler(), SVC(cache_size=500)), X, y))
    times["Nystroem + LinearSVC"].append(fit_time(make_pipeline(
        StandardScaler(), Nystroem(n_components=300, random_state=42), LinearSVC()), X, y))
    times["LinearSVC"].append(fit_time(make_pipeline(StandardScaler(), LinearSVC()), X, y))

for name, t in times.items():
    growth = np.log2(t[-1] / t[0]) / np.log2(sizes[-1] / sizes[0])    # empirical exponent
    print(f"{name:22s} times {[round(v, 2) for v in t]}  ~O(n^{growth:.1f})")

The empirical exponent is noisy for small sizes, but kernel SVC should show clearly super-linear growth.

Interview Tips

  • Say “between quadratic and cubic in n” for kernel SVMs and explain why: pairwise kernel values.

  • Mention prediction cost, which is often overlooked: many support vectors make inference slow.

18

How does SVM maximize the margin?

SVM maximizes the margin by finding the weights w and bias b that keep every training point on the correct side of the boundary with a score of at least 1, y_i (w·x_i + b) >= 1, while making ||w|| as small as possible. Since the distance between the two margin hyperplanes is 2 / ||w||, minimizing (1/2)||w||^2 is the same as maximizing the margin. This turns a geometric goal into a convex quadratic program with a unique optimum, solved in practice through its dual.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Step by Step

  1. Scale fixing: w and b can be multiplied by any positive constant without moving the boundary, so fix the scale by requiring the closest points to have |w·x + b| = 1.

  2. Margin width: the distance from a point to the boundary is |w·x + b| / ||w||, so the closest points are 1 / ||w|| away on each side, and the full margin is 2 / ||w||.

  3. Optimization: maximize 2 / ||w|| ⇔ minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1.

  4. Soft margin: add slack xi_i and a penalty C sum(xi_i) for data that is not separable.

  5. Solve: convert to the dual with Lagrange multipliers; only support vectors get non-zero multipliers.

Why Maximize the Margin?

  • A wider margin means small changes in the inputs (noise) are less likely to flip predictions.

  • Statistical learning theory links larger margins to tighter generalization bounds, largely independent of the number of features.

  • The solution depends only on support vectors, so it is robust to points far from the boundary.

Code Example

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y = make_blobs(n_samples=80, centers=2, cluster_std=1.0, random_state=42)
svc = SVC(kernel="linear", C=1e5).fit(X, y)       # near hard margin

w, b = svc.coef_[0], svc.intercept_[0]
distances = np.abs(X @ w + b) / np.linalg.norm(w)
print("margin width 2/||w||:", round(2 / np.linalg.norm(w), 4))
print("closest point distance × 2:", round(2 * distances.min(), 4))   # matches the margin width

# Any other separating line has a smaller minimum distance; compare with a perturbed boundary
w_alt = w + np.array([0.1, -0.1])
if np.all(np.sign(X @ w_alt + b) == np.where(y == 1, 1, -1)):
    print("alternative still separates; its margin:", round(2 * (np.abs(X @ w_alt + b) / np.linalg.norm(w_alt)).min(), 4))
else:
    print("alternative boundary misclassifies some points")

Common Pitfalls

  • Forgetting scaling: the margin is measured in feature units, so one large-scale feature dominates.

  • Believing the maximum margin is always best on noisy data: the soft margin with tuned C trades margin for errors.

19

Can you derive the optimization problem for the soft margin SVM?

The soft-margin SVM starts from the hard-margin problem, which maximizes the margin while requiring every point to be correctly classified, and relaxes it with slack variables so that some points may violate the margin at a cost. The derivation below builds the primal problem from geometry, shows its equivalence to regularized hinge loss, and derives the dual through Lagrange multipliers, which reveals the box constraint 0 <= alpha_i <= C and the role of support vectors.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

1. Hard Margin

  • Classifier: f(x) = w·x + b, labels y_i ∈ {-1, +1}.

  • Distance of point i to the boundary: y_i f(x_i) / ||w||.

  • Fix the scale so the closest points have y_i f(x_i) = 1; the margin is then 2 / ||w||.

  • Problem: minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1 for all i.

2. Adding Slack

  • Allow violations: y_i (w·x_i + b) >= 1 - xi_i, with xi_i >= 0.

  • Penalize the total violation with weight C.

  • Primal soft-margin problem: minimize (1/2)||w||^2 + C sum_i xi_i subject to y_i (w·x_i + b) >= 1 - xi_i and xi_i >= 0.

3. Equivalent Unconstrained Form

At the optimum each slack is as small as allowed: xi_i = max(0, 1 - y_i (w·x_i + b)). Substituting gives minimize (1/2)||w||^2 + C sum_i max(0, 1 - y_i f(x_i)): L2-regularized hinge loss.

4. Lagrangian

L = (1/2)||w||^2 + C sum xi_i - sum alpha_i [y_i (w·x_i + b) - 1 + xi_i] - sum mu_i xi_i, with alpha_i, mu_i >= 0.

Setting the derivatives to zero: - With respect to w: w = sum_i alpha_i y_i x_i. - With respect to b: sum_i alpha_i y_i = 0. - With respect to xi_i: C - alpha_i - mu_i = 0, so 0 <= alpha_i <= C.

5. Dual Problem

Substituting back, the slack terms cancel:

maximize sum_i alpha_i - (1/2) sum_i sum_j alpha_i alpha_j y_i y_j (x_i·x_j) subject to 0 <= alpha_i <= C and sum_i alpha_i y_i = 0.

Replacing x_i·x_j with K(x_i, x_j) gives the kernel SVM.

6. KKT Interpretation

Condition Point
alpha_i = 0 Correct side, outside the margin (y_i f(x_i) >= 1)
0 < alpha_i < C Exactly on the margin (xi_i = 0)
alpha_i = C Inside the margin or misclassified (xi_i > 0)

Code Check

import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=150, n_features=3, flip_y=0.1, random_state=42)
X = StandardScaler().fit_transform(X)
C = 0.5
svc = SVC(kernel="linear", C=C).fit(X, y)
y_pm = np.where(y == 1, 1, -1)
f = svc.decision_function(X)
xi = np.maximum(0, 1 - y_pm * f)
alpha = np.zeros(len(X))
alpha[svc.support_] = np.abs(svc.dual_coef_[0])

print("alpha = 0 implies y*f >= 1:", np.all(y_pm[alpha < 1e-8] * f[alpha < 1e-8] >= 1 - 1e-3))
print("xi > 0 implies alpha = C:", np.allclose(alpha[xi > 1e-3], C))
print("w = sum alpha_i y_i x_i:", np.allclose((alpha * y_pm) @ X, svc.coef_[0], atol=1e-4))

All three checks should print True (within solver tolerance).

Interview Tips

  • Explain where C appears in the dual (the upper bound on alpha).

  • Mention that the dual needs only inner products, enabling kernels.

20

What preprocessing steps should you take before training an SVM?

Preprocessing for an SVM matters more than for tree models, because SVMs work with distances and dot products. The essential steps are: handle missing values, encode categorical features numerically, scale numeric features (standardization is the default), deal with outliers and class imbalance, and optionally reduce dimensionality or select features. All of these should be fitted on training data only, which is easiest inside a scikit-learn Pipeline with a ColumnTransformer.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Steps

Step Why Tool
Missing values SVMs cannot handle NaN SimpleImputer, KNNImputer
Categorical encoding Needs numeric input OneHotEncoder(handle_unknown="ignore")
Scaling Distances and margins depend on scale StandardScaler; MaxAbsScaler for sparse text; RobustScaler with outliers
Skewed features Extreme values dominate RBF distances np.log1p, PowerTransformer
Outliers Can become support vectors and bend the boundary Investigate, cap, or reduce C
Class imbalance Margin favors the majority class class_weight="balanced"
High dimensionality Speed and noise SelectKBest, L1-penalized LinearSVC, PCA
Text Needs vectors TfidfVectorizer (with a linear SVM)

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, PowerTransformer, StandardScaler
from sklearn.svm import SVC

rng = np.random.default_rng(42)
n = 600
df = pd.DataFrame({
    "age": rng.integers(18, 80, n).astype(float),
    "income": rng.lognormal(10, 1, n),                       # skewed
    "plan": rng.choice(["basic", "plus", "pro"], n),
    "region": rng.choice(["north", "south", "east", "west"], n),
})
df.loc[rng.choice(n, 30, replace=False), "age"] = np.nan    # missing values
y = ((df["income"] > df["income"].median()) & (df["plan"] != "basic")).astype(int)

numeric = ["age", "income"]
categorical = ["plan", "region"]
preprocess = ColumnTransformer([
    ("num", make_pipeline(SimpleImputer(strategy="median"), PowerTransformer()), numeric),  # also standardizes
    ("cat", make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore")), categorical),
])
model = Pipeline([("prep", preprocess), ("svc", SVC(kernel="rbf", class_weight="balanced"))])

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, stratify=y, random_state=42)
grid = GridSearchCV(model, {"svc__C": [0.1, 1, 10], "svc__gamma": ["scale", 0.1]}, cv=5, scoring="f1")
grid.fit(X_train, y_train)
print(grid.best_params_, "test F1:", round(grid.score(X_test, y_test), 3))

Common Pitfalls

  • Scaling the whole dataset before the split, which leaks test information.

  • Label-encoding nominal categories as 0, 1, 2, which invents an order and false distances.

  • Using StandardScaler on sparse TF-IDF matrices (it densifies them); use MaxAbsScaler or none.

  • Forgetting to apply exactly the same preprocessing at prediction time; a saved Pipeline guarantees it.

21

How do you handle categorical variables when training an SVM?

SVMs need numeric inputs, and because they rely on distances and dot products, the way categories are encoded directly shapes the model. Nominal categories (colour, city) should usually be one-hot encoded, so every category is equally distant from the others; ordinal categories (small, medium, large) can be mapped to ordered numbers. High-cardinality features need grouping of rare levels, target encoding with cross-fitting, or hashing. All encoders must be fitted on training data only, which a ColumnTransformer inside a Pipeline ensures.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Encoding Options

Feature type Encoder Notes for SVMs
Nominal, low cardinality OneHotEncoder(handle_unknown="ignore") Default choice; works with RBF and linear kernels
Ordinal OrdinalEncoder(categories=[[...]]) then scale Only when the order is meaningful
Nominal, many levels OneHotEncoder(min_frequency=..., max_categories=...) Groups rare categories into “infrequent”
High cardinality with signal TargetEncoder (cross-fitted in scikit-learn) Compact; avoids thousands of columns
Very high cardinality FeatureHasher Fixed size; collisions possible
Binary 0/1 One column is enough

Avoid label-encoding nominal categories as 0, 1, 2: the SVM would treat “category 2” as twice as far from “category 0” as “category 1” is.

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, TargetEncoder
from sklearn.svm import SVC

rng = np.random.default_rng(42)
n = 1000
df = pd.DataFrame({
    "tenure": rng.integers(1, 60, n),
    "plan": rng.choice(["basic", "plus", "pro"], n),
    "size": rng.choice(["small", "medium", "large"], n),
    "city": rng.choice([f"city_{i}" for i in range(150)], n),          # high cardinality
})
y = ((df["plan"] == "pro") | (df["tenure"] > 40)).astype(int).to_numpy()

preprocess = ColumnTransformer([
    ("num", StandardScaler(), ["tenure"]),
    ("onehot", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
    ("ordinal", Pipeline([("enc", OrdinalEncoder(categories=[["small", "medium", "large"]])),
                          ("scale", StandardScaler())]), ["size"]),
    ("target", TargetEncoder(random_state=42), ["city"]),            # cross-fitted inside fit_transform
])
model = Pipeline([("prep", preprocess), ("svc", SVC(kernel="rbf", C=1.0))])
print("CV accuracy:", round(cross_val_score(model, df, y, cv=5).mean(), 3))

model.fit(df, y)
print("encoded feature count:", model.named_steps["prep"].transform(df.head(1)).shape[1])

Scaling One-Hot Columns

One-hot columns are 0/1; numeric features are usually standardized. Mixing them is fine, but very many one-hot columns can dominate RBF distances, so consider grouping rare levels or weighting groups of features.

Common Pitfalls

  • Fitting encoders on the full dataset before splitting.

  • Unknown categories at prediction time crashing the encoder; set handle_unknown="ignore".

  • Thousands of one-hot columns with an RBF kernel; a linear SVM suits sparse, wide data better.

22

What methods can you use to tune SVM hyperparameters?

Tuning SVM hyperparameters means choosing the kernel, the regularization strength C and kernel parameters such as gamma (RBF) or degree (polynomial) to maximize cross-validated performance. The standard methods are grid search over log-spaced values, randomized search from log-uniform distributions, successive halving to discard poor settings early, and Bayesian optimization (for example Optuna). Whatever the method, wrap scaling and the SVM in a Pipeline, use a metric that matches the goal, and keep a final test set untouched.

Read full answer

Applies toscikit-learn 1.5+, SciPy, Optuna 3.x (optional) (Python 3.12)

Hyperparameters

Parameter Typical range Effect
kernel linear, rbf, poly Shape of the boundary
C 1e-3 to 1e3 (log scale) Regularization: small = wider margin
gamma (rbf, poly, sigmoid) 1e-4 to 10 (log scale), or “scale” Kernel reach: large = more local, wiggly
degree (poly) 2 to 4 Interaction order
class_weight None, “balanced” Imbalance handling

Methods Compared

Method Pros Cons
GridSearchCV Exhaustive, simple, reproducible Cost grows multiplicatively with each parameter
RandomizedSearchCV Covers wide ranges with a fixed budget May miss narrow optima
HalvingGridSearchCV / HalvingRandomSearchCV Evaluates many candidates cheaply on small subsets first Small-sample results can mislead for SVMs
Bayesian optimization (Optuna) Uses past results to choose the next trial Extra dependency

Code Example

from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = Pipeline([("scale", StandardScaler()), ("svc", SVC())])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Grid search across kernels with kernel-specific parameters
param_grid = [
    {"svc__kernel": ["linear"], "svc__C": [0.01, 0.1, 1, 10]},
    {"svc__kernel": ["rbf"], "svc__C": [0.1, 1, 10, 100], "svc__gamma": [1e-3, 1e-2, 1e-1, "scale"]},
    {"svc__kernel": ["poly"], "svc__C": [0.1, 1, 10], "svc__degree": [2, 3], "svc__coef0": [0, 1]},
]
grid = GridSearchCV(pipe, param_grid, cv=cv, scoring="roc_auc", n_jobs=-1).fit(X_train, y_train)
print("grid best:", grid.best_params_, round(grid.best_score_, 4))

# Randomized search with log-uniform distributions
rand = RandomizedSearchCV(pipe, {"svc__kernel": ["rbf"], "svc__C": loguniform(1e-2, 1e3),
                                 "svc__gamma": loguniform(1e-4, 1e0)},
                          n_iter=40, cv=cv, scoring="roc_auc", random_state=42, n_jobs=-1).fit(X_train, y_train)
print("random best:", {k: (round(v, 4) if isinstance(v, float) else v) for k, v in rand.best_params_.items()})
print("test ROC AUC of grid winner:", round(grid.score(X_test, y_test), 4))

An Optuna version samples C and gamma with trial.suggest_float(..., log=True) and returns the mean of cross_val_score as the objective.

Common Pitfalls

  • Linear grids such as [1, 2, 3, ..., 10]; SVM parameters act on a log scale.

  • Scaling outside the Pipeline, which leaks fold information.

  • Reporting the best CV score as the final performance; use the held-out test set or nested cross-validation.

23

How do you handle an imbalanced dataset when using SVM?

Class imbalance pushes an SVM’s boundary toward the minority class, because the penalty term is dominated by the many majority-class points; the result often looks accurate while missing most minority cases. The main fixes are class weights (a larger C for the minority class), resampling (oversampling the minority class or undersampling the majority), adjusting the decision threshold on the SVM’s scores, and evaluating with metrics that reflect minority-class performance, such as precision, recall, F1 and average precision.

Read full answer

Applies toscikit-learn 1.5+, imbalanced-learn 0.12+ (optional) (Python 3.12)

Techniques

Technique How Notes
Class weights SVC(class_weight="balanced") or {0: 1, 1: 10} Each class’s C is multiplied by its weight; simplest first step
Threshold tuning Choose a cut-off on decision_function using validation data Matches the business trade-off between false positives and negatives
Oversampling SMOTE / random oversampling (imbalanced-learn) Must be done inside CV folds
Undersampling Remove majority examples Also speeds up kernel SVM training
Metrics Recall, precision, F1, PR AUC; confusion matrix Accuracy is misleading
Stratified splits StratifiedKFold Keeps class ratios in each fold

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics import average_precision_score, classification_report, precision_recall_curve
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=4000, n_features=10, n_informative=5, weights=[0.95, 0.05],
                           flip_y=0.01, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.25, stratify=y_train, random_state=42)

for weight in (None, "balanced"):
    model = make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight=weight)).fit(X_tr, y_tr)
    scores = model.decision_function(X_test)
    print(f"class_weight={weight}: PR AUC {average_precision_score(y_test, scores):.3f}")
    print(classification_report(y_test, model.predict(X_test), digits=3))

# Threshold tuning on validation data: pick the threshold with the best F1
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight="balanced")).fit(X_tr, y_tr)
prec, rec, thr = precision_recall_curve(y_val, model.decision_function(X_val))
f1 = 2 * prec[:-1] * rec[:-1] / (prec[:-1] + rec[:-1] + 1e-12)
best_thr = thr[np.argmax(f1)]
y_pred = (model.decision_function(X_test) >= best_thr).astype(int)
print(f"tuned threshold {best_thr:.3f}:")
print(classification_report(y_test, y_pred, digits=3))

With imbalanced-learn, put the sampler inside its Pipeline (make_pipeline(StandardScaler(), SMOTE(random_state=42), SVC())) so synthetic samples are created only from each training fold.

Common Pitfalls

  • Oversampling before the train/test split, which copies test-like points into training.

  • Reporting accuracy: 95% is achieved by always predicting the majority class here.

  • Tuning the threshold on the test set.

  • Treating decision_function values as probabilities.

24

Which metrics are commonly used to evaluate an SVM model?

SVMs are evaluated with the same metrics as other classifiers or regressors, chosen to match the problem: accuracy, precision, recall, F1 and the confusion matrix for classification; ROC AUC and average precision (PR AUC) for ranking quality, which can be computed from the SVM’s decision_function scores without probabilities; and MAE, RMSE and R² for Support Vector Regression. SVM-specific diagnostics, such as the number of support vectors and the margin, help explain model complexity and prediction cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Classification Metrics

Metric Meaning When to use
Accuracy Share of correct predictions Balanced classes, equal error costs
Precision Of predicted positives, how many are correct False positives are costly (spam filter)
Recall (sensitivity) Of actual positives, how many are found False negatives are costly (disease screening)
F1 Harmonic mean of precision and recall Imbalanced classes, single summary
ROC AUC Probability a random positive scores above a random negative Threshold-independent ranking
Average precision (PR AUC) Area under precision-recall curve Rare positive class
Confusion matrix Counts of each error type Always inspect
Log loss / Brier score Probability quality Only with calibrated probabilities

Regression (SVR) Metrics

MAE (average absolute error), RMSE (penalizes large errors), R² (variance explained), and the share of predictions within epsilon of the target.

SVM-Specific Diagnostics

  • len(model.support_): many support vectors mean slower predictions and possibly overlapping classes.

  • Margin width 2 / ||w|| for linear models.

  • Training vs validation score gap for overfitting.

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import (ConfusionMatrixDisplay, average_precision_score, classification_report,
                             confusion_matrix, roc_auc_score)
from sklearn.model_selection import cross_validate, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0))

cv = cross_validate(model, X_train, y_train, cv=5,
                    scoring=["accuracy", "precision", "recall", "f1", "roc_auc"])
for name in ("accuracy", "precision", "recall", "f1", "roc_auc"):
    print(f"CV {name:9s}: {cv['test_' + name].mean():.3f} ± {cv['test_' + name].std():.3f}")

model.fit(X_train, y_train)
scores = model.decision_function(X_test)            # no probability=True needed for AUC metrics
print("test ROC AUC:", round(roc_auc_score(y_test, scores), 3))
print("test PR AUC:", round(average_precision_score(y_test, scores), 3))
print(confusion_matrix(y_test, model.predict(X_test)))
print(classification_report(y_test, model.predict(X_test), digits=3))
print("support vectors:", len(model.named_steps["svc"].support_))

Common Pitfalls

  • Using accuracy alone on imbalanced data.

  • Computing log loss on uncalibrated decision_function outputs.

  • Calling predict_proba with probability=True and expecting it to agree exactly with predict; the Platt model can disagree near the boundary.

25

What is the trade-off between model complexity and generalization in SVM?

In an SVM, model complexity is controlled mainly by C and the kernel parameters. A more complex model (large C, large RBF gamma, high polynomial degree) fits the training data more closely, reducing bias but increasing variance, so it may generalize worse; a simpler model (small C, small gamma, linear kernel) has a wider margin and smoother boundary, which generalizes better until it starts to underfit. The goal is the setting with the best validation performance, which cross-validation finds.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Complexity Knobs

Knob Simpler (more regularized) More complex
C Small: wider margin, more violations Large: narrow margin, fits noise
gamma (RBF) Small: smooth, near-linear boundary Large: local “islands” around points
degree (poly) 2 5+
Kernel Linear RBF, high-degree poly
Features Fewer, selected Many raw or engineered features

Why the Margin Helps Generalization

Statistical learning theory bounds the generalization error of margin classifiers in terms of the margin relative to the data’s spread, not the raw number of features. A wide margin means low effective capacity even in very high dimensions, which is why SVMs work well on text. The fraction of support vectors also bounds the leave-one-out error: very many support vectors can signal a poor fit.

Code Example: Train vs Validation Across Gamma and C

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=500, noise=0.3, random_state=42)

print(f"{'C':>6} {'gamma':>6} {'train':>6} {'val':>6} {'gap':>6} {'#SV':>5}")
for C in (0.1, 1, 100):
    for gamma in (0.1, 1, 30):
        model = make_pipeline(StandardScaler(), SVC(C=C, gamma=gamma))
        res = cross_validate(model, X, y, cv=5, return_train_score=True)
        tr, va = res["train_score"].mean(), res["test_score"].mean()
        n_sv = len(model.fit(X, y).named_steps["svc"].support_)
        print(f"{C:>6} {gamma:>6} {tr:6.3f} {va:6.3f} {tr - va:6.3f} {n_sv:5d}")

Look for low-gamma, low-C rows with low scores on both sets (underfitting) and high-gamma, high-C rows with near-perfect training scores but a large gap (overfitting); the best rows sit in between.

Practical Guidance

  • Start simple (linear, or RBF with default gamma) and add complexity only if validation scores improve.

  • Use learning curves: if both curves plateau low, add complexity; if a big gap persists, regularize or add data.

  • Use nested cross-validation when you need an unbiased estimate after tuning.

Common Pitfalls

  • Judging complexity by training accuracy.

  • Tuning C and gamma separately, missing their interaction.

26

How does feature scaling affect SVM performance?

Feature scaling is essential for SVMs because both the margin and most kernels depend on distances or dot products between feature vectors. If one feature ranges in thousands and another between 0 and 1, the large feature dominates the distance, the margin is effectively measured along it alone, and the RBF kernel’s gamma becomes meaningless. Scaling also improves the numerical conditioning of the optimization, so training converges faster.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Effects of Not Scaling

Aspect Effect
Linear SVM Weights and margin biased toward large-scale features; regularization penalizes features unequally
RBF kernel Distances dominated by one feature; gamma either flattens or explodes the kernel
Polynomial kernel Large values raised to powers cause numerical overflow and instability
Training time Poor conditioning: many more iterations, convergence warnings
Interpretation coef_ values not comparable across features

Scalers

Scaler Use when
StandardScaler Default: mean 0, standard deviation 1
MinMaxScaler Bounded features; required range [0, 1]
RobustScaler Outliers present (uses median and IQR)
MaxAbsScaler Sparse data such as TF-IDF (keeps sparsity)
Normalizer Per-sample unit length, e.g. for cosine-like similarity

Code Example

import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler
from sklearn.svm import SVC

X, y = load_wine(return_X_y=True)
print("feature ranges:", np.ptp(X, axis=0).round(1))      # from about 0.5 to over 1000

configs = {
    "no scaling": SVC(kernel="rbf"),
    "StandardScaler": make_pipeline(StandardScaler(), SVC(kernel="rbf")),
    "MinMaxScaler": make_pipeline(MinMaxScaler(), SVC(kernel="rbf")),
    "RobustScaler": make_pipeline(RobustScaler(), SVC(kernel="rbf")),
}
for name, model in configs.items():
    print(f"{name:15s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

The wine dataset’s proline feature is in the hundreds to over a thousand while others are below 10, so the unscaled model should score clearly worse.

Common Pitfalls

  • Fitting the scaler on all data before cross-validation; keep it in the Pipeline.

  • Scaling one-hot columns unnecessarily (harmless with StandardScaler but changes their relative weight).

  • Forgetting to apply the same fitted scaler at prediction time.

27

How can you speed up SVM training on large datasets?

Kernel SVM training scales roughly between O(n^2) and O(n^3), so on large datasets the main speed-ups come from changing the problem: use a linear SVM (LinearSVC or SGDClassifier with hinge loss), approximate the kernel with random features or the Nyström method and train a linear SVM on top, reduce the data (subsampling, removing redundant points) or move to GPU implementations. Within SVC, a bigger kernel cache, a looser tolerance and well-scaled features also help.

Read full answer

Applies toscikit-learn 1.5+, optional cuML (GPU) (Python 3.12)

Options

Technique How Speed-up source
Linear SVM LinearSVC, SGDClassifier(loss="hinge") Linear-time solvers; often enough for high-dimensional data
Kernel approximation Nystroem(n_components=500) or RBFSampler + LinearSVC Explicit low-dimensional features approximate the RBF kernel
Subsampling for tuning Tune C/gamma on 10-20% of data, refit once on more Fewer expensive fits
Successive halving HalvingRandomSearchCV Discards poor settings early
Cache and tolerance SVC(cache_size=2000, tol=1e-2) Fewer kernel recomputations, earlier stop
Scaling StandardScaler Better conditioning: faster convergence
Parallelism n_jobs=-1 in search / OvR wrappers Uses all cores (SVC itself is single-threaded)
Feature reduction PCA, feature selection Cheaper kernel evaluations
GPU cuML SVC Large constant-factor speed-ups
Online learning SGDClassifier.partial_fit on chunks Data larger than memory

Code Example

import time

from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = make_classification(n_samples=30000, n_features=40, n_informative=15, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    "SVC rbf (exact)": make_pipeline(StandardScaler(), SVC(kernel="rbf", cache_size=1000)),
    "Nystroem(500) + LinearSVC": make_pipeline(StandardScaler(), Nystroem(gamma=0.02, n_components=500,
                                                                         random_state=42), LinearSVC()),
    "LinearSVC": make_pipeline(StandardScaler(), LinearSVC()),
    "SGD hinge": make_pipeline(StandardScaler(), SGDClassifier(loss="hinge", random_state=42)),
}
for name, model in models.items():
    start = time.perf_counter()
    model.fit(X_train, y_train)
    elapsed = time.perf_counter() - start
    print(f"{name:28s} fit {elapsed:6.2f}s  test accuracy {model.score(X_test, y_test):.3f}")

Expect the exact SVC to be the slowest by a wide margin; compare whether the approximations lose any accuracy on your data.

Common Pitfalls

  • Using SVC(kernel="linear") on large data instead of LinearSVC.

  • probability=True, which adds an internal 5-fold cross-validation.

  • Forgetting that prediction time grows with the number of support vectors too.

28

How do you diagnose and fix underfitting or overfitting in an SVM?

Diagnosing an SVM’s fit starts with comparing training and validation scores: low scores on both indicate underfitting (too simple), while a high training score with a much lower validation score indicates overfitting (too complex). Then adjust the complexity controls (C, gamma, kernel, degree), the features and the amount of data. Validation curves, learning curves and the number of support vectors make the diagnosis concrete.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Symptoms and Fixes

Problem Symptoms Fixes
Underfitting Train and validation scores both low and close Increase C; increase gamma; switch from linear to RBF/poly; add features; check scaling
Overfitting Train near perfect, validation much lower Decrease C; decrease gamma; lower degree; remove noisy features; more data; check for leakage
Many support vectors (> ~50% of data) Heavy class overlap or too small C Better features; tune C
Unstable results across folds High variance More data, simpler model, repeated CV

Code Example: Validation and Learning Curves

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=1500, n_features=20, n_informative=6, flip_y=0.08, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0))

gammas = np.logspace(-4, 1, 6)
train, val = validation_curve(pipe, X, y, param_name="svc__gamma", param_range=gammas, cv=5)
for g, tr, va in zip(gammas, train.mean(axis=1), val.mean(axis=1)):
    label = "underfit" if tr < 0.8 else ("overfit" if tr - va > 0.1 else "ok")
    print(f"gamma={g:<8.4f} train {tr:.3f} val {va:.3f}  -> {label}")

sizes, tr_lc, va_lc = learning_curve(make_pipeline(StandardScaler(), SVC(gamma=0.05)), X, y, cv=5,
                                     train_sizes=np.linspace(0.1, 1.0, 5), random_state=42)
for n, tr, va in zip(sizes, tr_lc.mean(axis=1), va_lc.mean(axis=1)):
    print(f"n={n:5d} train {tr:.3f} val {va:.3f}")

The labels in the first loop use simple illustrative thresholds; in practice judge from the curves. In the learning curve, a validation score still rising at full size means more data would help.

Diagnostic Checklist

  1. Confirm features are scaled inside the Pipeline.

  2. Check for leakage if validation scores look too good.

  3. Tune C and gamma jointly with cross-validation.

  4. Inspect misclassified examples for label noise or missing features.

Common Pitfalls

  • Fixing overfitting by lowering C alone when gamma is the real cause, or the reverse.

  • Drawing conclusions from a single train/validation split on small data.

29

What should you consider when deploying an SVM model in production?

Deploying an SVM involves the same concerns as any model (reproducible preprocessing, versioning, monitoring) plus some SVM-specific ones: prediction latency and memory grow with the number of support vectors for kernel models, scaling and encoding must be identical to training, probabilities need calibration if decisions depend on them, and models must be retrained because standard SVMs cannot update incrementally. Packaging the whole scikit-learn Pipeline as one artifact handles most preprocessing risks.

Read full answer

Applies toscikit-learn 1.5+, joblib, skops (optional) (Python 3.12)

Checklist

Area Consideration
Preprocessing Save scaler, encoders and SVM together in one Pipeline
Latency Kernel prediction cost ∝ number of support vectors × features; benchmark at peak load
Memory Support vectors are stored in the model; large SV sets mean large files
Probabilities Use CalibratedClassifierCV if scores drive decisions; don’t treat decision_function as probability
Thresholds Store the chosen decision threshold with the model version
Serialization joblib or skops.io; pin library versions (pickles are not portable across scikit-learn versions)
Security Never load pickles from untrusted sources; skops offers safer loading
Monitoring Input drift, score distribution, delayed-label accuracy
Retraining Scheduled or drift-triggered; batch retraining on recent data
Interpretability Linear weights or SHAP explanations for audits
Fallback Default behaviour on invalid input or service errors

Code Example: Package, Save, Load and Serve

import time

import joblib
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

pipeline = make_pipeline(StandardScaler(),
                         CalibratedClassifierCV(SVC(kernel="rbf", C=1.0), method="sigmoid", cv=5))
pipeline.fit(X_train, y_train)

artifact = {"model": pipeline, "threshold": 0.5, "version": "2026-11-10",
            "feature_count": X.shape[1], "sklearn": __import__("sklearn").__version__}
joblib.dump(artifact, "svm_model.joblib")

loaded = joblib.load("svm_model.joblib")            # load once at service start-up


def predict(features):
    x = np.asarray(features, dtype=float).reshape(1, -1)
    if x.shape[1] != loaded["feature_count"] or not np.isfinite(x).all():
        raise ValueError("invalid input")
    proba = loaded["model"].predict_proba(x)[0, 1]
    return {"probability": round(float(proba), 4), "label": int(proba >= loaded["threshold"]),
            "model_version": loaded["version"]}


start = time.perf_counter()
for row in X_test[:200]:
    predict(row)
print(f"mean latency {(time.perf_counter() - start) / 200 * 1000:.2f} ms")
print(predict(X_test[0]))

Common Pitfalls

  • Saving only the SVM without its scaler.

  • Loading a pickle with a different scikit-learn version.

  • No monitoring, so slow data drift goes unnoticed.

30

How does SVM handle incremental or online learning?

Standard kernel SVM solvers are batch algorithms: scikit-learn’s SVC must retrain on all data when new examples arrive. For incremental or online learning, the practical options are a linear SVM trained with stochastic gradient descent (SGDClassifier(loss="hinge"), which supports partial_fit), the same model on top of a fixed kernel approximation for non-linear boundaries, or specialized online kernel SVM algorithms such as LASVM and budgeted SVMs. Another approach is to retrain periodically, keeping only the support vectors plus new data.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Options

Approach How it works Trade-offs
SGDClassifier(loss="hinge").partial_fit Updates weights on each mini-batch with the hinge subgradient Linear only; fast; handles streams and data larger than memory
RBFSampler / Nystroem + SGD Fixed random/landmark features, then online linear SVM Feature map must be fitted on an initial sample and kept fixed
Support-vector retraining Keep the old SVs, add new data, retrain SVC Approximate; SV set can grow
Online kernel SVMs (LASVM, budgeted SGD) Add/remove SVs incrementally under a budget Not in scikit-learn
Periodic batch retraining Retrain on a sliding window Simple; handles drift

Code Example: Streaming Linear SVM

import numpy as np
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import RBFSampler
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=50000, n_features=20, n_informative=8, random_state=42)
X_stream, y_stream = X[:45000], y[:45000]
X_test, y_test = X[45000:], y[45000:]
classes = np.unique(y)

# Fit preprocessing on an initial chunk, then keep it fixed
scaler = StandardScaler().fit(X_stream[:2000])
rbf = RBFSampler(gamma=0.05, n_components=500, random_state=42).fit(scaler.transform(X_stream[:2000]))

linear = SGDClassifier(loss="hinge", alpha=1e-4, random_state=42)
kernelized = SGDClassifier(loss="hinge", alpha=1e-5, random_state=42)

for start in range(0, len(X_stream), 5000):                     # chunks arriving over time
    xb = scaler.transform(X_stream[start:start + 5000])
    yb = y_stream[start:start + 5000]
    linear.partial_fit(xb, yb, classes=classes)                 # classes required on the first call
    kernelized.partial_fit(rbf.transform(xb), yb, classes=classes)
    Xt = scaler.transform(X_test)
    print(f"after {start + 5000:5d} samples: linear {linear.score(Xt, y_test):.3f}, "
          f"RBF features {kernelized.score(rbf.transform(Xt), y_test):.3f}")

Concept Drift

For streams whose distribution changes, monitor accuracy on recent labelled data, use a constant or adaptive learning rate so old data is gradually forgotten, or retrain on a sliding window.

Common Pitfalls

  • Refitting the scaler on each chunk, which changes the feature space under the model.

  • Forgetting classes= on the first partial_fit call.

  • Expecting SVC to have partial_fit; it does not.

31

What are the challenges of training SVMs in distributed computing environments?

Distributing SVM training is hard because the kernel SVM’s optimization couples every pair of training points through the kernel matrix, so the problem does not split cleanly across machines the way gradient descent on independent mini-batches does. The main challenges are communication cost, memory for kernel values, the global equality constraint in the dual, and load imbalance, since the number and location of support vectors are unknown in advance. Practical systems therefore use linear SVMs with distributed SGD, kernel approximations, or divide-and-conquer schemes that train local SVMs and merge their support vectors.

Read full answer

Applies toscikit-learn 1.5+, Spark MLlib 3.x, Dask-ML (Python 3.12)

Challenges

Challenge Why it happens
Pairwise coupling Every alpha_i interacts with every other through K(x_i, x_j)
Kernel matrix size n × n values; 1 million samples means 10^12 entries
Global constraint sum alpha_i y_i = 0 ties all partitions together
Communication Solvers such as SMO update two variables at a time: too fine-grained to distribute
Load imbalance Support vectors concentrate in some partitions
Hyperparameter search Each (C, gamma) setting needs a full expensive training
Non-identical partitions Local models trained on skewed shards disagree

Approaches

Approach Idea Tools
Distributed linear SVM Data-parallel gradient or L-BFGS updates of w Spark MLlib LinearSVC, Vowpal Wabbit
Kernel approximation + linear SVM Random Fourier features / Nyström make the problem linear scikit-learn on Dask, Spark
Cascade SVM Train SVMs on partitions, pass their support vectors up a tree, retrain Custom
Divide-and-conquer kernel SVM Cluster the data, train local SVMs, use them to warm-start a global one Research implementations
Parallel hyperparameter search Distribute independent fits across workers GridSearchCV(n_jobs=-1) with joblib/Dask/Ray backends
GPU acceleration Massively parallel kernel computation on one node cuML SVC

Code Example: Cascade-Style Approximation

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=20000, n_features=20, n_informative=8, flip_y=0.03, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

# Layer 1: each "worker" trains on its own shard (run in parallel in a real cluster)
shards = np.array_split(np.random.default_rng(42).permutation(len(X_train)), 4)
sv_idx = []
for shard in shards:
    local = SVC(kernel="rbf", C=1.0).fit(X_train[shard], y_train[shard])
    sv_idx.extend(shard[local.support_])               # keep only support vectors

# Layer 2: train on the union of support vectors
sv_idx = np.array(sv_idx)
final = SVC(kernel="rbf", C=1.0).fit(X_train[sv_idx], y_train[sv_idx])
print(f"final model trained on {len(sv_idx)} of {len(X_train)} points")
print("cascade accuracy:", round(final.score(X_test, y_test), 3))

With noisy data, many points are support vectors, so the cascade’s second stage can still be large; that is the method’s main limitation.

Spark MLlib

Spark’s pyspark.ml.classification.LinearSVC trains a linear SVM with OWL-QN over partitioned data; there is no distributed kernel SVM in MLlib, which reflects the difficulty described above.

Common Pitfalls

  • Expecting linear speed-ups from more machines for exact kernel SVMs.

  • Scaling each shard separately, which gives inconsistent feature spaces.

  • Ignoring that a gradient-boosted tree or neural network may suit very large data better.

32

How can you reduce SVM model storage and inference time?

A kernel SVM stores every support vector and evaluates the kernel against each of them for every prediction, so its size and latency grow with the number of support vectors. To shrink both, reduce the number of support vectors (tune C and gamma, clean noisy labels, reduce the training set), reduce features, or replace the kernel model with a cheaper equivalent: a linear SVM, an explicit kernel approximation, or a compact model distilled from the SVM. Batching predictions and lower-precision storage add further gains.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Techniques

Technique Effect on storage Effect on latency
Linear kernel (LinearSVC) Stores only w and b One dot product per prediction
Tune C / gamma for fewer SVs Fewer stored vectors Proportionally faster
Remove label noise / duplicates Fewer margin violators Faster
Feature selection or PCA Smaller vectors Cheaper kernel evaluations
Nyström / random features + linear model Fixed size (k components) O(k × d) per prediction
Reduced-set methods / distillation Train a small model on the SVM’s outputs Much faster
float32 storage Half the memory of float64 Slightly faster
Batch predictions — Vectorized kernel computation
Compression of the saved file joblib.dump(..., compress=3) Load time only

Code Example

import os
import time

import joblib
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = make_classification(n_samples=12000, n_features=30, n_informative=10, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    "SVC C=10": make_pipeline(StandardScaler(), SVC(C=10)),
    "SVC C=0.5": make_pipeline(StandardScaler(), SVC(C=0.5)),
    "Nystroem(300)+LinearSVC": make_pipeline(StandardScaler(), Nystroem(n_components=300, random_state=42),
                                             LinearSVC()),
    "LinearSVC": make_pipeline(StandardScaler(), LinearSVC()),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    path = f"{name.split('(')[0].replace(' ', '_').replace('=', '')}.joblib"
    joblib.dump(model, path, compress=3)
    start = time.perf_counter()
    acc = model.score(X_test, y_test)
    latency_ms = (time.perf_counter() - start) / len(X_test) * 1000
    n_sv = len(model[-1].support_) if hasattr(model[-1], "support_") else "-"
    print(f"{name:25s} acc {acc:.3f}  SVs {n_sv!s:>5}  size {os.path.getsize(path) / 1024:8.1f} KB  "
          f"{latency_ms:.4f} ms/sample")

Compare accuracy against size and latency to choose the cheapest model that meets the accuracy target.

Common Pitfalls

  • Reducing C so much that accuracy drops; measure both.

  • Benchmarking single predictions in a loop when the service could batch them.

  • Forgetting the scaler and encoders in size and latency measurements.

33

What is Support Vector Regression, and where is it used?

Support Vector Regression (SVR) adapts the SVM idea to predicting continuous values. Instead of a margin between classes, it fits a function with an epsilon-insensitive tube around it: errors smaller than epsilon are ignored, and only points outside the tube (the support vectors) contribute to the loss, while the model is kept as flat (regularized) as possible. With kernels, SVR fits non-linear relationships, and it is used for small to medium datasets in areas such as forecasting, engineering and chemistry.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Objective

Minimize (1/2)||w||^2 + C * sum(xi_i + xi_i*) subject to each prediction being within epsilon + slack of its target.

In words: keep the function flat, ignore errors up to epsilon, and pay a linear penalty, weighted by C, for anything beyond.

Parameters

Parameter Effect
epsilon Tube width: larger = fewer support vectors, smoother fit, ignores more error
C Penalty on errors outside the tube: larger = fits data more closely
kernel, gamma Shape and flexibility of the function

Epsilon is in the units of the target, so scale the target or set epsilon relative to its noise level.

Applications

Domain Example
Finance Volatility or price-change estimation on engineered features
Energy Short-term load or solar output forecasting
Engineering Surrogate models for expensive simulations
Chemistry / QSAR Predicting molecular properties
Healthcare Dose-response or biomarker estimation

Code Example

import numpy as np
from sklearn.compose import TransformedTargetRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR

X, y = fetch_california_housing(return_X_y=True)           # downloads the data
X, y = X[:5000], y[:5000]                                   # subset: kernel SVR is slow on large n
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

svr = TransformedTargetRegressor(                            # scales y so epsilon is in standard units
    regressor=make_pipeline(StandardScaler(), SVR(kernel="rbf")),
    transformer=StandardScaler())
grid = GridSearchCV(svr, {"regressor__svr__C": [1, 10, 100],
                          "regressor__svr__epsilon": [0.05, 0.1, 0.3],
                          "regressor__svr__gamma": ["scale", 0.1]},
                    cv=5, scoring="neg_mean_absolute_error", n_jobs=-1)
grid.fit(X_train, y_train)
pred = grid.predict(X_test)
print("best:", grid.best_params_)
print(f"test MAE {mean_absolute_error(y_test, pred):.3f}, R2 {r2_score(y_test, pred):.3f}")
print("support vectors:", len(grid.best_estimator_.regressor_[-1].support_))

Common Pitfalls

  • Leaving the target unscaled, so a default epsilon of 0.1 is meaningless.

  • Large datasets: use LinearSVR or kernel approximations.

  • Expecting prediction intervals; SVR gives point predictions.

34

What is the linear kernel in SVM, and when should you use it?

The linear kernel is simply the dot product, K(x, z) = x·z, so the SVM learns a straight hyperplane w·x + b = 0 in the original feature space with no implicit mapping. It is the fastest and most interpretable SVM: training scales almost linearly with data size using specialized solvers, the model is just a weight per feature, and it is often as accurate as non-linear kernels on high-dimensional data such as text, where classes are usually linearly separable.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

When to Use It

Situation Why linear works well
Many features relative to samples (text, genomics) High-dimensional data is often linearly separable; non-linear kernels add overfitting risk
Large datasets (100k+ samples) LinearSVC / SGDClassifier scale to millions of rows
Need for interpretability Each weight shows a feature’s direction and strength
Low latency Prediction is one dot product
Baseline Always worth trying before RBF

Use a non-linear kernel when there are few features, many samples relative to features, and evidence that the boundary is curved.

Implementations

Class Solver Notes
LinearSVC liblinear Fast; squared hinge by default; supports L1 penalty
SGDClassifier(loss="hinge") SGD Streaming, partial_fit, huge data
SVC(kernel="linear") libsvm Slow for large n; gives support vectors and dual coefficients

Code Example: Text Classification

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC

cats = ["sci.space", "rec.autos", "comp.graphics"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))

model = make_pipeline(TfidfVectorizer(sublinear_tf=True, min_df=2, ngram_range=(1, 2)), LinearSVC(C=0.5))
model.fit(train.data, train.target)
print(classification_report(test.target, model.predict(test.data), target_names=train.target_names))

# Interpret: top weighted terms per class
vocab = np.array(model[0].get_feature_names_out())
for i, name in enumerate(train.target_names):
    print(name, "->", ", ".join(vocab[np.argsort(model[-1].coef_[i])[-8:]]))

The data is downloaded on first use. Tens of thousands of TF-IDF features train in seconds.

Common Pitfalls

  • Reading weights as importance without scaling numeric features first.

  • Using SVC(kernel="linear") for large data instead of LinearSVC.

  • Convergence warnings in LinearSVC; scale features or raise max_iter.

35

What is the RBF kernel, and how does it transform the feature space?

The Radial Basis Function (RBF), or Gaussian, kernel measures similarity between two points as K(x, z) = exp(-gamma * ||x - z||^2): 1 when the points coincide and decaying toward 0 as they move apart. It corresponds to an implicit mapping into an infinite-dimensional feature space, where the SVM finds a linear boundary that appears as a flexible, curved boundary in the original space. In effect, the decision function becomes a weighted sum of Gaussian “bumps” centred on the support vectors.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

How It Transforms the Space

  • The squared distance in the exponent means similarity depends only on how far apart points are (it is “radial”).

  • Expanding exp(-gamma ||x - z||^2) as a Taylor series gives polynomial terms of every degree, so the implicit feature map has infinitely many dimensions.

  • All points are mapped onto a sphere of radius 1 in that space (K(x, x) = 1).

  • Decision function: f(x) = sum_i alpha_i y_i exp(-gamma ||x_i - x||^2) + b, so each support vector contributes a bump whose width is set by gamma.

Parameters

Parameter Role Too small Too large
gamma (= 1 / (2 sigma^2)) Width of each bump Very smooth, near-linear: underfit Bumps around single points: overfit
C Penalty for violations Wide margin, underfit Fits noise

Code Example

import numpy as np
from sklearn.datasets import make_circles
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

x = np.array([[0.0, 0.0]])
points = np.array([[0.0, 0.0], [0.5, 0.0], [1.0, 0.0], [2.0, 0.0]])
for gamma in (0.5, 2.0):
    print(f"gamma={gamma}: similarities {rbf_kernel(x, points, gamma=gamma).round(3)}")

# Manual decision function equals sklearn's
X, y = make_circles(n_samples=300, factor=0.5, noise=0.1, random_state=42)
svc = SVC(kernel="rbf", gamma=1.0, C=1.0).fit(X, y)
K = rbf_kernel(X[:5], svc.support_vectors_, gamma=1.0)
manual = K @ svc.dual_coef_[0] + svc.intercept_[0]
print("matches decision_function:", np.allclose(manual, svc.decision_function(X[:5])))

for kernel in ("linear", "rbf"):
    print(kernel, round(cross_val_score(SVC(kernel=kernel), X, y, cv=5).mean(), 3))

The concentric circles cannot be separated by a line, but the RBF SVM separates them easily.

When to Use It

  • Default non-linear kernel for low- to medium-dimensional, dense numeric data.

  • Less useful for very high-dimensional sparse text, where linear kernels are as accurate and faster.

Common Pitfalls

  • Unscaled features, which make distances (and gamma) dominated by one feature.

  • Tuning gamma without C.

  • Large datasets: exact RBF SVMs are slow; use Nystroem approximations.

36

When would you use a polynomial kernel in SVM?

The polynomial kernel, K(x, z) = (gamma * x·z + coef0)^degree, lets an SVM learn boundaries built from interactions between features up to a chosen degree, as if all products of features up to that degree had been added. Use it when you expect feature interactions of a known, low order to matter, for example pairwise interactions (degree 2), and when the data has been normalized. In practice the RBF kernel is often more accurate and easier to tune, so the polynomial kernel is chosen for specific structure, not as a default.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Parameters

Parameter Role Notes
degree Highest interaction order 2-3 typical; higher degrees overfit and become unstable
coef0 Weight of lower-order terms coef0=0 keeps only terms of exactly that degree; coef0=1 includes all lower degrees
gamma Scales the dot product Interacts with degree and C
C Regularization Tune jointly

Good Use Cases

Situation Reason
Known pairwise interactions (e.g. price × quantity) Degree 2 captures them directly
Normalized image features (e.g. digit pixels) Degree 2-4 polynomial SVMs historically worked well on MNIST
NLP with bag-of-words and feature conjunctions Degree 2 captures word pairs
When you want an alternative to explicit PolynomialFeatures in high dimensions Avoids building the huge feature matrix

Code Example

import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

rng = np.random.default_rng(42)
X = rng.normal(size=(800, 4))
y = (X[:, 0] * X[:, 1] + 0.5 * X[:, 2] ** 2 > 0.5).astype(int)    # degree-2 structure

for kernel, params in [("linear", {}), ("poly", {"degree": 2, "coef0": 1}), ("poly", {"degree": 5, "coef0": 1}),
                       ("rbf", {})]:
    grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel=kernel, **params)),
                        {"svc__C": [0.1, 1, 10]}, cv=5)
    grid.fit(X, y)
    label = f"{kernel} {params.get('degree', '')}".strip()
    print(f"{label:8s} CV accuracy {grid.best_score_:.3f}")

Because the target is built from degree-2 terms, the degree-2 polynomial kernel should do well here; the linear kernel cannot represent the product x0 * x1.

Common Pitfalls

  • High degrees on unscaled data: values explode and training becomes slow or unstable.

  • Forgetting coef0: with the default coef0=0, only terms of exactly that degree are used.

  • Tuning degree, gamma, coef0 and C together without enough data for reliable cross-validation.

37

What is the purpose of the sigmoid kernel in SVM?

The sigmoid kernel, K(x, z) = tanh(gamma * x·z + coef0), was proposed because an SVM using it has the same form as a two-layer neural network with tanh hidden units, one per support vector. In practice it is rarely the best choice: it is not positive semi-definite for many parameter values, so it is not always a valid kernel and the optimization can behave poorly, and RBF or linear kernels usually perform as well or better. It is mainly of historical and theoretical interest.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Properties

Aspect Sigmoid kernel
Formula tanh(gamma * x·z + coef0)
Neural-network link Decision function = weighted sum of tanh units over support vectors
Validity PSD only for some (gamma, coef0) combinations, e.g. small gamma and negative coef0
Typical performance Often below RBF; sensitive to parameters
Parameters gamma (> 0), coef0 (often ≤ 0), C

When It Might Be Tried

  • As one candidate in a kernel search when the data was preprocessed to small values.

  • When teaching the connection between SVMs and neural networks.

  • In legacy systems or papers that used it.

Code Example: Checking Validity and Performance

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics.pairwise import sigmoid_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
Xs = StandardScaler().fit_transform(X)[:200]

for gamma, coef0 in [(0.01, -1.0), (0.01, 1.0), (1.0, 0.0)]:
    K = sigmoid_kernel(Xs, gamma=gamma, coef0=coef0)
    min_eig = np.linalg.eigvalsh(K).min()
    print(f"gamma={gamma}, coef0={coef0}: smallest eigenvalue {min_eig:.3f} "
          f"({'PSD' if min_eig > -1e-8 else 'not PSD'})")

for kernel, params in [("sigmoid", {"gamma": 0.01, "coef0": -1.0}), ("sigmoid", {}), ("rbf", {}), ("linear", {})]:
    acc = cross_val_score(make_pipeline(StandardScaler(), SVC(kernel=kernel, **params)), X, y, cv=5).mean()
    print(f"{kernel:8s} {params!s:32s} CV accuracy {acc:.3f}")

A negative smallest eigenvalue means the kernel matrix is not positive semi-definite for that data and parameter setting.

Interview Tips

  • Mention the neural-network analogy and the PSD problem.

  • Recommend RBF or linear in practice, and a real neural network if that kind of model is wanted.

38

How do you create a custom SVM kernel, and what should you consider?

A custom kernel lets an SVM use a similarity measure designed for your data, such as a string kernel for DNA sequences, a histogram kernel for image descriptors, or a combination of kernels for different feature groups. In scikit-learn you either pass a Python function that returns the kernel (Gram) matrix between two sets of samples, or set kernel="precomputed" and supply the matrix yourself. The key requirement is that the kernel is positive semi-definite (PSD), so it corresponds to an inner product in some feature space and the optimization stays convex.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Ways to Build Valid Kernels

Valid kernels stay valid under these operations: - Sum: K1 + K2 - Positive scaling: c * K with c > 0 - Product: K1 * K2 - Exponential of a valid kernel - Applying a valid kernel to a fixed feature transform: K(phi(x), phi(z))

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics.pairwise import linear_kernel, rbf_kernel
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=500, n_features=10, n_informative=6, random_state=42)
X = StandardScaler().fit_transform(X)   # for brevity; in real work, fit the scaler inside each fold


# 1. Callable kernel: RBF on the first 5 features + linear on the rest
def mixed_kernel(A, B):
    return rbf_kernel(A[:, :5], B[:, :5], gamma=0.2) + 0.5 * linear_kernel(A[:, 5:], B[:, 5:])

print("custom kernel CV:", round(cross_val_score(SVC(kernel=mixed_kernel), X, y, cv=5).mean(), 3))


# 2. Precomputed kernel: you build train and test Gram matrices yourself
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
K_train = mixed_kernel(X_train, X_train)            # (n_train, n_train)
K_test = mixed_kernel(X_test, X_train)              # (n_test, n_train): rows = test, columns = train
svc = SVC(kernel="precomputed").fit(K_train, y_train)
print("precomputed test accuracy:", round(svc.score(K_test, y_test), 3))


# 3. Check positive semi-definiteness on a sample
eigvals = np.linalg.eigvalsh(K_train)
print("smallest eigenvalue:", round(float(eigvals.min()), 6), "(PSD)" if eigvals.min() > -1e-8 else "(not PSD)")

Considerations

Consideration Detail
Validity (PSD) Non-PSD kernels can make the solver fail or give poor solutions; check eigenvalues
Symmetry K(x, z) must equal K(z, x)
Normalization Normalize so K(x, x) = 1 to avoid scale effects: K(x,z) / sqrt(K(x,x) K(z,z))
Computational cost Callable kernels run in Python; precompute for repeated fits
Hyperparameters Kernel parameters need tuning like gamma
Precomputed shapes Test matrix is test × train, built against the training samples
Leakage Kernel parameters learned from data (e.g. bandwidth) must come from training data only

Common Pitfalls

  • Passing a precomputed test matrix of shape test × test.

  • Using similarity measures that are not PSD (e.g. some edit-distance transforms) without correction.

  • Very slow Python kernels on large datasets.

39

Implement a basic linear SVM from scratch in Python.

A linear SVM from scratch can be trained by minimizing the regularized hinge loss with (sub)gradient descent: the objective is (lambda/2)||w||^2 + mean(max(0, 1 - y_i (w·x_i + b))). For each point that violates the margin (y_i f(x_i) < 1), the gradient pushes w toward y_i x_i; for all points, the regularization term shrinks w. The implementation below uses NumPy, labels in {-1, +1}, mini-batches and the Pegasos-style decreasing learning rate, then checks the result against scikit-learn.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (for data and comparison) (Python 3.12)

Gradient in Words

  • Regularization term: gradient lambda * w.

  • Hinge term for a violating point: gradient -y_i x_i for w and -y_i for b; zero for points outside the margin.

  • Update: w ← w - eta * (lambda * w - mean over violators of y_i x_i).

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC


class LinearSVM:
    def __init__(self, lam=0.01, epochs=50, batch_size=32, random_state=42):
        self.lam, self.epochs, self.batch_size = lam, epochs, batch_size
        self.rng = np.random.default_rng(random_state)

    def fit(self, X, y):
        y = np.where(y > 0, 1.0, -1.0)                      # labels as -1/+1
        n, d = X.shape
        self.w, self.b = np.zeros(d), 0.0
        t = 0
        for _ in range(self.epochs):
            for batch in np.array_split(self.rng.permutation(n), max(1, n // self.batch_size)):
                t += 1
                eta = 1.0 / (self.lam * t)                  # Pegasos learning-rate schedule
                Xb, yb = X[batch], y[batch]
                violators = yb * (Xb @ self.w + self.b) < 1
                grad_w = self.lam * self.w - (yb[violators, None] * Xb[violators]).sum(axis=0) / len(batch)
                grad_b = -yb[violators].sum() / len(batch)
                self.w -= eta * grad_w
                self.b -= eta * grad_b
        return self

    def decision_function(self, X):
        return X @ self.w + self.b

    def predict(self, X):
        return (self.decision_function(X) >= 0).astype(int)

    def objective(self, X, y):
        y = np.where(y > 0, 1.0, -1.0)
        return 0.5 * self.lam * self.w @ self.w + np.maximum(0, 1 - y * self.decision_function(X)).mean()


X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

svm = LinearSVM(lam=0.01, epochs=50).fit(X_train, y_train)
print("objective:", round(svm.objective(X_train, y_train), 4))
print("from scratch accuracy:", round(accuracy_score(y_test, svm.predict(X_test)), 3))

# Equivalent scikit-learn model: C = 1 / (lambda * n_samples) with plain hinge loss
ref = LinearSVC(C=1 / (0.01 * len(X_train)), loss="hinge", max_iter=10000).fit(X_train, y_train)
print("LinearSVC accuracy:  ", round(ref.score(X_test, y_test), 3))

The two accuracies should be close; the from-scratch version regularizes the bias differently and stops after a fixed number of epochs.

Common Pitfalls

  • Using 0/1 labels in the hinge formula.

  • Not scaling features, which makes gradient descent slow and the margin distorted.

  • A constant large learning rate, which makes the weights oscillate; decay it or tune it.

40

Write a Python function to select an SVM's C parameter using cross-validation.

Selecting C with cross-validation means training the SVM for a range of C values, scoring each with k-fold cross-validation on the training data only, and picking the value with the best mean score, optionally preferring a simpler (smaller C) model when scores are within one standard error. Because C acts multiplicatively, search it on a logarithmic grid. Scaling must happen inside each fold, so the function below builds a Pipeline.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def select_C(X, y, C_values=None, kernel="rbf", cv=5, scoring="roc_auc", one_se_rule=True, random_state=42):
    """Return the chosen C and a table of cross-validated scores.

    With one_se_rule=True, picks the smallest C whose mean score is within one
    standard error of the best mean score (a simpler, more regularized model).
    """
    if C_values is None:
        C_values = np.logspace(-3, 3, 13)
    folds = StratifiedKFold(n_splits=cv, shuffle=True, random_state=random_state)
    results = []
    for C in C_values:
        model = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=C))
        scores = cross_val_score(model, X, y, cv=folds, scoring=scoring, n_jobs=-1)
        results.append((C, scores.mean(), scores.std() / np.sqrt(cv)))

    best_C, best_mean, best_se = max(results, key=lambda r: r[1])
    if one_se_rule:
        chosen = min(C for C, mean, _ in results if mean >= best_mean - best_se)
    else:
        chosen = best_C
    return chosen, results


X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

C, table = select_C(X_train, y_train)
for c, mean, se in table:
    print(f"C={c:>9.4f}  ROC AUC {mean:.4f} ± {se:.4f}")
print("chosen C:", C)

final = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C)).fit(X_train, y_train)
print("test accuracy:", round(final.score(X_test, y_test), 3))

The same search can be written as GridSearchCV(pipe, {"svc__C": C_values}, cv=folds); the hand-written version makes the one-standard-error rule explicit.

Notes

  • With an RBF kernel, gamma also matters; search C and gamma together for the final model.

  • Use stratified folds for classification so each fold keeps the class balance.

  • If the best C is at the edge of the grid, widen the grid.

Common Pitfalls

  • Using the test set inside the selection function.

  • Scaling before cross-validation, which leaks fold information.

  • Choosing by training accuracy, which always prefers the largest C.

41

Code an SVM in scikit-learn to classify text using TF-IDF features.

Text classification with an SVM usually pairs TF-IDF features with a linear SVM: TF-IDF turns each document into a sparse vector weighting words that are frequent in the document but rare in the corpus, and a linear SVM separates classes well in that high-dimensional space while training in seconds. Putting the vectorizer and classifier in one Pipeline keeps vocabulary building inside each cross-validation fold and makes the model accept raw strings.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Code Example

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC

categories = ["comp.graphics", "rec.sport.hockey", "sci.med", "talk.politics.guns"]
train = fetch_20newsgroups(subset="train", categories=categories, remove=("headers", "footers", "quotes"),
                           random_state=42)                    # downloads on first use
test = fetch_20newsgroups(subset="test", categories=categories, remove=("headers", "footers", "quotes"),
                          random_state=42)

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(sublinear_tf=True, stop_words="english", min_df=2)),
    ("svm", LinearSVC(class_weight="balanced")),
])
grid = GridSearchCV(pipe, {"tfidf__ngram_range": [(1, 1), (1, 2)], "svm__C": [0.1, 0.5, 1, 5]},
                    cv=5, scoring="f1_macro", n_jobs=-1)
grid.fit(train.data, train.target)
print("best parameters:", grid.best_params_, "CV macro F1:", round(grid.best_score_, 3))

pred = grid.predict(test.data)
print(classification_report(test.target, pred, target_names=train.target_names, digits=3))
print(confusion_matrix(test.target, pred))

# Most indicative terms per class
model = grid.best_estimator_
terms = np.array(model.named_steps["tfidf"].get_feature_names_out())
for i, name in enumerate(train.target_names):
    top = np.argsort(model.named_steps["svm"].coef_[i])[-8:][::-1]
    print(f"{name:20s} {', '.join(terms[top])}")

print(grid.predict(["The goalie made thirty saves in the playoff game"]))

Removing headers, footers and quotes stops the model from relying on metadata such as email addresses, which would inflate the scores.

Design Choices

Choice Reason
LinearSVC instead of RBF SVC Sparse, high-dimensional text is usually linearly separable; much faster
sublinear_tf=True Uses 1 + log(tf) so repeated words don’t dominate
min_df=2 Drops typos and very rare tokens
Bigrams Capture short phrases (“not good”)
class_weight="balanced" Handles uneven class sizes
Macro F1 Treats every class equally

Common Pitfalls

  • Fitting the vectorizer on all documents before splitting, which leaks vocabulary and IDF statistics.

  • Scaling TF-IDF with StandardScaler, which destroys sparsity; TF-IDF rows are already L2-normalized.

  • Using SVC(probability=True) when only labels are needed; use CalibratedClassifierCV if probabilities are required.

42

Build a multi-class SVM classifier using the one-vs-one strategy.

The one-vs-one (OvO) strategy trains a separate binary SVM for every pair of classes, K(K-1)/2 classifiers for K classes, and predicts by majority vote among them. Each classifier sees only the data of its two classes, so the individual problems are small, which suits kernel SVMs whose cost grows faster than linearly with the number of samples. scikit-learn’s SVC uses OvO internally; the example below also builds it explicitly with OneVsOneClassifier and by hand to show how voting works.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Code Example

from itertools import combinations

import numpy as np
from sklearn.datasets import load_wine
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsOneClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_wine(return_X_y=True)                       # 3 classes -> 3 pairwise classifiers
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

# 1. scikit-learn's wrapper
ovo = make_pipeline(StandardScaler(), OneVsOneClassifier(SVC(kernel="rbf", C=1.0)))
ovo.fit(X_train, y_train)
print("pairwise models:", len(ovo[-1].estimators_))
print(classification_report(y_test, ovo.predict(X_test), digits=3))

# 2. Manual one-vs-one with voting
scaler = StandardScaler().fit(X_train)
Xtr, Xte = scaler.transform(X_train), scaler.transform(X_test)
classes = np.unique(y_train)
votes = np.zeros((len(Xte), len(classes)))
scores = np.zeros((len(Xte), len(classes)))             # summed confidences break ties

for a, b in combinations(range(len(classes)), 2):
    mask = np.isin(y_train, [classes[a], classes[b]])
    clf = SVC(kernel="rbf", C=1.0).fit(Xtr[mask], (y_train[mask] == classes[b]).astype(int))
    d = clf.decision_function(Xte)                      # > 0 means class b wins
    votes[:, b] += d > 0
    votes[:, a] += d <= 0
    scores[:, b] += d
    scores[:, a] -= d

pred = classes[np.argmax(votes + 1e-3 * np.tanh(scores), axis=1)]
print("manual OvO accuracy:", round(accuracy_score(y_test, pred), 3))

# 3. SVC does the same internally
svc = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, decision_function_shape="ovo")).fit(X_train, y_train)
print("SVC accuracy:", round(svc.score(X_test, y_test), 3),
      "| pairwise scores per sample:", svc.decision_function(X_test[:1]).shape[1])

The tiny confidence term only breaks ties in the vote; it never overrides a clear majority.

OvO Versus OvR

Aspect One-vs-one One-vs-rest
Number of models K(K-1)/2 K
Data per model Two classes only All data
Class balance per model Roughly balanced Imbalanced (one vs many)
Best with Kernel SVMs, moderate K Linear models, many classes

Common Pitfalls

  • Scaling inside each pair with different statistics; scale once on the training data.

  • Ignoring ties in voting, which are common with few classes.

  • Many classes (e.g. 100 → 4,950 models): OvR or a linear model may be more practical.

43

Show how different kernels change SVM decision boundaries on 2D data.

Plotting SVM decision boundaries on 2D data is the clearest way to see what each kernel can represent: a linear kernel draws a straight line, a polynomial kernel draws curves built from feature interactions, and an RBF kernel draws flexible, local boundaries whose smoothness depends on gamma. The script below trains each kernel on a non-linear dataset, colours the predicted regions, marks the support vectors and prints cross-validated accuracy, so the visual impression can be checked against numbers.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+ (Python 3.12)

Code Example

import matplotlib

matplotlib.use("Agg")                                  # render to a file without a display
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_moons(n_samples=300, noise=0.25, random_state=42)

configs = [
    ("linear", SVC(kernel="linear", C=1)),
    ("poly, degree 3", SVC(kernel="poly", degree=3, coef0=1, C=1)),
    ("rbf, gamma 0.5", SVC(kernel="rbf", gamma=0.5, C=1)),
    ("rbf, gamma 50", SVC(kernel="rbf", gamma=50, C=1)),
]

fig, axes = plt.subplots(1, len(configs), figsize=(18, 4.5))
for ax, (name, svc) in zip(axes, configs):
    model = make_pipeline(StandardScaler(), svc)
    cv_acc = cross_val_score(model, X, y, cv=5).mean()
    model.fit(X, y)

    DecisionBoundaryDisplay.from_estimator(model, X, response_method="predict", alpha=0.3, ax=ax,
                                           cmap="coolwarm")
    DecisionBoundaryDisplay.from_estimator(model, X, response_method="decision_function", plot_method="contour",
                                           levels=[-1, 0, 1], linestyles=["--", "-", "--"], colors="k", ax=ax)
    ax.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", edgecolors="k", s=20)
    sv = model.named_steps["standardscaler"].inverse_transform(model.named_steps["svc"].support_vectors_)
    ax.scatter(sv[:, 0], sv[:, 1], s=80, facecolors="none", edgecolors="k", linewidths=1)
    ax.set_title(f"{name}\nCV acc {cv_acc:.2f}, {len(sv)} SVs")
    print(f"{name:15s} CV accuracy {cv_acc:.3f}, support vectors {len(sv)}")

plt.tight_layout()
plt.savefig("svm_kernels.png", dpi=120)

The solid line is the decision boundary (f(x) = 0) and the dashed lines are the margins (f(x) = ±1); circled points are support vectors.

What to Look For

Kernel Expected boundary on moons
Linear Straight line; misses the interlocking shape
Polynomial (degree 3) Smooth S-like curve
RBF, moderate gamma Follows the moons smoothly; usually best
RBF, very large gamma Tight islands around points; many SVs; overfits

Common Pitfalls

  • Plotting in scaled space but scattering raw points (or the reverse); keep the pipeline and inverse-transform the support vectors as above.

  • Judging the best kernel from the picture alone; check cross-validated scores.

44

Implement an SVM in Python using stochastic gradient descent.

Training an SVM with stochastic gradient descent (SGD) minimizes the primal objective, L2 regularization plus hinge loss, by updating the weights one example (or mini-batch) at a time using the hinge subgradient. It scales to millions of samples and streaming data because each step costs only O(d). Below is a from-scratch NumPy version with per-sample updates and a learning-rate schedule, compared with scikit-learn’s SGDClassifier(loss="hinge"), which implements the same idea.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (Python 3.12)

Update Rule in Words

For each example (x, y) with y ∈ {-1, +1} and learning rate eta: - If y (w·x + b) < 1 (inside the margin or wrong): w ← w - eta (lambda w - y x) and b ← b + eta y. - Otherwise: w ← w - eta lambda w (only shrinkage from regularization).

Code Example

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler


def sgd_svm(X, y, lam=1e-4, epochs=10, eta0=0.01, random_state=42):
    """Linear SVM by per-sample SGD on L2-regularized hinge loss. y in {0, 1}."""
    rng = np.random.default_rng(random_state)
    y_pm = np.where(y == 1, 1.0, -1.0)
    n, d = X.shape
    w, b, t = np.zeros(d), 0.0, 0
    history = []
    for epoch in range(epochs):
        for i in rng.permutation(n):
            t += 1
            eta = eta0 / (1 + eta0 * lam * t)                # slowly decaying step size
            if y_pm[i] * (X[i] @ w + b) < 1:
                w -= eta * (lam * w - y_pm[i] * X[i])
                b += eta * y_pm[i]
            else:
                w -= eta * lam * w
        hinge = np.maximum(0, 1 - y_pm * (X @ w + b)).mean()
        history.append(0.5 * lam * w @ w + hinge)
    return w, b, history


X, y = make_classification(n_samples=20000, n_features=30, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)

w, b, history = sgd_svm(X_train, y_train)
print("objective per epoch:", [round(h, 4) for h in history])
print("from-scratch accuracy:", round(((X_test @ w + b >= 0).astype(int) == y_test).mean(), 3))

ref = SGDClassifier(loss="hinge", alpha=1e-4, max_iter=10, tol=None, random_state=42).fit(X_train, y_train)
print("SGDClassifier accuracy:", round(ref.score(X_test, y_test), 3))

The objective should fall over the first epochs and then level off; both accuracies should be similar.

Practical Notes

  • Shuffle every epoch; ordered data (e.g. sorted by label) breaks SGD.

  • SGDClassifier adds averaging (average=True), early stopping, partial_fit for streams and class_weight.

  • For non-linear boundaries, apply RBFSampler or Nystroem features first.

Common Pitfalls

  • Unscaled features, which make one step size wrong for every feature.

  • Too large a learning rate, which makes the loss oscillate.

  • Comparing alpha in SGDClassifier directly with C in SVC; roughly alpha = 1 / (C * n_samples).

45

Write a script to visualize the support vectors of a trained SVM.

Visualizing support vectors shows which training points actually define an SVM: the points on the margin lines and those violating the margin. The script below trains a linear SVM on 2D data, draws the decision boundary and both margins, circles the support vectors, distinguishes margin support vectors (0 < alpha < C) from bounded ones (alpha = C), and, for higher-dimensional data, projects onto two principal components to show where the support vectors lie.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x, Matplotlib 3.8+ (Python 3.12)

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer, make_blobs
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

# --- 2D case: exact boundary and margins ---
X, y = make_blobs(n_samples=120, centers=2, cluster_std=1.6, random_state=42)
C = 1.0
svc = SVC(kernel="linear", C=C).fit(X, y)
alpha = np.abs(svc.dual_coef_[0])
on_margin = svc.support_[alpha < C - 1e-6]
bounded = svc.support_[alpha >= C - 1e-6]

fig, axes = plt.subplots(1, 2, figsize=(13, 5))
ax = axes[0]
ax.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", s=25, edgecolors="k")
xx, yy = np.meshgrid(np.linspace(*ax.get_xlim(), 200), np.linspace(*ax.get_ylim(), 200))
Z = svc.decision_function(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
ax.contour(xx, yy, Z, levels=[-1, 0, 1], linestyles=["--", "-", "--"], colors="k")
ax.scatter(X[on_margin, 0], X[on_margin, 1], s=150, facecolors="none", edgecolors="green", linewidths=2,
           label=f"on margin ({len(on_margin)})")
ax.scatter(X[bounded, 0], X[bounded, 1], s=150, facecolors="none", edgecolors="orange", linewidths=2,
           label=f"alpha = C ({len(bounded)})")
ax.legend()
ax.set_title(f"Linear SVM, C={C}: {len(svc.support_)} support vectors")

# --- High-dimensional case: PCA projection ---
Xb, yb = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0)).fit(Xb, yb)
Z2 = PCA(n_components=2, random_state=42).fit_transform(model.named_steps["standardscaler"].transform(Xb))
sv_mask = np.zeros(len(Xb), dtype=bool)
sv_mask[model.named_steps["svc"].support_] = True
ax = axes[1]
ax.scatter(Z2[~sv_mask, 0], Z2[~sv_mask, 1], c=yb[~sv_mask], cmap="coolwarm", s=10, alpha=0.4)
ax.scatter(Z2[sv_mask, 0], Z2[sv_mask, 1], c=yb[sv_mask], cmap="coolwarm", s=40, edgecolors="k")
ax.set_title(f"RBF SVM on 30 features (PCA view): {sv_mask.sum()} SVs of {len(Xb)}")

plt.tight_layout()
plt.savefig("support_vectors.png", dpi=120)
print("margin SVs:", len(on_margin), "| bounded SVs:", len(bounded))

In the PCA view, support vectors should cluster where the two classes meet; points far from that region are not needed by the model.

Reading the Plot

  • Green circles lie exactly on the dashed margin lines.

  • Orange circles are inside the margin or misclassified; their count grows as C decreases.

  • Re-running with a smaller C widens the margin and adds support vectors.

Common Pitfalls

  • Plotting support_vectors_ from a scaled pipeline on unscaled axes; use indices (support_) into the original data as above.

  • Reading the PCA plot as the true boundary: it is a projection of a 30-dimensional model.

46

Write a grid-search function to find the best SVM kernel and parameters.

A grid search over SVM kernels must account for the fact that each kernel has its own parameters: gamma for RBF, degree and coef0 for polynomial, and only C for linear. scikit-learn’s GridSearchCV accepts a list of parameter grids, one per kernel, so no meaningless combinations are tried. The function below wraps this with a scaling Pipeline, stratified folds, a configurable metric and a summary of the best result for each kernel.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Code Example

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def svm_kernel_search(X, y, scoring="f1", cv=5, n_jobs=-1, random_state=42):
    """Grid-search kernel, C and kernel-specific parameters for an SVC.

    Returns the fitted GridSearchCV object and a DataFrame with the best
    cross-validated score per kernel.
    """
    pipe = Pipeline([("scale", StandardScaler()), ("svc", SVC())])
    C_values = np.logspace(-2, 2, 5)
    param_grid = [
        {"svc__kernel": ["linear"], "svc__C": C_values},
        {"svc__kernel": ["rbf"], "svc__C": C_values, "svc__gamma": ["scale", 1e-3, 1e-2, 1e-1]},
        {"svc__kernel": ["poly"], "svc__C": C_values, "svc__degree": [2, 3],
         "svc__gamma": ["scale"], "svc__coef0": [0.0, 1.0]},
        {"svc__kernel": ["sigmoid"], "svc__C": C_values, "svc__gamma": [1e-3, 1e-2], "svc__coef0": [-1.0, 0.0]},
    ]
    folds = StratifiedKFold(n_splits=cv, shuffle=True, random_state=random_state)
    search = GridSearchCV(pipe, param_grid, scoring=scoring, cv=folds, n_jobs=n_jobs)
    search.fit(X, y)

    results = pd.DataFrame(search.cv_results_)
    results["kernel"] = results["param_svc__kernel"]
    summary = (results.sort_values("mean_test_score", ascending=False)
               .groupby("kernel", as_index=False).first()[["kernel", "mean_test_score", "std_test_score", "params"]]
               .sort_values("mean_test_score", ascending=False))
    return search, summary


X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

search, summary = svm_kernel_search(X_train, y_train)
pd.set_option("display.max_colwidth", 120)
print(summary.to_string(index=False))
print("overall best:", search.best_params_)
print("test F1 of the refitted best model:", round(search.score(X_test, y_test), 3))

Notes

  • GridSearchCV refits the best configuration on all training data (refit=True), so search.predict is ready to use.

  • The grid has 5 + 20 + 20 + 20 = 65 candidates × 5 folds = 325 fits; for larger grids use RandomizedSearchCV or HalvingGridSearchCV.

  • Compare kernels by their best scores and standard deviations; if the linear kernel is within noise of RBF, prefer it for speed and simplicity.

Common Pitfalls

  • One flat grid with gamma and degree for every kernel, which wastes fits on ignored parameters.

  • Evaluating on the training data used for the search; keep a held-out test set.

  • Scaling outside the pipeline.

47

How are quasi-Newton methods used for SVM training?

Quasi-Newton methods such as BFGS and its memory-efficient variant L-BFGS approximate second-order (curvature) information from successive gradients, converging in far fewer iterations than plain gradient descent. For SVMs they are applied to the primal problem of linear SVMs, usually with a smooth loss such as the squared hinge or a smoothed (Huberized) hinge, since the standard hinge is not differentiable at the margin. L-BFGS variants are used in large-scale and distributed linear SVM training, for example OWL-QN in Spark MLlib’s LinearSVC.

Read full answer

Applies toSciPy 1.13+, scikit-learn 1.5+, NumPy 2.x (Python 3.12)

Why and When

| Aspect | Detail | |—|—| | Objective | (1/2)||w||^2 + C sum max(0, 1 - y_i w·x_i)^2 (squared hinge) is once differentiable | | Advantage | Superlinear convergence; few passes over the data; easy to parallelize gradient computation | | Memory | L-BFGS stores only the last m gradient pairs (e.g. m = 10) | | Limitation | Plain hinge is non-smooth; kernel SVMs are usually solved in the dual with SMO instead | | Variants | OWL-QN for L1 regularization; truncated Newton (as in liblinear’s TRON for L2-loss SVMs) |

Code Example: Linear SVM via L-BFGS

import numpy as np
from scipy.optimize import minimize
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
y_pm = np.where(y_train == 1, 1.0, -1.0)
C = 0.1
Xa = np.hstack([X_train, np.ones((len(X_train), 1))])     # bias as an extra feature


def objective(theta):
    margins = 1 - y_pm * (Xa @ theta)
    active = margins > 0
    loss = 0.5 * theta @ theta + C * np.sum(margins[active] ** 2)
    grad = theta - 2 * C * (y_pm[active] * margins[active]) @ Xa[active]
    return loss, grad


res = minimize(objective, x0=np.zeros(Xa.shape[1]), jac=True, method="L-BFGS-B")
theta = res.x
pred = (np.hstack([X_test, np.ones((len(X_test), 1))]) @ theta >= 0).astype(int)
print("L-BFGS converged:", res.success, "in", res.nit, "iterations")
print("L-BFGS accuracy:", round((pred == y_test).mean(), 3))

ref = LinearSVC(C=C, loss="squared_hinge", max_iter=10000).fit(X_train, y_train)
print("LinearSVC accuracy:", round(ref.score(X_test, y_test), 3))
print("weight correlation:", round(np.corrcoef(theta[:-1], ref.coef_[0])[0, 1], 4))

The bias is regularized here, as in liblinear’s default, so the weights should be nearly identical to LinearSVC‘s.

Compared with Other Solvers

Solver Problem form Typical use
SMO (libsvm) Dual, any kernel SVC
Dual coordinate descent (liblinear) Dual, linear LinearSVC default
Trust-region Newton (liblinear) Primal, squared hinge Dense, well-conditioned problems
L-BFGS / OWL-QN Primal, smooth loss Distributed linear SVMs
SGD Primal, hinge Streaming and huge data

Common Pitfalls

  • Applying L-BFGS to the plain hinge loss and getting poor convergence because of the kink.

  • Unscaled features, which make the curvature approximation poor.

48

What is Sequential Minimal Optimization (SMO), and why does it matter for SVMs?

Sequential Minimal Optimization (SMO), introduced by John Platt in 1998, solves the SVM dual problem by repeatedly optimizing just two Lagrange multipliers at a time while keeping the others fixed. Two is the minimum, because the constraint sum alpha_i y_i = 0 means changing one multiplier forces a change in another. Each two-variable subproblem has a closed-form solution, so SMO needs no general quadratic programming solver and only a kernel cache instead of the full n × n matrix. It made kernel SVMs practical and is the basis of libsvm, which scikit-learn’s SVC and SVR use.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+ (Python 3.12)

Algorithm Outline

  1. Start with all alpha_i = 0, b = 0.

  2. Pick a multiplier alpha_i that violates the KKT conditions (heuristics choose the worst violator).

  3. Pick a second alpha_j that will give a large step (e.g. maximizing |E_i - E_j|, where E is the prediction error).

  4. Compute the unconstrained optimum for alpha_j along the constraint line, using eta = K_ii + K_jj - 2 K_ij.

  5. Clip alpha_j to the box [L, H] implied by 0 <= alpha <= C and the equality constraint; update alpha_i to compensate.

  6. Update b and the error cache.

  7. Repeat until all KKT conditions hold within a tolerance.

Code Example: Simplified SMO

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC


def simplified_smo(X, y, C=1.0, tol=1e-3, max_passes=20, random_state=42):
    """Simplified SMO for a linear kernel; y in {-1, +1}. Picks the second index at random."""
    rng = np.random.default_rng(random_state)
    n = len(y)
    K = X @ X.T
    alpha, b, passes = np.zeros(n), 0.0, 0
    while passes < max_passes:
        changed = 0
        for i in range(n):
            E_i = (alpha * y) @ K[:, i] + b - y[i]
            if (y[i] * E_i < -tol and alpha[i] < C) or (y[i] * E_i > tol and alpha[i] > 0):
                j = rng.choice([k for k in range(n) if k != i])
                E_j = (alpha * y) @ K[:, j] + b - y[j]
                a_i, a_j = alpha[i], alpha[j]
                if y[i] != y[j]:
                    L, H = max(0, a_j - a_i), min(C, C + a_j - a_i)
                else:
                    L, H = max(0, a_i + a_j - C), min(C, a_i + a_j)
                eta = 2 * K[i, j] - K[i, i] - K[j, j]
                if L == H or eta >= 0:
                    continue
                alpha[j] = np.clip(a_j - y[j] * (E_i - E_j) / eta, L, H)
                if abs(alpha[j] - a_j) < 1e-5:
                    continue
                alpha[i] = a_i + y[i] * y[j] * (a_j - alpha[j])
                b1 = b - E_i - y[i] * (alpha[i] - a_i) * K[i, i] - y[j] * (alpha[j] - a_j) * K[i, j]
                b2 = b - E_j - y[i] * (alpha[i] - a_i) * K[i, j] - y[j] * (alpha[j] - a_j) * K[j, j]
                b = b1 if 0 < alpha[i] < C else b2 if 0 < alpha[j] < C else (b1 + b2) / 2
                changed += 1
        passes = passes + 1 if changed == 0 else 0
    return alpha, b


X, y01 = make_blobs(n_samples=80, centers=2, cluster_std=1.5, random_state=42)
y = np.where(y01 == 1, 1.0, -1.0)
alpha, b = simplified_smo(X, y)
w = (alpha * y) @ X
ref = SVC(kernel="linear", C=1.0).fit(X, y)
print("SMO w:", w.round(3), "b:", round(b, 3), "| SVs:", int((alpha > 1e-5).sum()))
print("sklearn w:", ref.coef_[0].round(3), "b:", round(ref.intercept_[0], 3), "| SVs:", len(ref.support_))
print("prediction agreement:", (np.sign(X @ w + b) == ref.predict(X)).mean())

The simplified version picks the second multiplier at random, so it is slower than libsvm’s heuristics, but it should reach nearly the same solution.

Why It Matters

  • Memory: no full kernel matrix needed; libsvm caches the rows it uses (cache_size).

  • Simplicity: analytic updates, no external QP solver.

  • Speed: practical training for tens of thousands of samples.

Common Pitfalls

  • Expecting SMO to scale to millions of samples; it is still roughly quadratic or worse.

  • Loose tolerance (tol) settings that stop early with an inaccurate solution.

49

What is Platt scaling, and why use probabilistic outputs from SVMs?

Platt scaling converts an SVM’s raw decision scores into probabilities by fitting a logistic (sigmoid) function to them: P(y = 1 | x) = 1 / (1 + exp(A * f(x) + B)), where f(x) is the SVM score and A, B are learned on held-out data. SVMs naturally output distances to the boundary, not probabilities, but many applications need calibrated probabilities: risk-based decisions, cost-sensitive thresholds, combining models, and reporting confidence to users.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Why Probabilities Matter

Need Example
Decision with costs Approve a loan if expected profit = P(repay) × gain − (1 − P) × loss > 0
Ranking by risk Prioritize patients by probability of disease
Model combination Average probabilities across models in an ensemble
Metrics Log loss, Brier score and calibration curves need probabilities

In scikit-learn

Option How Notes
SVC(probability=True) Internal 5-fold CV fits Platt’s sigmoid Slower training; predict_proba can disagree with predict near the boundary
CalibratedClassifierCV(SVC(), method="sigmoid", cv=5) Platt scaling with explicit CV More control; works for LinearSVC too
CalibratedClassifierCV(..., method="isotonic") Non-parametric monotone mapping Needs more data (roughly 1,000+ calibration samples)

Code Example

from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.datasets import make_classification
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC

X, y = make_classification(n_samples=6000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

for method in ("sigmoid", "isotonic"):
    model = make_pipeline(StandardScaler(), CalibratedClassifierCV(LinearSVC(C=0.1), method=method, cv=5))
    model.fit(X_train, y_train)
    proba = model.predict_proba(X_test)[:, 1]
    frac_pos, mean_pred = calibration_curve(y_test, proba, n_bins=10)
    print(f"{method:8s} log loss {log_loss(y_test, proba):.4f}  Brier {brier_score_loss(y_test, proba):.4f}")
    print("   predicted vs observed per bin:",
          [(round(p, 2), round(o, 2)) for p, o in zip(mean_pred, frac_pos)][:5], "...")

A well-calibrated model has observed frequencies close to the predicted probabilities in each bin.

Common Pitfalls

  • Fitting the sigmoid on the same data used to train the SVM, which gives over-confident probabilities (scikit-learn uses cross-validation to avoid this).

  • Treating decision_function values as probabilities.

  • Using probability=True when only labels or rankings (ROC AUC) are needed; it only adds training time.

50

What is the "Reese kernel" in SVM, and is it a standard kernel?

There is no standard SVM kernel called the “Reese kernel”. It does not appear among the kernels in the main SVM libraries (libsvm, scikit-learn, LIBLINEAR) or in the standard kernel-methods literature, so the name is most likely a misremembering or a mix-up with another kernel. In an interview, the best answer is to say so plainly, then explain which kernels are commonly used and how you would evaluate an unfamiliar kernel if someone proposed one.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Kernels Commonly Used with SVMs

Kernel Formula (in words) Typical use
Linear Dot product Text, high-dimensional data
Polynomial (gamma × dot product + coef0) to a power Feature interactions
RBF / Gaussian exp(−gamma × squared distance) Default non-linear kernel
Laplacian exp(−gamma × L1 distance) Data with sharp changes
Sigmoid tanh(gamma × dot product + coef0) Rare; neural-network analogy
Rational quadratic Mixture of RBF kernels at many length scales Gaussian processes, smooth multi-scale data
Chi-squared / histogram intersection Compare histograms Computer vision descriptors
String, tree and graph kernels Count shared substructures Sequences, molecules, parse trees

How to Evaluate an Unfamiliar Kernel

  1. Find the definition: the formula and its parameters, and a reference.

  2. Check validity: the kernel must be symmetric and positive semi-definite (PSD); test the smallest eigenvalue of its Gram matrix on real data.

  3. Implement it as a callable or precomputed kernel.

  4. Compare it with linear and RBF baselines under the same cross-validation.

Code Example: Testing a Proposed Kernel

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC


def proposed_kernel(A, B, gamma=0.05, alpha=1.0):
    """Example 'unfamiliar' kernel: rational quadratic, (1 + gamma * d^2 / (2 * alpha)) ** -alpha."""
    sq = (A ** 2).sum(1)[:, None] + (B ** 2).sum(1)[None, :] - 2 * A @ B.T
    return (1 + gamma * np.maximum(sq, 0) / (2 * alpha)) ** -alpha


X, y = load_breast_cancer(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
K = proposed_kernel(Xs[:300], Xs[:300])
print("symmetric:", np.allclose(K, K.T), "| smallest eigenvalue:", round(float(np.linalg.eigvalsh(K).min()), 6))

for name, svc in [("proposed", SVC(kernel=proposed_kernel)), ("rbf", SVC(kernel="rbf")),
                  ("linear", SVC(kernel="linear"))]:
    score = cross_val_score(make_pipeline(StandardScaler(), svc), X, y, cv=5).mean()
    print(f"{name:8s} CV accuracy {score:.3f}")

Interview Tips

  • It is better to say “I’m not aware of a standard kernel by that name” than to invent a description.

  • Show you can still reason about any kernel: validity (PSD), parameters, cost and empirical comparison.

51

How would you build an anomaly detection system with a one-class SVM?

A one-class SVM learns a boundary around normal data only: it maps the data with a kernel (usually RBF) and finds the smallest region, separated from the origin with maximum margin, that contains most of the training points. New points outside the region are flagged as anomalies. It suits problems where anomalies are rare, varied or unlabelled, such as equipment faults, fraud or intrusions. The parameter nu sets an upper bound on the fraction of training points treated as outliers (and a lower bound on the fraction of support vectors), and gamma controls how tightly the boundary wraps the data.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

System Design

Step Details
1. Data Collect mostly normal operating data; remove known faults from training
2. Features Domain features (rolling means, variances, frequency bands for sensors); scale them
3. Model OneClassSVM(kernel="rbf", nu=..., gamma=...) in a Pipeline
4. Tuning If a few labelled anomalies exist, tune nu/gamma on a validation set for precision/recall; otherwise set nu to the expected anomaly rate
5. Scoring decision_function: negative = outside the boundary; rank alerts by score
6. Thresholding Adjust the threshold to the alert budget
7. Monitoring Track alert rate and drift; retrain on recent normal data

Code Example: Machine Sensor Monitoring

import numpy as np
from sklearn.metrics import average_precision_score, classification_report
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import OneClassSVM

rng = np.random.default_rng(42)
# Normal operation: temperature and vibration are correlated
temp = rng.normal(70, 3, 2000)
vib = 0.05 * temp + rng.normal(0, 0.2, 2000)
normal = np.column_stack([temp, vib])
# Faults: overheating, or vibration unusually high for the temperature
faults = np.vstack([np.column_stack([rng.normal(85, 2, 30), rng.normal(4.3, 0.2, 30)]),
                    np.column_stack([rng.normal(70, 3, 30), rng.normal(5.5, 0.3, 30)])])

X_train = normal[:1500]                                        # normal data only
X_test = np.vstack([normal[1500:], faults])
y_test = np.r_[np.zeros(500), np.ones(len(faults))]           # 1 = fault (for evaluation only)

detector = make_pipeline(StandardScaler(), OneClassSVM(kernel="rbf", nu=0.01, gamma=0.5))
detector.fit(X_train)

scores = -detector.decision_function(X_test)                  # higher = more anomalous
pred = (detector.predict(X_test) == -1).astype(int)
print(classification_report(y_test, pred, target_names=["normal", "fault"], digits=3))
print("average precision of anomaly scores:", round(average_precision_score(y_test, scores), 3))
print("training points flagged:", round((detector.predict(X_train) == -1).mean(), 3), "(about nu or less)")

The second fault type has a normal temperature and a vibration level that is only abnormal relative to temperature, which a one-class SVM can catch because it models the joint shape of the data.

Alternatives

Isolation Forest (fast, scales well), Local Outlier Factor, autoencoders for high-dimensional signals, and SGDOneClassSVM with kernel approximation for large datasets.

Common Pitfalls

  • Training data contaminated with anomalies, which the model then learns as normal.

  • Unscaled features, which distort the RBF boundary.

  • A very large gamma that wraps each training point individually and flags all new data.

52

How can SVMs be used for feature selection?

SVMs support feature selection in three main ways: SVM-RFE (recursive feature elimination), which repeatedly trains a linear SVM and removes the features with the smallest absolute weights; L1-regularized linear SVMs, whose penalty drives many weights exactly to zero so only useful features remain; and wrapper or model-agnostic methods such as permutation importance with a kernel SVM. Feature selection reduces noise and cost and improves interpretability, which is especially valuable in high-dimensional data such as gene expression or text.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Methods

Method How Notes
SVM-RFE RFE / RFECV with LinearSVC or SVC(kernel="linear"); drop lowest ‘ w ‘ features step by step Classic in genomics (Guyon et al., 2002)
L1-penalized LinearSVC LinearSVC(penalty="l1", dual=False, C=...) + SelectFromModel Smaller C = fewer features
Permutation importance Shuffle each feature and measure the score drop Works with RBF kernels
Filter + SVM SelectKBest(f_classif) before the SVM Fast; ignores interactions

Weights are only comparable when features are scaled, so always standardize first.

Code Example

from sklearn.datasets import make_classification
from sklearn.feature_selection import RFECV, SelectFromModel
from sklearn.inspection import permutation_importance
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = make_classification(n_samples=600, n_features=40, n_informative=6, n_redundant=4,
                           shuffle=False, random_state=42)      # first 10 columns carry the signal
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# 1. SVM-RFE with cross-validation to choose the number of features
rfe = make_pipeline(StandardScaler(), RFECV(LinearSVC(C=0.1, max_iter=5000), step=2, cv=cv, scoring="accuracy"))
rfe.fit(X, y)
selected = rfe[-1].support_.nonzero()[0]
print("RFECV kept", len(selected), "features:", selected)

# 2. L1-penalized LinearSVC
l1 = make_pipeline(StandardScaler(), SelectFromModel(LinearSVC(penalty="l1", dual=False, C=0.05, max_iter=5000)))
l1.fit(X, y)
print("L1 kept:", l1[-1].get_support().nonzero()[0])

# 3. Compare accuracy with and without selection (selection inside the pipeline, so no leakage)
full = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
selected_model = make_pipeline(StandardScaler(),
                               SelectFromModel(LinearSVC(penalty="l1", dual=False, C=0.05, max_iter=5000)),
                               SVC(kernel="rbf"))
for name, model in [("all features", full), ("L1-selected", selected_model)]:
    print(f"{name:13s} CV accuracy {cross_val_score(model, X, y, cv=cv).mean():.3f}")

# 4. Permutation importance for an RBF SVM
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
full.fit(X_train, y_train)
imp = permutation_importance(full, X_test, y_test, n_repeats=10, random_state=42)
print("top features by permutation importance:", imp.importances_mean.argsort()[::-1][:8])

Because shuffle=False, the informative and redundant features are columns 0-9, so the selected indices should mostly fall in that range.

Common Pitfalls

  • Selecting features on the full dataset and then cross-validating the SVM on them, which gives optimistic scores.

  • Interpreting weights of unscaled features.

  • Dropping correlated features one at a time: RFE may keep one of a correlated pair arbitrarily.

53

How are SVMs used in bioinformatics and computational biology?

SVMs have been widely used in bioinformatics because biological datasets often have many features and few samples, such as tens of thousands of gene expression values for a few hundred patients, where margin-based models generalize well. Classic applications include cancer subtype classification from microarray or RNA-seq data, protein function and structure prediction, splice-site and promoter detection in DNA, protein-protein interaction prediction, and drug activity prediction. Specialized string kernels (spectrum, mismatch kernels) let SVMs work directly on DNA and protein sequences.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Applications

Problem Input SVM approach
Cancer classification Gene expression (p >> n) Linear SVM, often with feature selection (SVM-RFE)
Splice sites, promoters DNA sequences String / spectrum kernels
Protein family / remote homology Protein sequences Mismatch or profile kernels
Secondary structure Sequence windows Multi-class SVM
Protein-protein interaction Pair features Pairwise kernels
Drug activity (QSAR) Molecular fingerprints Tanimoto kernel SVM
Mass spectrometry diagnostics Spectra peaks Linear or RBF SVM

Code Example: Spectrum Kernel for DNA Sequences

The spectrum kernel counts shared k-mers (substrings of length k) between two sequences.

from itertools import product

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

rng = np.random.default_rng(42)
MOTIF = "TATAAT"                                        # promoter-like motif

def random_seq(n=60, with_motif=False):
    seq = "".join(rng.choice(list("ACGT"), n))
    if with_motif:
        pos = rng.integers(0, n - len(MOTIF))
        seq = seq[:pos] + MOTIF + seq[pos + len(MOTIF):]
    return seq

seqs = [random_seq(with_motif=i % 2 == 0) for i in range(400)]
y = np.array([int(i % 2 == 0) for i in range(400)])

K_MER = 3
kmers = ["".join(p) for p in product("ACGT", repeat=K_MER)]
index = {k: i for i, k in enumerate(kmers)}

def spectrum_features(seq):
    v = np.zeros(len(kmers))
    for i in range(len(seq) - K_MER + 1):
        v[index[seq[i:i + K_MER]]] += 1
    return v / np.linalg.norm(v)                       # normalize so long sequences don't dominate

X = np.array([spectrum_features(s) for s in seqs])     # explicit k-mer counts: K(a, b) = X_a · X_b
print("spectrum kernel SVM CV accuracy:", round(cross_val_score(SVC(kernel="linear", C=1.0), X, y, cv=5).mean(), 3))

A linear SVM on k-mer count vectors is exactly a spectrum-kernel SVM; for long k, sparse vectors or tries keep it efficient.

Practical Concerns

  • p >> n: use strong regularization and nested cross-validation; select features inside the folds.

  • Batch effects: samples from different labs or runs differ systematically; correct or validate across batches.

  • Interpretability: linear weights or SVM-RFE rankings point to candidate genes or motifs.

Common Pitfalls

  • Selecting genes on the whole dataset before cross-validation, which gives wildly optimistic accuracy.

  • Homologous sequences in both training and test sets, which inflates protein-prediction results.

54

How would you apply SVMs to image classification tasks?

Applying SVMs to image classification requires turning each image into a fixed-length feature vector, because SVMs cannot learn spatial features from raw pixels as CNNs do. Historically this meant hand-crafted descriptors (HOG, SIFT with bag-of-visual-words, colour histograms) fed to RBF or chi-squared kernel SVMs. Today the most effective approach is to extract embeddings from a pretrained CNN or vision transformer and train a linear SVM on them, which gives strong accuracy with few labelled images and little compute.

Read full answer

Applies toscikit-learn 1.5+, scikit-image 0.24+ (Python 3.12)

Feature Options

Features Strength Typical SVM
Raw pixels (small, aligned images) Simple RBF
HOG Shape and edges Linear
Colour / texture histograms (LBP) Appearance Chi-squared or RBF
Bag of visual words (SIFT/ORB) Classic object recognition Chi-squared / histogram intersection
Pretrained deep embeddings Best accuracy Linear

Code Example: HOG + Colour Features with an SVM

import numpy as np
from skimage.color import rgb2gray
from skimage.feature import hog
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

rng = np.random.default_rng(42)

def make_image(label):
    """Synthetic 32x32 RGB images: 0 = horizontal stripes (bluish), 1 = vertical stripes (reddish)."""
    img = rng.normal(0.4, 0.1, (32, 32, 3))
    stripes = (np.arange(32) // 4) % 2
    if label == 0:
        img[stripes == 1, :, 2] += 0.3
    else:
        img[:, stripes == 1, 0] += 0.3
    return np.clip(img, 0, 1)

labels = rng.integers(0, 2, 600)
images = [make_image(label) for label in labels]

def features(img):
    shape = hog(rgb2gray(img), orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2))
    colour = np.concatenate([np.histogram(img[..., c], bins=8, range=(0, 1))[0] / img[..., c].size
                             for c in range(3)])
    return np.concatenate([shape, colour])

X = np.array([features(img) for img in images])
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.25, stratify=labels, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": [0.1, 1, 10], "svc__kernel": ["linear", "rbf"]},
                    cv=5).fit(X_train, y_train)
print(grid.best_params_)
print(classification_report(y_test, grid.predict(X_test), target_names=["horizontal", "vertical"]))

For real photos, replace features with embeddings from a pretrained network (for example ResNet or a ViT with its classification head removed) and use LinearSVC.

Common Pitfalls

  • Raw pixels of large images: huge dimensions, no translation invariance, slow RBF training.

  • Inconsistent resizing and colour handling between training and inference.

  • Too few images per class without augmentation or pretrained features.

55

Describe a financial application where SVMs are used for forecasting.

A typical financial forecasting application of SVMs is predicting the direction or size of short-term changes, for example whether a stock index or currency pair will rise over the next day (SVM classifier), or next-day volatility (Support Vector Regression), from lagged returns, volatility measures, volume and macro indicators. Another well-established use is credit risk: predicting the probability that a borrower will default. SVMs handle non-linear relationships with modest data, but financial series are noisy and non-stationary, so the design must center on leakage-free features, time-ordered validation and comparison with naive baselines.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, NumPy 2.x (Python 3.12)

Application: Next-Day Volatility Forecasting with SVR

Element Design
Target Absolute return tomorrow (a simple volatility proxy)
Features Rolling volatility (5, 20 days), lagged absolute returns, range, volume change; all known at today’s close
Model Standardized features + RBF SVR
Validation TimeSeriesSplit walk-forward; final test on the most recent period
Baselines Yesterday’s absolute return; 20-day rolling mean
Use Position sizing, risk limits, option pricing inputs

Code Example

import numpy as np
import pandas as pd
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import GridSearchCV, TimeSeriesSplit
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR

rng = np.random.default_rng(42)
n = 2500
vol = np.empty(n); vol[0] = 0.01
for t in range(1, n):                                     # simulated volatility clustering
    vol[t] = 0.0003 + 0.9 * vol[t - 1] + 0.1 * abs(rng.normal(0, vol[t - 1]))
returns = rng.normal(0, vol)

df = pd.DataFrame({"ret": returns})
df["abs_ret"] = df["ret"].abs()
df["vol_5"] = df["ret"].rolling(5).std()
df["vol_20"] = df["ret"].rolling(20).std()
df["abs_lag1"] = df["abs_ret"].shift(1)
df["target"] = df["abs_ret"].shift(-1)                    # tomorrow's absolute return
df = df.dropna()

features = ["abs_ret", "vol_5", "vol_20", "abs_lag1"]
split = int(len(df) * 0.8)
train, test = df.iloc[:split], df.iloc[split:]

grid = GridSearchCV(make_pipeline(StandardScaler(), SVR(kernel="rbf")),
                    {"svr__C": [0.01, 0.1, 1], "svr__epsilon": [1e-4, 1e-3], "svr__gamma": ["scale", 0.1]},
                    cv=TimeSeriesSplit(n_splits=5), scoring="neg_mean_absolute_error")
grid.fit(train[features], train["target"])
pred = grid.predict(test[features])

print("SVR MAE:            ", round(mean_absolute_error(test["target"], pred), 6))
print("baseline (today):   ", round(mean_absolute_error(test["target"], test["abs_ret"]), 6))
print("baseline (20d vol): ", round(mean_absolute_error(test["target"], test["vol_20"] * np.sqrt(2 / np.pi)), 6))

The last baseline converts the 20-day standard deviation into an expected absolute return (for normal returns, the mean absolute value is about 0.8 times the standard deviation).

Common Pitfalls

  • Look-ahead bias: features or scaling that use future data; random splits instead of time-ordered ones.

  • Claiming trading profits without transaction costs and realistic execution.

  • Overfitting through repeated testing on the same historical period.

  • Regime changes (crises) that break relationships learned in calm periods.

56

How can SVMs be used for sentiment analysis on social media data?

SVM-based sentiment analysis on social media classifies posts as positive, negative or neutral using a linear SVM on text features. Social media text is short, informal and noisy, full of hashtags, mentions, emojis, slang, elongated words (“sooo good”) and sarcasm, so preprocessing and feature design matter: normalizing URLs and user names, keeping emojis and negations, and adding character n-grams that are robust to misspellings. A linear SVM on these features is fast and a strong baseline, though fine-tuned transformers usually do better when enough labelled data and compute are available.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Feature Design

Challenge Handling
URLs, @mentions Replace with tokens URL, USER
Hashtags Keep the word (#happy → happy plus the hashtag token)
Emojis and emoticons Keep them; they carry strong sentiment
Elongations (“sooo”) Reduce repeated characters to two (“soo”)
Negation (“not good”) Word bigrams capture it
Misspellings, slang Character n-grams (2-5)
Class imbalance class_weight="balanced"

Code Example

import re

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import FeatureUnion, make_pipeline
from sklearn.svm import LinearSVC


def normalize(text):
    text = text.lower()
    text = re.sub(r"https?://\S+", " URL ", text)
    text = re.sub(r"@\w+", " USER ", text)
    text = re.sub(r"#(\w+)", r" \1 HASHTAG_\1 ", text)
    return re.sub(r"(.)\1{2,}", r"\1\1", text)               # "sooo" -> "soo"


posts = ["Loving the new update!!! 😍 #happy", "@support this app keeps crashing, sooo annoying 😡",
         "not bad at all, works fine", "Worst customer service ever http://x.co/abc",
         "Great battery life, really impressed 👍", "meh, it's ok I guess",
         "Totally disappointed, not worth the money", "Best purchase this year!! #win"]
labels = ["pos", "neg", "pos", "neg", "pos", "neu", "neg", "pos"]

features = FeatureUnion([
    ("words", TfidfVectorizer(preprocessor=normalize, ngram_range=(1, 2), token_pattern=r"[^\s]+")),
    ("chars", TfidfVectorizer(preprocessor=normalize, analyzer="char_wb", ngram_range=(2, 5))),
])
model = make_pipeline(features, LinearSVC(C=0.5, class_weight="balanced"))
model.fit(posts, labels)

for post in ["sooo happy with this!!! 😍", "app crashed again, not good", "it is fine"]:
    print(f"{post!r} -> {model.predict([post])[0]}")

The tiny dataset is only for illustration; a real model needs thousands of labelled posts (for example from public benchmarks such as SemEval Twitter sentiment tasks or your own annotated data) and evaluation with macro F1 on a held-out set.

Common Pitfalls

  • Removing emojis, punctuation and stop words such as “not”, which carry sentiment.

  • Training on one platform or time period and deploying on another, where slang differs.

  • Ignoring sarcasm and mixed sentiment; review errors to see where the model fails.

  • Accuracy as the only metric when neutral posts dominate.

57

How are SVMs applied to text categorization?

Text categorization assigns documents to predefined categories, such as topics, departments, languages or spam vs ham, and linear SVMs on TF-IDF features have been a top-performing approach since Joachims’ work in 1998. Text produces very high-dimensional, sparse vectors in which categories are usually close to linearly separable, exactly where margin-based linear models shine. SVMs handle single-label, multi-label (one binary SVM per label) and hierarchical categorization, and they remain strong, fast baselines next to fine-tuned transformers.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Why SVMs Fit Text

Property of text Why SVMs handle it well
Tens of thousands of features Margin-based generalization doesn’t depend on dimension directly
Sparse vectors Linear solvers (liblinear) exploit sparsity
Few irrelevant features overall Weights spread over many informative terms
Linearly separable categories Linear kernel is enough; fast

Task Variants

Variant Approach
Single-label, many classes LinearSVC (one-vs-rest)
Multi-label (document can have several tags) OneVsRestClassifier(LinearSVC()) with a binary indicator matrix
Hierarchical (topic → subtopic) Classifier per level or per parent node
Short texts (titles, tweets) Add character n-grams

Code Example: Multi-Label Tagging

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.multiclass import OneVsRestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.svm import LinearSVC

docs = [
    "Central bank raises interest rates to curb inflation",
    "New vaccine trial shows strong immune response",
    "Tech company reports record quarterly profit from cloud services",
    "Hospital adopts AI system to read chest X-rays",
    "Stock markets fall as inflation data surprises investors",
    "Startup raises funding for medical imaging software",
    "Government announces budget with new healthcare spending",
    "Chipmaker shares jump after earnings beat expectations",
]
tags = [["economy"], ["health"], ["tech", "business"], ["health", "tech"], ["economy", "business"],
        ["business", "health", "tech"], ["economy", "health"], ["business", "tech"]]

mlb = MultiLabelBinarizer()
Y = mlb.fit_transform(tags)
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
                      OneVsRestClassifier(LinearSVC(C=1.0)))
model.fit(docs, Y)

new_docs = ["Investors worry about inflation and interest rates",
            "Software helps doctors analyse scans faster"]
for doc, row in zip(new_docs, model.predict(new_docs)):
    print(doc, "->", list(mlb.classes_[row.astype(bool)]))

The training set here is tiny for illustration; real systems use thousands of labelled documents per category and evaluate with micro- and macro-averaged F1 on a held-out set (classification_report supports multi-label indicator matrices).

Common Pitfalls

  • Using an RBF kernel on sparse TF-IDF features: much slower, rarely more accurate.

  • Fitting the vectorizer on test documents.

  • Ignoring label imbalance in multi-label data; use class_weight="balanced" and per-label thresholds.

  • Reporting only micro-F1, which hides poor performance on rare categories.

58

How can SVMs be used for handwriting recognition?

SVMs were among the most accurate handwriting recognizers before deep learning: on digit benchmarks such as MNIST, RBF and polynomial SVMs on raw or lightly processed pixels reached error rates around 1-2%, and SVMs on engineered features such as HOG did even better. The approach is to normalize each character image (size, position, slant), convert it to a feature vector (pixels, HOG or other descriptors), and train a multi-class SVM. Today CNNs are more accurate, but SVMs remain a strong baseline for small datasets and constrained devices.

Read full answer

Applies toscikit-learn 1.5+, scikit-image 0.24+ (Python 3.12)

Pipeline

Step Purpose
Segmentation Split a word or form into individual characters (or use whole-word models)
Normalization Centre, resize to a fixed grid, correct slant, binarize or scale to [0, 1]
Features Raw pixels, HOG (stroke directions), zoning, projection histograms
Classifier Multi-class SVM (one-vs-one in SVC) with RBF kernel
Post-processing Dictionary or language model for words

Code Example: Digits with Pixels vs HOG Features

import numpy as np
from skimage.feature import hog
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

digits = load_digits()                                      # 1,797 images of 8x8 pixels
X_pix = digits.data / 16.0
X_hog = np.array([hog(img, orientations=8, pixels_per_cell=(4, 4), cells_per_block=(1, 1))
                  for img in digits.images])

for name, X in [("pixels", X_pix), ("pixels + HOG", np.hstack([X_pix, X_hog]))]:
    X_train, X_test, y_train, y_test = train_test_split(X, digits.target, test_size=0.25,
                                                        stratify=digits.target, random_state=42)
    grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
                        {"svc__C": [1, 10, 100], "svc__gamma": ["scale", 0.001, 0.01]}, cv=5, n_jobs=-1)
    grid.fit(X_train, y_train)
    pred = grid.predict(X_test)
    print(f"{name:13s} test accuracy {accuracy_score(y_test, pred):.4f}  best {grid.best_params_}")

print(confusion_matrix(y_test, pred))                       # which digits get confused

Improving Accuracy

  • Data augmentation with small shifts and rotations (the “virtual support vector” method added transformed copies of support vectors).

  • Deskewing images before feature extraction.

  • Larger images and richer descriptors for full characters and words.

Common Pitfalls

  • Inconsistent normalization between training and new handwriting samples.

  • Using raw pixels from large images with an RBF SVM on huge datasets: training becomes very slow; reduce dimensions or use CNNs.

  • Evaluating on writers who also appear in training; split by writer for realistic results.

59

How would you use SVMs for intrusion detection in cybersecurity?

SVMs are used for intrusion detection in two ways: as supervised classifiers that learn to separate normal traffic from known attack types using labelled network-flow or log features, and as one-class SVMs that learn the boundary of normal behaviour and flag anything outside it, which can catch previously unseen attacks. A practical system extracts features from flows (durations, byte counts, flags, rates), scales and encodes them, trains the model on representative traffic, and tunes thresholds to keep false alarms manageable for analysts.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+ (Python 3.12)

Design

Component Choice
Data Flow records (e.g. NetFlow/Zeek logs) labelled with attack types; public datasets such as CIC-IDS2017 or UNSW-NB15 for research
Features Protocol, service, flags (one-hot); bytes, packets, duration, rates (log-transformed and scaled)
Supervised model Linear or RBF SVM with class_weight="balanced" for known attacks
Anomaly model OneClassSVM trained on normal traffic only
Metrics Recall per attack type, false-positive rate per day, precision at the alert budget
Deployment Streaming feature extraction, periodic retraining, analyst feedback loop

Code Example: Supervised and One-Class Detectors

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.compose import ColumnTransformer
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler
from sklearn.svm import SVC, OneClassSVM

rng = np.random.default_rng(42)
n_normal, n_attack = 4000, 400
normal = pd.DataFrame({"proto": rng.choice(["tcp", "udp"], n_normal, p=[0.8, 0.2]),
                       "bytes": rng.lognormal(8, 1, n_normal), "duration": rng.exponential(2, n_normal),
                       "syn_ratio": rng.beta(2, 8, n_normal)})
attack = pd.DataFrame({"proto": rng.choice(["tcp", "udp"], n_attack, p=[0.95, 0.05]),
                       "bytes": rng.lognormal(5, 0.5, n_attack), "duration": rng.exponential(0.2, n_attack),
                       "syn_ratio": rng.beta(8, 2, n_attack)})          # SYN-flood-like behaviour
df = pd.concat([normal, attack], ignore_index=True)
y = np.r_[np.zeros(n_normal), np.ones(n_attack)].astype(int)

prep = ColumnTransformer([
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["proto"]),
    ("num", make_pipeline(FunctionTransformer(np.log1p), StandardScaler()), ["bytes", "duration", "syn_ratio"]),
])
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.3, stratify=y, random_state=42)

supervised = make_pipeline(prep, SVC(kernel="rbf", C=1.0, class_weight="balanced")).fit(X_train, y_train)
print("Supervised SVM:")
print(classification_report(y_test, supervised.predict(X_test), target_names=["normal", "attack"], digits=3))

one_class = make_pipeline(clone(prep), OneClassSVM(kernel="rbf", nu=0.02, gamma="scale"))
one_class.fit(X_train[y_train == 0])                                 # trained on normal traffic only
pred = (one_class.predict(X_test) == -1).astype(int)                 # -1 = outlier -> alert
print("One-class SVM:")
print(classification_report(y_test, pred, target_names=["normal", "attack"], digits=3))

nu is an upper bound on the share of training points treated as outliers, so it roughly sets the false-alarm rate on normal traffic.

Common Pitfalls

  • Random train/test splits of time-ordered traffic; split by time to mimic deployment.

  • Training on outdated datasets whose attacks no longer look like current ones.

  • Ignoring alert volume: a 1% false-positive rate on millions of flows overwhelms analysts.

  • Adversaries adapting to the model; combine with rules and other detectors.

60

Propose an SVM application in healthcare for disease diagnosis.

A good SVM application in healthcare is a diagnostic support model that flags patients at high risk of a disease from routinely collected data, for example detecting chronic kidney disease or diabetes from lab results and vital signs, or classifying tumours as malignant or benign from imaging-derived features. SVMs suit these problems because clinical datasets are often small to medium, have many correlated numeric features, and need good generalization from limited labels. The model supports clinicians rather than replacing them, so evaluation focuses on sensitivity, specificity, calibration and external validation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Proposal: Malignancy Screening from Biopsy Image Features

Element Design
Goal Flag likely malignant fine-needle aspirate samples for priority pathologist review
Data Numeric features of cell nuclei (radius, texture, concavity…) per sample
Model Standardized features + RBF SVM, tuned with stratified CV
Outputs Calibrated probability of malignancy, plus a threshold chosen with clinicians
Primary metric Sensitivity (recall) at a fixed acceptable specificity; ROC AUC
Validation Hold-out test set; later, external data from other hospitals
Governance Privacy (de-identified data), bias checks across groups, documentation, regulatory review if used clinically

Code Example

import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import confusion_matrix, roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

data = load_breast_cancer()
X, y = data.data, (data.target == 0).astype(int)              # 1 = malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight="balanced")),
                    {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.001, 0.01, 0.1]},
                    cv=cv, scoring="roc_auc")
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV ROC AUC:", round(grid.best_score_, 3))

calibrated = CalibratedClassifierCV(grid.best_estimator_, method="sigmoid", cv=cv).fit(X_train, y_train)
proba = calibrated.predict_proba(X_test)[:, 1]
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 3))

# Choose the threshold for high sensitivity using training-fold predictions, not the test set
train_proba = calibrated.predict_proba(X_train)[:, 1]
thresholds = np.linspace(0.05, 0.5, 10)
sens = [((train_proba >= t) & (y_train == 1)).sum() / (y_train == 1).sum() for t in thresholds]
threshold = max((t for t, s in zip(thresholds, sens) if s >= 0.98), default=thresholds[0])
tn, fp, fn, tp = confusion_matrix(y_test, (proba >= threshold).astype(int)).ravel()
print(f"threshold {threshold:.2f}: sensitivity {tp / (tp + fn):.3f}, specificity {tn / (tn + fp):.3f}")

For a stricter threshold choice, use out-of-fold predictions (cross_val_predict) instead of in-sample probabilities.

Risks to Address

  • Dataset shift: different scanners, labs or populations change feature distributions.

  • Label quality: diagnoses must be confirmed (e.g. by histopathology).

  • Patient-level splits: several samples from one patient must stay in the same split.

  • Explainability: provide feature attributions (e.g. permutation importance or SHAP) for clinician review.

Common Pitfalls

  • Optimizing accuracy when missed disease is far more costly than a false alarm.

  • Reporting only internal validation results.

61

What are recent advances in SVMs, and what do they mean for ML?

SVM research has matured, and recent progress is mostly about making kernel methods scale, combining them with learned representations, and connecting them to deep learning theory, rather than new core formulations. Key developments include fast large-scale solvers and kernel approximations (random features, Nyström), GPU implementations, linear SVMs on deep embeddings, theoretical links between wide neural networks and kernels (the neural tangent kernel), and quantum kernel research. For practitioners, this means SVMs are now a strong option on top of pretrained features and on small-to-medium structured problems, while deep learning dominates raw perception tasks.

Read full answer

Applies toscikit-learn 1.5+, cuML (optional) (Python 3.12)

Developments

Area Advance Implication
Scalability Random Fourier features, Nyström, budgeted/online SVMs Kernel-like accuracy at near-linear cost
Hardware GPU SVMs (e.g. cuML, ThunderSVM) Much faster training on large data
Representations Linear SVM / logistic “probes” on pretrained embeddings Strong results with little labelled data
Theory Neural tangent kernel; wide networks behave like kernel machines Kernel methods help explain deep learning
Robustness Adversarially robust and distributionally robust SVM formulations Better behaviour under shift
Interpretability Sparse (L1) SVMs, SHAP explanations Use in regulated domains
Quantum kernels Quantum feature maps + classical SVM Research stage; no proven practical advantage
Tooling Better hyperparameter search (successive halving, Bayesian) Easier tuning

Code Example: Random Fourier Features vs Exact RBF

import time

from sklearn.datasets import make_classification
from sklearn.kernel_approximation import RBFSampler
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

X, y = make_classification(n_samples=25000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

for name, model in [
    ("exact RBF SVC", make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma=0.05))),
    ("RFF(2000) + LinearSVC", make_pipeline(StandardScaler(), RBFSampler(gamma=0.05, n_components=2000,
                                                                        random_state=42), LinearSVC(C=1.0))),
]:
    start = time.perf_counter()
    model.fit(X_train, y_train)
    print(f"{name:24s} {time.perf_counter() - start:6.1f}s  accuracy {model.score(X_test, y_test):.3f}")

Random Fourier features approximate the RBF kernel with explicit cosine features, so a fast linear solver can be used; accuracy approaches the exact kernel as the number of components grows.

Implications for ML Practice

  • Use SVMs on embeddings from foundation models for small labelled datasets.

  • For tabular data, compare with gradient-boosted trees; SVMs win mainly on smaller, cleaner, high-dimensional problems.

  • Kernel theory remains valuable for understanding generalization in deep networks.

Interview Tips

  • Distinguish practical advances (scaling, embeddings, GPUs) from research directions (NTK, quantum kernels).

  • Be ready to explain when you would still pick an SVM today.

62

How can deep learning techniques be integrated with SVMs?

Deep learning and SVMs are combined mainly by using a neural network as a feature extractor and an SVM as the classifier: a pretrained CNN or transformer turns raw images or text into embeddings, and a linear or RBF SVM is trained on those embeddings. This works especially well with small labelled datasets, where fine-tuning the whole network would overfit. Other integrations include replacing the softmax layer with a hinge-loss (SVM) output layer trained end to end, and deep kernel learning, where a network learns the input representation for a kernel.

Read full answer

Applies toscikit-learn 1.5+, PyTorch 2.x, torchvision 0.19+ (Python 3.12)

Integration Patterns

Pattern How When
Embeddings + SVM Freeze a pretrained network, extract features, train an SVM Small datasets, fast experiments
Hinge-loss output layer Use nn.MultiMarginLoss (multi-class hinge) instead of cross-entropy End-to-end training with margin objective
Deep kernel learning Network learns features that feed an RBF kernel Research; Gaussian-process style models
Ensemble Combine CNN probabilities with SVM scores Diversity in errors

Code Example: CNN Embeddings + SVM

import numpy as np
import torch
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from torchvision import datasets, models

torch.manual_seed(42)
weights = models.ResNet18_Weights.DEFAULT
backbone = models.resnet18(weights=weights)                     # downloads pretrained weights
backbone.fc = torch.nn.Identity()                               # output 512-d embeddings
backbone.eval()
preprocess = weights.transforms()

data = datasets.CIFAR10(root="data", train=True, download=True)   # downloads CIFAR-10
idx = np.random.default_rng(42).choice(len(data), 1000, replace=False)   # small labelled set

with torch.no_grad():
    feats, labels = [], []
    for start in range(0, len(idx), 100):
        batch = [data[i] for i in idx[start:start + 100]]
        x = torch.stack([preprocess(img) for img, _ in batch])
        feats.append(backbone(x).numpy())
        labels.extend(label for _, label in batch)
X, y = np.vstack(feats), np.array(labels)

svm = make_pipeline(StandardScaler(), LinearSVC(C=0.01, max_iter=5000))
print("5-fold accuracy, ResNet18 features + linear SVM:", round(cross_val_score(svm, X, y, cv=5).mean(), 3))

Code Example: Hinge-Loss Output Layer

import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(512, 128), nn.ReLU(), nn.Linear(128, 10))
loss_fn = nn.MultiMarginLoss(margin=1.0)                      # Crammer-Singer-style multi-class hinge
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)   # weight decay ≈ the SVM's L2 term

x, y = torch.randn(64, 512), torch.randint(0, 10, (64,))
loss = loss_fn(model(x), y)
loss.backward()
optimizer.step()
print("hinge loss:", round(loss.item(), 4))

Common Pitfalls

  • Using the backbone’s training-time augmentations or forgetting eval(), which makes embeddings noisy.

  • Mismatched preprocessing between feature extraction and the network’s pretraining.

  • Fine-tuning a large network on a few hundred samples when frozen embeddings + SVM would generalize better.

63

How are SVMs used in reinforcement learning contexts?

SVMs are not a standard component of modern reinforcement learning (RL), where neural networks dominate, but they have been used in several supporting roles: as function approximators for value functions (support vector regression for Q-values in batch methods such as fitted Q-iteration), as policy classifiers in classification-based policy iteration (predict the best action for a state), in imitation learning to copy an expert’s actions, and for safety or feasibility classification of states. Their appeal is good generalization from small amounts of data; their limitations are batch training and poor scaling to the large, changing datasets that RL generates.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Roles

Role How the SVM is used Example
Value approximation SVR fits Q(s, a) targets in batch RL Fitted Q-iteration on logged data
Policy as classifier SVC maps states to the best action from rollouts Classification-based approximate policy iteration
Imitation learning SVC learns expert actions from demonstrations Behaviour cloning for a simple controller
State classification One-class or binary SVM flags unsafe states Safety layer that blocks risky actions
Inverse RL Max-margin formulation of reward learning Apprenticeship learning (Abbeel & Ng)

Code Example: Behaviour Cloning with an SVM

An expert balances a simple 1D system: the action pushes left or right depending on position and velocity. The SVM learns the expert’s policy from logged states.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

rng = np.random.default_rng(42)

def expert_action(pos, vel):
    return int(pos + 0.5 * vel < 0)                  # 1 = push right, 0 = push left

def step(pos, vel, action):
    vel = vel + (0.1 if action == 1 else -0.1) + rng.normal(0, 0.02)
    return pos + vel * 0.1, vel

# Collect expert demonstrations
states, actions = [], []
for _ in range(200):
    pos, vel = rng.uniform(-1, 1), rng.uniform(-1, 1)
    for _ in range(30):
        a = expert_action(pos, vel)
        states.append([pos, vel]); actions.append(a)
        pos, vel = step(pos, vel, a)
X, y = np.array(states), np.array(actions)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

policy = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=10)).fit(X_train, y_train)
print("action agreement with expert:", round(policy.score(X_test, y_test), 3))

# Roll out the learned policy and measure how close it keeps the system to 0
def rollout(policy_fn, episodes=50):
    errors = []
    for _ in range(episodes):
        pos, vel = rng.uniform(-1, 1), rng.uniform(-1, 1)
        for _ in range(50):
            pos, vel = step(pos, vel, policy_fn(pos, vel))
        errors.append(abs(pos))
    return np.mean(errors)

print("mean |position| expert:", round(rollout(expert_action), 3))
print("mean |position| SVM:   ", round(rollout(lambda p, v: int(policy.predict([[p, v]])[0])), 3))

Limitations

  • RL data is non-stationary and grows continuously; kernel SVMs must be retrained in batch.

  • Behaviour cloning suffers from compounding errors when the agent visits states the expert never did (DAgger addresses this).

Common Pitfalls

  • Judging an imitation policy only by action accuracy; evaluate it by rollouts.

  • Using SVMs where deep RL with experience replay is clearly the better tool.

64

What role have SVMs played in developing self-driving cars?

SVMs played an important role in early autonomous-driving research, before deep learning became dominant in the mid-2010s. The classic example is pedestrian detection with HOG features and a linear SVM (Dalal and Triggs, 2005), which became a standard baseline; SVMs were also used for vehicle and traffic-sign classification, lane and road-surface classification, and obstacle classification from LiDAR features. Today, convolutional neural networks and transformers handle perception, while SVMs remain useful for small, well-defined subproblems, fast baselines and resource-constrained components.

Read full answer

Applies toscikit-learn 1.5+, scikit-image 0.24+ (Python 3.12)

Historical and Current Uses

Task SVM approach Today
Pedestrian detection HOG features + linear SVM, sliding window CNN/transformer detectors
Traffic-sign recognition HOG or colour histograms + SVM CNNs (e.g. trained on GTSRB)
Vehicle detection Haar/HOG features + SVM Single-stage detectors
Road / terrain classification Colour and texture features + SVM (e.g. off-road robots) Segmentation networks
LiDAR object classification Hand-crafted point-cloud features + SVM Point-cloud networks (PointNet-style, BEV)
Driver monitoring Eye/face features + SVM for drowsiness CNNs on camera input

Code Example: HOG + Linear SVM Classifier

The example classifies small grayscale patches as “pedestrian-like” (vertical body structure) or background, using synthetic images so it runs anywhere.

import numpy as np
from skimage.feature import hog
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC

rng = np.random.default_rng(42)

def synthetic_patch(person):
    img = rng.normal(0.5, 0.15, (64, 32))
    if person:
        c = rng.integers(12, 20)
        img[8:16, c - 3:c + 3] += 0.4                          # head
        img[16:44, c - 5:c + 5] += 0.3                         # torso
        img[44:62, c - 5:c - 1] += 0.3                         # legs
        img[44:62, c + 1:c + 5] += 0.3
    return np.clip(img, 0, 1)

patches = [synthetic_patch(i % 2 == 0) for i in range(1200)]
labels = np.array([int(i % 2 == 0) for i in range(1200)])
features = np.array([hog(p, orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2)) for p in patches])

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.25, stratify=labels,
                                                    random_state=42)
clf = make_pipeline(StandardScaler(), LinearSVC(C=0.01)).fit(X_train, y_train)
print("HOG feature length:", features.shape[1])
print(classification_report(y_test, clf.predict(X_test), target_names=["background", "pedestrian"]))

Real systems train on datasets such as INRIA Person or Caltech Pedestrians and slide the detector over image pyramids, followed by non-maximum suppression.

Why Deep Learning Replaced SVMs Here

  • CNNs learn features directly from pixels, outperforming hand-crafted HOG features.

  • Detection, segmentation and tracking can be learned jointly.

  • GPUs make large models fast enough for real time.

Interview Tips

  • Cite HOG + SVM as the classic pedestrian detector and explain sliding windows.

  • Note that safety-critical systems still value simple, verifiable components.

65

How can SVMs be used for domain adaptation in transfer learning?

Domain adaptation trains a model on a labelled source domain and applies it to a related target domain whose data distribution differs, for example reviews of books (source) and electronics (target). With SVMs, the main approaches are: reweighting source samples so they resemble the target (importance weighting via sample_weight), aligning feature distributions before training (e.g. CORAL or a shared feature space), adapting a source SVM with a small number of labelled target samples (adaptive SVM, which regularizes the target model toward the source model), and transductive or self-training SVMs that use unlabelled target data.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x, SciPy (Python 3.12)

Approaches

Approach Idea Needs
Importance weighting Weight source points by p_target(x) / p_source(x), estimated with a domain classifier Unlabelled target data
Feature alignment (CORAL) Match the covariance of source features to the target’s Unlabelled target data
Augmented features (“frustratingly easy” domain adaptation) Copy features into shared, source-only and target-only blocks Some labelled target data
Adaptive SVM Learn a correction on top of the source SVM’s decision function Few labelled target samples
Transductive / self-training SVM Iteratively label confident target points Unlabelled target data

Code Example: Importance Weighting and CORAL

import numpy as np
from scipy import linalg
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

rng = np.random.default_rng(42)

def make_domain(n, shift, scale):
    X = rng.normal(size=(n, 2)) * scale + shift
    y = (X[:, 0] + X[:, 1] > shift.sum()).astype(int)
    return X, y

X_src, y_src = make_domain(800, np.array([0.0, 0.0]), np.array([1.0, 1.0]))
X_tgt, y_tgt = make_domain(800, np.array([1.5, 0.5]), np.array([0.6, 1.4]))   # shifted, rescaled

baseline = SVC(kernel="rbf").fit(X_src, y_src)
print("source-only on target:", round(baseline.score(X_tgt, y_tgt), 3))

# 1. Importance weights from a domain classifier: p(target | x) / p(source | x)
domain_X = np.vstack([X_src, X_tgt])
domain_y = np.r_[np.zeros(len(X_src)), np.ones(len(X_tgt))]
dc = LogisticRegression().fit(domain_X, domain_y)
p = dc.predict_proba(X_src)[:, 1]
weights = np.clip(p / (1 - p), 0.05, 20)
weighted = SVC(kernel="rbf").fit(X_src, y_src, sample_weight=weights / weights.mean())
print("importance-weighted:  ", round(weighted.score(X_tgt, y_tgt), 3))

# 2. CORAL: whiten source features, then re-colour with target statistics
def coral(Xs, Xt, eps=1.0):
    cs = np.cov(Xs, rowvar=False) + eps * np.eye(Xs.shape[1])
    ct = np.cov(Xt, rowvar=False) + eps * np.eye(Xt.shape[1])
    Xs_c = (Xs - Xs.mean(0)) @ linalg.fractional_matrix_power(cs, -0.5) @ linalg.fractional_matrix_power(ct, 0.5)
    return np.real(Xs_c) + Xt.mean(0)

aligned = SVC(kernel="rbf").fit(coral(X_src, X_tgt), y_src)
print("CORAL-aligned:        ", round(aligned.score(X_tgt, y_tgt), 3))

Target labels are used here only for evaluation; in practice, keep a small labelled target set for validation, because which method helps depends on the kind of shift.

Common Pitfalls

  • Tuning hyperparameters on source validation data only, which ignores the shift.

  • Extreme importance weights that let a few points dominate; clip or normalize them.

  • Assuming adaptation always helps: if the label relationship changes between domains, reweighting cannot fix it.

66

What are the potential uses of SVMs in recommendation systems?

SVMs can play several roles in recommendation systems, though they are rarely the core engine today: they can classify whether a user will engage with an item from user, item and context features (content-based or hybrid scoring), rank candidates with Ranking SVM (learning from pairwise preferences such as “clicked A over B”), and filter or re-rank results, for example detecting spam or inappropriate items. Their strengths are high-dimensional sparse features and good generalization from limited data; their weakness is scaling to the billions of user-item pairs that collaborative-filtering and deep models handle.

Read full answer

Applies toscikit-learn 1.5+, NumPy 2.x (Python 3.12)

Roles

Role How Example
Content-based classifier One linear SVM per user on item text features (TF-IDF) Recommend news articles similar to those a user liked
Hybrid engagement model Features from user, item, context, plus embeddings Predict click on a recommended product
Ranking SVM Train on feature differences x_A - x_B labelled +1 if A was preferred Learning-to-rank for search results
Cold-start Uses item and user attributes instead of interaction history New items with no ratings
Filtering Classify items as spam, low quality or unsafe Clean candidate lists

Code Example: Pairwise Ranking SVM

import numpy as np
from sklearn.svm import LinearSVC

rng = np.random.default_rng(42)
n_items, n_features = 200, 6
items = rng.normal(size=(n_items, n_features))
true_pref = np.array([1.5, -1.0, 0.8, 0.0, 0.3, -0.5])           # hidden user taste
utility = items @ true_pref + rng.normal(0, 0.3, n_items)

# Build pairwise training data from observed preferences
pairs_X, pairs_y = [], []
for _ in range(2000):
    a, b = rng.choice(n_items, 2, replace=False)
    diff = items[a] - items[b]
    label = 1 if utility[a] > utility[b] else -1
    pairs_X.extend([diff, -diff])                                 # symmetric pairs keep classes balanced
    pairs_y.extend([label, -label])

ranker = LinearSVC(C=1.0, fit_intercept=False).fit(np.array(pairs_X), np.array(pairs_y))
scores = items @ ranker.coef_[0]                                  # score items with the learned weights

top10_learned = set(np.argsort(scores)[-10:])
top10_true = set(np.argsort(utility)[-10:])
print("learned weights:", ranker.coef_[0].round(2))
print("overlap of top-10 lists:", len(top10_learned & top10_true), "of 10")
print("rank correlation:", round(np.corrcoef(np.argsort(np.argsort(scores)),
                                            np.argsort(np.argsort(utility)))[0, 1], 3))

The learned weights should point in the same direction as the hidden preference vector, so ranking by them recovers most of the best items.

Limitations

  • Pairwise training data grows quadratically; sample pairs carefully.

  • Collaborative signals (who liked what) are better captured by matrix factorization or two-tower neural models.

  • Kernel SVMs are too slow for large candidate sets; use linear SVMs.

Common Pitfalls

  • Evaluating with accuracy instead of ranking metrics (NDCG, MAP, recall@k).

  • Leaking future interactions into training features.

67

How might quantum machine learning research affect SVM algorithms?

Quantum machine learning research touches SVMs mainly through quantum kernels: a quantum circuit maps each data point to a quantum state, and the kernel value is the overlap between two such states, estimated on quantum hardware or a simulator. The resulting kernel matrix is then used by an ordinary classical SVM solver. The hope is that some quantum feature maps produce kernels that are hard to compute classically yet useful for certain data. So far, no practical advantage over classical kernels on real-world datasets has been established; the field faces noise, sampling cost and the problem that many quantum kernels concentrate toward constant values as the number of qubits grows.

Read full answer

Applies toQuantum kernel research as of 2026; Qiskit Machine Learning (optional), scikit-learn 1.5+ (Python 3.12)

How a Quantum Kernel SVM Works

  1. Encode each feature vector into a quantum state with a parameterized circuit (the feature map).

  2. Estimate K(x, z) = |<phi(x)|phi(z)>|^2 for every pair by running circuits and counting measurement outcomes.

  3. Train a classical SVM on the precomputed kernel matrix.

  4. Predict by estimating kernel values between new points and the support vectors.

Potential Impacts and Open Problems

Topic Status
Expressive feature maps Kernels classical computers may struggle to evaluate; benefit on practical data unproven
Provable advantage Shown for specially constructed problems (e.g. based on discrete logarithms), not typical datasets
Exponential concentration Kernel values become nearly identical for many qubits, so the SVM cannot distinguish points
Sampling cost Each kernel entry needs many circuit runs; n^2 entries for training
Hardware noise Current devices add errors to kernel estimates
Quantum-inspired classical algorithms Some proposed speed-ups were matched by classical “dequantized” methods

Code Example: Classical Simulation of a Simple Quantum-Style Kernel

The kernel below is the fidelity of product states produced by single-qubit rotations, which can be computed classically in closed form; it illustrates the workflow without quantum hardware.

import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC


def product_state_fidelity_kernel(A, B):
    """Each feature x_k is encoded as a qubit rotated by angle x_k; K = prod_k cos^2((a_k - b_k) / 2)."""
    diff = A[:, None, :] - B[None, :, :]
    return np.prod(np.cos(diff / 2) ** 2, axis=2)


X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
X = MinMaxScaler(feature_range=(0, np.pi)).fit_transform(X)     # angles in [0, pi]

K = product_state_fidelity_kernel(X, X)
print("kernel PSD:", np.linalg.eigvalsh(K).min() > -1e-9)

for name, clf in [("fidelity kernel", SVC(kernel=product_state_fidelity_kernel)),
                  ("classical RBF", SVC(kernel="rbf"))]:
    print(f"{name:16s} CV accuracy {cross_val_score(clf, X, y, cv=5).mean():.3f}")

Entangling feature maps (as in Qiskit’s ZZFeatureMap with FidelityQuantumKernel and QSVC) cannot in general be simulated this cheaply for many qubits; that is where any quantum advantage would have to come from.

Interview Tips

  • Explain that the SVM itself stays classical; only the kernel computation changes.

  • Mention benchmarking against tuned classical kernels and the concentration problem.

  • Avoid claiming practical speed-ups that have not been demonstrated.

68

How can SVMs be combined with other models in an ensemble?

SVMs combine well with other models because they make different kinds of errors from tree ensembles, linear models and neural networks. The main options are voting (average the predictions or probabilities of several models), stacking (train a meta-model on the out-of-fold predictions of base models such as an SVM, a random forest and gradient boosting), and pipelines where another model produces features for the SVM (for example neural-network embeddings or tree-based leaf encodings). Ensembles help most when the members are individually good and their errors are not strongly correlated.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Combination Strategies

Strategy How the SVM participates Notes
Hard voting Contributes a class label Simple; no probabilities needed
Soft voting Contributes a probability Needs calibrated SVM outputs
Stacking Its decision_function becomes a meta-feature Meta-model learns how much to trust each member
Feature pipeline SVM trained on learned features (CNN embeddings, PCA) Often strong for small labelled datasets
Cascade Cheap model filters easy cases; SVM handles hard ones Reduces latency

Code Example: Voting and Stacking

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import (HistGradientBoostingClassifier, RandomForestClassifier, StackingClassifier,
                              VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

svm = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, probability=True, random_state=42))
rf = RandomForestClassifier(n_estimators=300, random_state=42)
gb = HistGradientBoostingClassifier(random_state=42)
lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

voting = VotingClassifier([("svm", svm), ("rf", rf), ("gb", gb)], voting="soft")
stacking = StackingClassifier(
    estimators=[("svm", svm), ("rf", rf), ("gb", gb)],
    final_estimator=LogisticRegression(),
    stack_method="auto",           # uses predict_proba where available
    cv=5,                          # meta-features come from out-of-fold predictions
)

for name, model in [("SVM", svm), ("RandomForest", rf), ("GradientBoosting", gb), ("LogReg", lr),
                    ("Soft voting", voting), ("Stacking", stacking)]:
    scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
    print(f"{name:17s} ROC AUC {scores.mean():.4f} ± {scores.std():.4f}")

On an easy dataset the differences may be within noise; ensembles pay off more on harder problems with diverse members.

Practical Tips

  • Check that members disagree on some examples (low error correlation); identical models add nothing.

  • Use out-of-fold predictions for stacking (scikit-learn does this) to avoid leakage.

  • Weigh the gain against the extra latency and maintenance of several models.

Common Pitfalls

  • Soft voting with uncalibrated SVM scores.

  • Stacking evaluated on the same data used to create meta-features.

  • Adding the SVM without scaling inside its own pipeline.

69

How do bagging and boosting work with SVM classifiers?

Bagging trains several SVMs on different bootstrap samples (and optionally feature subsets) and combines them by voting or averaging, which mainly reduces variance and allows training on large data in parallel pieces. Boosting trains SVMs sequentially, each focusing on the examples the previous ones got wrong (through sample weights in AdaBoost), and combines them with weights, which mainly reduces bias. Because a tuned SVM is already a strong, low-variance learner, the gains from either are usually modest; they help most with large or noisy data, or with deliberately weakened SVMs.

Read full answer

Applies toscikit-learn 1.6+ (Python 3.12)

Comparison

Aspect Bagging SVMs Boosting SVMs (AdaBoost)
Training Parallel, independent Sequential
Data per model Bootstrap or subsample All data with updated weights
Main effect Lower variance, faster on large n Lower bias
Base SVM Normal or subsampled Should be weak (small C, linear, or few features)
Risk Little Overfitting noisy labels; strong SVMs leave little to correct
scikit-learn BaggingClassifier(SVC()) AdaBoostClassifier(SVC(...)); SVC accepts sample_weight

How AdaBoost Uses Sample Weights

  1. Start with equal weights.

  2. Train an SVM with those weights (fit(X, y, sample_weight=w)), which scales each point’s penalty C.

  3. Increase the weights of misclassified points, decrease the others.

  4. Give each SVM a vote proportional to its accuracy and repeat.

Code Example

from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)
X = StandardScaler().fit_transform(X)   # for brevity; fit inside each fold in real projects

models = {
    "single SVC rbf": SVC(kernel="rbf", C=1.0),
    "bagging (15 SVCs, 30% samples)": BaggingClassifier(SVC(kernel="rbf", C=1.0), n_estimators=15,
                                                        max_samples=0.3, n_jobs=-1, random_state=42),
    "weak linear SVC": SVC(kernel="linear", C=0.01),
    "AdaBoost (30 weak linear SVCs)": AdaBoostClassifier(SVC(kernel="linear", C=0.01), n_estimators=30,
                                                         random_state=42),
}
for name, model in models.items():
    print(f"{name:32s} CV accuracy {cross_val_score(model, X, y, cv=5, n_jobs=-1).mean():.3f}")

Since scikit-learn 1.6, AdaBoostClassifier uses the SAMME algorithm, which needs only class predictions, so the SVMs do not need probability=True (older versions defaulted to SAMME.R, which requires probabilities).

When to Use

  • Bagging: large datasets (train subsets in parallel), noisy data, or when you need an ensemble’s variance reduction.

  • Boosting: rarely the first choice for SVMs; gradient-boosted trees are usually better boosted learners.

Common Pitfalls

  • Boosting a strong RBF SVM: after the first round there is little left to correct, and later rounds overfit noise.

  • Expecting large gains over a single well-tuned SVM; always compare against it.

70

Describe a scenario where an SVM is a weak learner in an ensemble.

An SVM can serve as a weak learner when it is deliberately constrained, for example trained on a small random subset of samples or features, heavily regularized, or limited to a linear kernel, so that each model is only modestly accurate but different from the others. A realistic scenario is a large text or sensor dataset where one exact kernel SVM is too slow: training many small SVMs on random subsets (bagging or random subspaces) and combining their votes gives a faster, often more robust model. SVMs are less common in boosting because they are strong, stable learners, which limits what boosting can add.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Scenario: Fraud Detection on 200,000 Transactions

  • One RBF SVM on all rows would take a very long time to train.

  • Instead, train 20 RBF SVMs, each on 5% of the rows and 70% of the features, in parallel.

  • Each model is individually weaker, but their errors differ; averaging their scores reduces variance.

  • Training cost per model grows roughly quadratically with its sample size, so 20 models on 5% each cost far less than one model on 100%.

Ways to Make SVMs “Weak”

Constraint Effect
Small random sample per model Fast; diverse boundaries
Random feature subspace Different views of the data
Small C Simple, high-bias boundaries
Linear kernel on non-linear data Limited capacity

Code Example: Bagged SVMs with Random Subspaces

import time

from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = make_classification(n_samples=40000, n_features=30, n_informative=12, weights=[0.9, 0.1],
                           flip_y=0.02, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)

weak_svm = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, class_weight="balanced"))
ensemble = BaggingClassifier(
    estimator=weak_svm,
    n_estimators=20,
    max_samples=0.05,          # each SVM sees 5% of the rows
    max_features=0.7,          # and 70% of the features
    n_jobs=-1,
    random_state=42,
)

start = time.perf_counter()
ensemble.fit(X_train, y_train)
print(f"ensemble trained in {time.perf_counter() - start:.1f}s")

single_small = weak_svm.fit(X_train[:1500], y_train[:1500])          # one weak learner alone
print("single weak SVM ROC AUC:", round(roc_auc_score(y_test, single_small.decision_function(X_test)), 3))
print("bagged SVMs ROC AUC:   ", round(roc_auc_score(y_test, ensemble.decision_function(X_test)), 3))

BaggingClassifier.decision_function averages the members’ scores, which works for ranking metrics without needing probabilities.

When It Helps (and When It Doesn’t)

  • Helps: large datasets, noisy labels, high variance between subsets, parallel hardware.

  • Doesn’t help much: small, clean datasets where a single tuned SVM is already stable; trees benefit more from bagging because they are less stable.

Common Pitfalls

  • Fitting the scaler outside the bagged estimator; keep it inside so each member scales its own sample.

  • Using probability=True in every member, which multiplies training time.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 70 SVM answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects