What is a Support Vector Machine (SVM) in machine learning?
KnownRevise
A Support Vector Machine (SVM) is a supervised learning algorithm that classifies data by finding the boundary (hyperplane) that separates the classes with the widest possible margin. Only the training points closest to the boundary, the support vectors, determine where it lies. With the kernel trick, SVMs can learn non-linear boundaries efficiently, and a variant called Support Vector Regression (SVR) handles numeric targets. SVMs work very well on small to medium, high-dimensional datasets such as text.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Core Ideas
Idea
Meaning
Hyperplane
The decision boundary w·x + b = 0; a line in 2D, a plane in 3D
Margin
Distance between the boundary and the closest points of each class
Support vectors
Points on or inside the margin; removing other points does not change the model
Allows some violations; large C = fewer violations, less regularization
Kernel
Computes similarity in a higher-dimensional space without building it: linear, polynomial, RBF, sigmoid
Hinge loss
Penalizes points on the wrong side of their margin
The soft-margin objective, in words: minimize (1/2)||w||^2 (which maximizes the margin) plus C times the total slack (how far points violate the margin).
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf")) # scaling is essential for SVMs
grid = GridSearchCV(pipe, {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.1]}, cv=5)
grid.fit(X_train, y_train)
print("best parameters:", grid.best_params_)
svc = grid.best_estimator_.named_steps["svc"]
print("support vectors per class:", svc.n_support_)
print(classification_report(y_test, grid.predict(X_test)))
Strengths and Weaknesses
Strengths
Weaknesses
Effective in high dimensions (text, genomics)
Kernel SVM training scales roughly O(n^2) to O(n^3): slow beyond tens of thousands of samples
Margin maximization gives good generalization
Needs feature scaling and tuning of C and gamma
Flexible boundaries via kernels
No native probabilities (Platt scaling is extra cost)
Model depends only on support vectors
Less interpretable with non-linear kernels
Interview Tips
Explain the margin, support vectors, C and the kernel trick in plain words.
Mention LinearSVC or SGDClassifier(loss="hinge") for large datasets.
Note that decision_function values are distances to the boundary, not probabilities.
2
What is a hyperplane in SVM?
KnownRevise
A hyperplane is a flat decision boundary with one dimension fewer than the feature space: a line in 2D, a plane in 3D, and a “hyperplane” in higher dimensions. In an SVM it is defined by w·x + b = 0, where w is a weight vector perpendicular to the boundary and b shifts it from the origin. Points with w·x + b > 0 are assigned to one class and points with a negative value to the other, and the SVM chooses the hyperplane that leaves the widest margin between the classes.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Geometry
Element
Meaning
w
Normal vector: points toward the positive class; its direction sets the boundary’s orientation
b
Bias: shifts the hyperplane
w·x + b
Signed score; the sign gives the class
| (w·x + b) / ||w|| | Signed distance of x from the hyperplane | | w·x + b = ±1 | The two margin hyperplanes passing through the support vectors |
With a non-linear kernel, the boundary is still a hyperplane, but in the implicit high-dimensional feature space; in the original space it appears curved.
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0, n_clusters_per_class=1,
class_sep=2.0, random_state=42)
model = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0)).fit(X, y)
svc = model.named_steps["svc"]
w, b = svc.coef_[0], svc.intercept_[0]
print(f"hyperplane (scaled space): {w[0]:.3f}*x1 + {w[1]:.3f}*x2 + {b:.3f} = 0")
x_new = model.named_steps["standardscaler"].transform(X[:3])
scores = x_new @ w + b
print("scores:", scores.round(3))
print("same as decision_function:", np.allclose(scores, model.decision_function(X[:3])))
print("signed distances:", (scores / np.linalg.norm(w)).round(3))
print("predicted classes:", model.predict(X[:3]))
Common Pitfalls
Reading coef_ as feature importance without scaling the features first.
Expecting coef_ for non-linear kernels: it exists only for the linear kernel.
Thinking a hyperplane can only separate linearly separable data in the original space; kernels change the space.
3
What is the maximum margin classifier in the context of SVM?
KnownRevise
The maximum margin classifier is the linear classifier that separates two perfectly separable classes with the largest possible gap, or margin, between the decision boundary and the nearest points of either class. Among the infinitely many lines that separate the data, it picks the one farthest from both classes, which tends to generalize best. It is the foundation of the SVM: the soft-margin SVM relaxes it for overlapping data, and kernels extend it to non-linear boundaries.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
The Optimization Problem
Boundary: w·x + b = 0. Labels are y = +1 or -1.
Require every point to be on the correct side with a functional margin of at least 1: y_i (w·x_i + b) >= 1.
The geometric margin width is 2 / ||w||. Maximizing it is the same as minimizing (1/2)||w||^2.
So the hard-margin SVM solves: minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1 for all points.
In words: find the smallest weight vector that still puts every point on the right side, at least one unit of “score” away from the boundary; the smaller the weights, the wider the gap.
Hard Versus Soft Margin
Aspect
Hard margin (maximum margin classifier)
Soft margin SVM
Data requirement
Perfectly linearly separable
Any data
Violations allowed
None
Yes, penalized by C × slack
Sensitivity to outliers
Very high: one point can move the boundary
Controlled by C
Practical use
Mostly theoretical
Standard SVM
Code Example
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC
X, y = make_blobs(n_samples=60, centers=2, cluster_std=0.8, random_state=42) # separable data
clf = SVC(kernel="linear", C=1e6) # very large C approximates a hard margin
clf.fit(X, y)
w, b = clf.coef_[0], clf.intercept_[0]
print("w =", w.round(3), "b =", round(b, 3))
print("margin width =", round(2 / np.linalg.norm(w), 3))
print("support vectors:\n", clf.support_vectors_)
print("functional margins of support vectors:",
(np.where(y[clf.support_] == 1, 1, -1) * clf.decision_function(clf.support_vectors_)).round(3)) # about 1
Support vectors sit exactly on the margin, so their functional margin is 1 (up to solver tolerance).
Common Pitfalls
Assuming real data is separable; use a soft margin with tuned C.
Forgetting feature scaling: the margin is measured in feature units, so unscaled features distort it.
Confusing the functional margin (score) with the geometric margin (distance).
4
What are support vectors, and why are they important in SVM?
KnownRevise
Support vectors are the training points that lie on the margin boundaries or inside the margin (including misclassified points). They are the only points that define the SVM’s decision boundary: if you delete any other point and retrain, the model does not change. This makes SVM solutions sparse, which matters for interpretation, memory and prediction speed, since a kernel SVM’s prediction is a weighted sum of kernel values between the new point and each support vector.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Why They Matter
Aspect
Role of support vectors
Decision boundary
Determined entirely by them
Dual coefficients
Only support vectors have non-zero Lagrange multipliers (alpha_i > 0)
Prediction
f(x) = sum over SVs of alpha_i y_i K(x_i, x) + b; more SVs = slower predictions
Model complexity
Many SVs (a large share of the data) can signal overlap, a small C or an overly flexible kernel
Generalization
The leave-one-out error is bounded by the fraction of support vectors
Types
On the margin (0 < alpha_i < C): exactly at functional margin 1.
Margin violators (alpha_i = C): inside the margin or misclassified.
Code Example
import numpy as np
from sklearn.datasets import make_moons
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
for C in (0.1, 1, 100):
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C)).fit(X, y)
svc = model.named_steps["svc"]
at_bound = np.isclose(np.abs(svc.dual_coef_[0]), C).sum()
print(f"C={C:>5}: {len(svc.support_)} support vectors "
f"({len(svc.support_) / len(X):.0%} of data), {at_bound} with alpha = C")
# Removing non-support vectors does not change the model
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1)).fit(X, y)
sv_idx = model.named_steps["svc"].support_
model_sv = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1))
model_sv.named_steps["standardscaler"].fit(X) # keep the same scaling
model_sv.named_steps["svc"].fit(model_sv.named_steps["standardscaler"].transform(X[sv_idx]), y[sv_idx])
print("same predictions:", (model.predict(X) == model_sv.predict(X)).mean())
The last line should print 1.0 or very close to it (tiny differences can come from solver tolerance).
Common Pitfalls
Assuming fewer support vectors is always better: too few can mean overfitting with a very large C on separable data.
Ignoring prediction cost: an RBF SVM with 50,000 support vectors is slow at inference.
Forgetting that support_vectors_ are in the scaled feature space when a scaler is used.
5
What is the difference between linear and non-linear SVM?
KnownRevise
A linear SVM separates classes with a straight hyperplane in the original feature space, while a non-linear SVM uses a kernel (RBF, polynomial) to fit a hyperplane in an implicit higher-dimensional space, which appears as a curved boundary in the original space. Linear SVMs are fast, scale to millions of samples and are interpretable through their weights; non-linear SVMs capture complex patterns but train much more slowly and need careful tuning of kernel parameters.
High-dimensional data (text, many features), large n
Low/medium dimensions with complex boundaries, n up to tens of thousands
Code Example
import time
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = make_moons(n_samples=2000, noise=0.2, random_state=42)
models = {
"LinearSVC": make_pipeline(StandardScaler(), LinearSVC(C=1.0)),
"LinearSVC + poly features": make_pipeline(PolynomialFeatures(3), StandardScaler(), LinearSVC(C=1.0, max_iter=5000)),
"SVC rbf": make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0)),
}
for name, model in models.items():
start = time.perf_counter()
acc = cross_val_score(model, X, y, cv=5).mean()
print(f"{name:28s} accuracy {acc:.3f} ({time.perf_counter() - start:.2f}s)")
The moons are not linearly separable, so the plain linear SVM should score lowest; adding polynomial features or using the RBF kernel lets it follow the curves.
Choosing
Try a linear model first; if it underfits and n is moderate, try an RBF kernel.
For large non-linear problems, use kernel approximations (Nystroem, RBFSampler) followed by a linear SVM.
Common Pitfalls
Using SVC(kernel="linear") on large datasets instead of the much faster LinearSVC.
Assuming an RBF kernel is always better; on high-dimensional data it often adds cost without accuracy.
6
How does the kernel trick work in SVM?
KnownRevise
The kernel trick lets an SVM behave as if the data had been mapped into a high- (even infinite-) dimensional feature space, without ever computing that mapping. It works because the SVM’s dual optimization and its predictions use the data only through inner products between pairs of points; replacing every inner product x·z with a kernel function K(x, z) = φ(x)·φ(z) gives the same result as working in the mapped space, at the cost of computing K in the original space.
Dual problem: maximize sum(alpha_i) - (1/2) sum_i sum_j alpha_i alpha_j y_i y_j (x_i·x_j) subject to 0 <= alpha_i <= C and sum(alpha_i y_i) = 0.
Prediction: f(x) = sum_i alpha_i y_i (x_i·x) + b.
Both contain only inner products, so x_i·x_j can be replaced by K(x_i, x_j).
A function is a valid kernel when its kernel matrix is always positive semi-definite (Mercer’s condition).
Concrete Example
For 2D points, the polynomial kernel K(x, z) = (x·z)^2 equals the inner product of the explicit features φ(x) = (x1^2, sqrt(2) x1 x2, x2^2). Computing K takes one dot product and a square; computing φ grows quickly with the degree and dimension. The RBF kernel corresponds to an infinite-dimensional φ, which could never be computed explicitly.
import numpy as np
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
rng = np.random.default_rng(42)
x, z = rng.normal(size=2), rng.normal(size=2)
def phi(v):
return np.array([v[0] ** 2, np.sqrt(2) * v[0] * v[1], v[1] ** 2])
print("kernel in original space:", round(float(np.dot(x, z) ** 2), 6))
print("dot product after mapping:", round(float(np.dot(phi(x), phi(z))), 6)) # identical
# Circles are not linearly separable in 2D, but are with the degree-2 kernel
X, y = make_circles(n_samples=400, factor=0.4, noise=0.08, random_state=42)
for kernel in ("linear", "poly", "rbf"):
clf = SVC(kernel=kernel, degree=2, coef0=0) if kernel == "poly" else SVC(kernel=kernel)
print(kernel, round(cross_val_score(clf, X, y, cv=5).mean(), 3))
Trade-Offs
Benefit
Cost
Non-linear boundaries with a linear algorithm
Kernel matrix is n × n: memory and time grow quickly with n
Works on non-vector data (strings, graphs) via custom kernels
Kernel and its parameters must be chosen and tuned
Infinite-dimensional features (RBF)
Less interpretable; predictions depend on all support vectors
Common Pitfalls
Thinking the data is actually transformed: only pairwise similarities are computed.
Applying kernels to unscaled features, which distorts distances.
Using exact kernels on very large data; use Nystroem or RBFSampler approximations with a linear model.
7
Which kernels can be used in SVM? Give examples of each.
KnownRevise
A kernel is a function K(x, z) that returns the inner product of two points in some feature space, which lets an SVM learn non-linear boundaries without computing that space explicitly. The standard kernels are linear, polynomial, RBF (Gaussian) and sigmoid; specialized kernels exist for strings, graphs, histograms and other structured data, and you can define custom kernels as long as they are valid (positive semi-definite). Choosing a kernel encodes your assumption about what makes two examples similar.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Kernel Overview
Kernel
Formula
Key parameters
Example use
Linear
x·z
C
Text classification with TF-IDF, high-dimensional sparse data
Polynomial
(gamma * x·z + coef0)^degree
degree, gamma, coef0
Feature interactions, normalized image pixels
| RBF (Gaussian) | exp(-gamma * ||x - z||^2) | gamma, C | Default for non-linear tabular data (moons, sensor data) | | Sigmoid | tanh(gamma * x·z + coef0) | gamma, coef0 | Rarely; resembles a two-layer neural network; not always valid | | Laplacian | exp(-gamma * ||x - z||_1) | gamma | Data with sharp changes | | Chi-squared | Based on histogram differences | gamma | Bag-of-visual-words, histograms | | String / graph kernels | Count shared substrings or substructures | Problem-specific | Protein sequences, molecules |
The RBF kernel in words: similarity is 1 for identical points and decays toward 0 as distance grows; gamma sets how fast it decays.
Code Example
from sklearn.datasets import make_moons
from sklearn.metrics.pairwise import laplacian_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=400, noise=0.25, random_state=42)
configs = {
"linear": SVC(kernel="linear"),
"poly (degree 3)": SVC(kernel="poly", degree=3, coef0=1),
"rbf": SVC(kernel="rbf", gamma="scale"),
"sigmoid": SVC(kernel="sigmoid", coef0=0.0),
"laplacian (custom)": SVC(kernel=laplacian_kernel),
}
for name, svc in configs.items():
scores = cross_val_score(make_pipeline(StandardScaler(), svc), X, y, cv=5)
print(f"{name:20s} accuracy {scores.mean():.3f} ± {scores.std():.3f}")
Any callable that returns the kernel matrix between two sets of rows can be passed as kernel, as shown with laplacian_kernel.
Choosing a Kernel
Start with linear when features are many (text) or n is large; it is fast and hard to beat there.
Use RBF as the default for low- to medium-dimensional non-linear data, tuning C and gamma together.
Use polynomial when interactions of a known degree matter.
Use domain kernels when data is not a fixed-length vector.
Common Pitfalls
Using RBF on very high-dimensional sparse text, where linear is usually as good and much faster.
High polynomial degrees, which overfit and become numerically unstable.
Custom kernels that are not positive semi-definite, which can make the optimization fail.
8
What is a soft margin in SVM, and why is it used?
KnownRevise
A soft margin lets an SVM accept some points inside the margin or on the wrong side of the boundary, paying a penalty for each violation. Real data is rarely perfectly separable and often has noise or outliers, so a hard margin either has no solution or produces a narrow, overfitted boundary. The soft margin adds slack variables that measure each violation and a parameter C that sets how expensive violations are, trading margin width against training errors.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
The Objective
Minimize (1/2)||w||^2 + C * sum(xi_i) subject to y_i (w·x_i + b) >= 1 - xi_i and xi_i >= 0.
xi_i (slack) is 0 for points outside the margin on the correct side, between 0 and 1 for points inside the margin but correctly classified, and greater than 1 for misclassified points.
In words: keep the weights small (wide margin) while keeping the total amount of violation small; C decides which matters more.
Equivalent form: minimize (1/2)||w||^2 + C * sum(max(0, 1 - y_i f(x_i))), which is hinge loss with L2 regularization.
Effect of C
C
Margin
Violations
Risk
Small (e.g. 0.01)
Wide
Many tolerated
Underfitting
Moderate (e.g. 1)
Balanced
Some
Usually good
Large (e.g. 1000)
Narrow
Few
Overfitting to noise and outliers
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=300, n_features=2, n_redundant=0, n_clusters_per_class=1,
flip_y=0.1, class_sep=1.0, random_state=42) # overlapping, noisy labels
for C in (0.01, 0.1, 1, 10, 1000):
model = make_pipeline(StandardScaler(), SVC(kernel="linear", C=C))
cv = cross_val_score(model, X, y, cv=5).mean()
model.fit(X, y)
svc = model.named_steps["svc"]
margin = 2 / np.linalg.norm(svc.coef_)
print(f"C={C:>7}: margin {margin:.3f}, support vectors {len(svc.support_)}, CV accuracy {cv:.3f}")
Expect the margin and the number of support vectors to shrink as C grows; the best C is the one with the highest cross-validated score.
Common Pitfalls
Choosing C by training accuracy, which always favors large C.
Tuning C without scaling features; the right C depends on feature scale.
Using a hard margin (huge C) on noisy data.
9
How does SVM handle multi-class classification?
KnownRevise
SVMs are binary classifiers by design, so multi-class problems are handled by combining several binary SVMs. The two standard strategies are one-vs-rest (OvR), which trains one classifier per class against all others, and one-vs-one (OvO), which trains one classifier for every pair of classes and predicts by voting. There are also direct multi-class formulations (Crammer-Singer). In scikit-learn, SVC uses one-vs-one internally, while LinearSVC uses one-vs-rest.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Strategies
Strategy
Classifiers for K classes
Each trained on
Prediction
Used by
One-vs-rest
K
All data (one class vs the rest)
Highest decision score
LinearSVC (default), OneVsRestClassifier
One-vs-one
K(K-1)/2
Only the two classes’ data
Majority vote
SVC, NuSVC, OneVsOneClassifier
Crammer-Singer
One joint problem
All data
Highest score
LinearSVC(multi_class="crammer_singer")
Trade-offs: OvO trains many more models but each on less data, which suits kernel SVMs whose cost grows faster than linearly with n. OvR trains fewer models but creates imbalanced binary problems.
Code Example
from sklearn.datasets import load_digits
from sklearn.model_selection import cross_val_score
from sklearn.multiclass import OneVsRestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = load_digits(return_X_y=True) # 10 classes
ovo = make_pipeline(StandardScaler(), SVC(kernel="rbf", decision_function_shape="ovo"))
ovr = make_pipeline(StandardScaler(), OneVsRestClassifier(SVC(kernel="rbf")))
linear = make_pipeline(StandardScaler(), LinearSVC(C=0.1)) # one-vs-rest by default
for name, model in [("SVC (one-vs-one)", ovo), ("OneVsRest(SVC)", ovr), ("LinearSVC (OvR)", linear)]:
print(f"{name:18s} accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
ovo.fit(X, y)
print("OvO decision_function shape:", ovo.decision_function(X[:1]).shape) # (1, 45) = 10*9/2 pairs
SVC‘s default decision_function_shape="ovr" only reshapes the one-vs-one scores into one column per class; training is still one-vs-one.
Common Pitfalls
Assuming decision_function_shape="ovr" changes the training strategy.
Using probability=True without need; it adds an internal cross-validation and slows training.
Ignoring class imbalance in OvR; use class_weight="balanced".
10
What are the limitations of SVMs?
KnownRevise
SVMs have several practical limitations: kernel SVMs scale poorly to large datasets, they are sensitive to feature scaling and to the choice of C and kernel parameters, they do not output calibrated probabilities natively, and non-linear models are hard to interpret. They also handle noisy, heavily overlapping or imbalanced data less gracefully without tuning, and they need numeric inputs. Knowing these helps you decide when gradient-boosted trees, logistic regression or neural networks are better choices.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Limitations and Workarounds
Limitation
Why
Workaround
Training time and memory
Kernel SVM training is roughly O(n^2) to O(n^3); kernel matrix is n × n
LinearSVC, SGDClassifier(loss="hinge"), Nystroem + linear SVM, subsampling
Slow prediction with many support vectors
Each prediction evaluates the kernel against every support vector
Fewer SVs (tune C), linear models, kernel approximation
Hyperparameter sensitivity
C, gamma, degree interact strongly
Grid or Bayesian search with cross-validation
Needs scaling
Margins and RBF distances depend on feature scale
StandardScaler in a Pipeline
No native probabilities
Outputs distances to the boundary
Platt scaling (probability=True or CalibratedClassifierCV)
Interpretability
Non-linear decision functions
Linear kernel weights, SHAP, permutation importance
Noisy, overlapping classes
Many support vectors, unstable boundary
Smaller C, better features
Imbalanced data
Margin dominated by the majority class
class_weight="balanced", threshold tuning
Categorical and missing values
Requires numeric, complete inputs
One-hot encoding, imputation
Multi-class
Binary by design
OvO/OvR combinations (built in)
Online learning
Batch solvers
SGDClassifier with partial_fit
Code Example: Training Time Growth
import time
from sklearn.datasets import make_classification
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
for n in (2000, 4000, 8000, 16000):
X, y = make_classification(n_samples=n, n_features=20, flip_y=0.05, random_state=42)
timings = {}
for name, clf in [("SVC rbf", SVC()), ("LinearSVC", LinearSVC())]:
start = time.perf_counter()
make_pipeline(StandardScaler(), clf).fit(X, y)
timings[name] = time.perf_counter() - start
print(n, {k: f"{v:.2f}s" for k, v in timings.items()})
Each doubling of n should make the RBF SVC noticeably more than twice as slow, while LinearSVC grows roughly linearly; exact times depend on the machine.
When to Prefer Other Models
Large tabular datasets: gradient-boosted trees (XGBoost, LightGBM).
Probabilities needed for decisions: logistic regression or calibrated boosting.
Images, audio, raw text: neural networks.
11
What is the objective function of the SVM?
KnownRevise
The SVM objective function balances two goals: a wide margin, achieved by keeping the weight vector small, and few margin violations, measured with slack variables or, equivalently, the hinge loss. For the soft-margin SVM the primal objective is to minimize (1/2)||w||^2 + C * sum(xi_i) subject to each point being on the correct side of its margin up to its slack. The same problem can be written as regularized hinge-loss minimization, and it has a dual form that enables kernels.
| Form | Objective | Notes | |—|—|—| | Primal (constrained) | Minimize (1/2)||w||^2 + C * sum(xi_i) s.t. y_i (w·x_i + b) >= 1 - xi_i, xi_i >= 0 | Quadratic program | | Unconstrained (hinge loss) | Minimize (1/2)||w||^2 + C * sum(max(0, 1 - y_i (w·x_i + b))) | Same as L2-regularized hinge loss; used by SGD and LinearSVC | | Dual | Maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j K(x_i, x_j) s.t. 0 <= alpha_i <= C, sum(alpha_i y_i) = 0 | Depends only on kernel values; solved by SMO in SVC |
In words: - (1/2)||w||^2: the margin width is 2/||w||, so a smaller w means a wider margin (regularization). - Hinge loss: zero for points beyond their margin on the correct side, growing linearly as a point moves into the margin or across the boundary. - C: how much a unit of violation costs compared with margin width; large C = less regularization.
Code Example: Evaluating the Objective
import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=200, n_features=4, random_state=42)
X = StandardScaler().fit_transform(X)
y_pm = np.where(y == 1, 1, -1) # labels as +1 / -1
C = 1.0
svc = SVC(kernel="linear", C=C).fit(X, y)
w, b = svc.coef_[0], svc.intercept_[0]
margins = y_pm * (X @ w + b)
hinge = np.maximum(0, 1 - margins)
print("regularization term:", round(0.5 * w @ w, 4))
print("hinge-loss term:", round(C * hinge.sum(), 4))
print("primal objective:", round(0.5 * w @ w + C * hinge.sum(), 4))
print("points with non-zero slack:", int((hinge > 1e-6).sum()), "| support vectors:", len(svc.support_))
Points with non-zero slack are support vectors; support vectors exactly on the margin have zero slack, so the second count is at least as large as the first.
Variants
Squared hinge (LinearSVC default): penalizes violations quadratically.
L1 penalty on w (LinearSVC(penalty="l1")): sparse weights for feature selection.
SVR: epsilon-insensitive loss, ignoring errors smaller than epsilon.
Common Pitfalls
Forgetting that labels are encoded as ±1 in the formulas.
Comparing C values across differently scaled datasets.
12
What role do Lagrange multipliers play in SVM?
KnownRevise
Lagrange multipliers turn the SVM’s constrained optimization problem (maximize the margin while keeping points on the correct side) into its dual problem. Each training point gets a multiplier alpha_i >= 0 that measures how strongly its constraint pushes on the solution. Points with alpha_i = 0 have no influence, and points with alpha_i > 0 are the support vectors. The dual depends on the data only through inner products, which is what makes the kernel trick possible.
Primal (soft margin): minimize (1/2)||w||^2 + C sum(xi_i) s.t. y_i (w·x_i + b) >= 1 - xi_i, xi_i >= 0.
Lagrangian: add each constraint times a multiplier, alpha_i for the margin constraints and mu_i for xi_i >= 0.
Set the derivatives with respect to w, b and xi to zero:
w = sum_i alpha_i y_i x_i: the weight vector is a weighted sum of training points.
sum_i alpha_i y_i = 0.
alpha_i = C - mu_i, so 0 <= alpha_i <= C.
Substitute back to get the dual: maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j (x_i·x_j).
What the Multipliers Tell You (KKT Conditions)
Value of alpha_i
Point’s position
Role
0
Beyond the margin, correctly classified
Not a support vector
0 < alpha_i < C
Exactly on the margin
Support vector; used to compute b
C
Inside the margin or misclassified
Bounded support vector (margin violator)
The complementary slackness condition, alpha_i * (y_i f(x_i) - 1 + xi_i) = 0, says that a multiplier can be non-zero only when its constraint is exactly active.
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0, n_clusters_per_class=1,
flip_y=0.05, random_state=42)
X = StandardScaler().fit_transform(X)
C = 1.0
svc = SVC(kernel="linear", C=C).fit(X, y)
alpha_y = svc.dual_coef_[0] # alpha_i * y_i for each support vector
alphas = np.abs(alpha_y)
w_from_dual = alpha_y @ svc.support_vectors_ # w = sum alpha_i y_i x_i
print("w (coef_): ", svc.coef_[0].round(4))
print("w from alphas:", w_from_dual.round(4))
print("sum alpha_i y_i =", round(float(alpha_y.sum()), 6)) # about 0
print("on margin (0 < alpha < C):", int(((alphas > 1e-8) & (alphas < C - 1e-8)).sum()),
"| at bound (alpha = C):", int(np.isclose(alphas, C).sum()))
Interview Tips
Explain that the multipliers make the solution sparse: only support vectors have non-zero alpha.
Connect w = sum alpha_i y_i x_i to the kernel trick: predictions become sum alpha_i y_i K(x_i, x) + b.
Mention the box constraint 0 <= alpha_i <= C as where C enters the dual.
13
How is the dual problem solved in SVM optimization?
KnownRevise
The SVM dual problem is a convex quadratic program over one Lagrange multiplier alpha_i per training point: maximize sum(alpha_i) - (1/2) sum_ij alpha_i alpha_j y_i y_j K(x_i, x_j) subject to 0 <= alpha_i <= C and sum(alpha_i y_i) = 0. Because it is convex, any local optimum is global. It is solved with general QP solvers for small problems, or with specialized decomposition methods, mainly Sequential Minimal Optimization (SMO) in libsvm (used by scikit-learn’s SVC), which update a small working set of multipliers at a time.
Optimize a subset of variables while fixing the rest
Early SVM solvers (SVMlight)
SMO
Optimize two multipliers at a time analytically (two are needed to keep sum alpha_i y_i = 0)
libsvm → SVC, SVR, NuSVC
Dual coordinate descent
Update one multiplier at a time (linear kernel without bias constraint)
liblinear → LinearSVC
Steps After Solving
Support vectors are the points with alpha_i > 0.
For a linear kernel: w = sum alpha_i y_i x_i.
Bias b: average y_k - sum_i alpha_i y_i K(x_i, x_k) over margin support vectors (0 < alpha_k < C).
Predict: sign(sum_i alpha_i y_i K(x_i, x) + b).
Code Example: Solving the Dual with a Generic Optimizer
import numpy as np
from scipy.optimize import minimize
from sklearn.datasets import make_blobs
from sklearn.svm import SVC
X, y01 = make_blobs(n_samples=40, centers=2, cluster_std=1.2, random_state=42)
y = np.where(y01 == 1, 1.0, -1.0)
C = 1.0
K = X @ X.T # linear kernel matrix
Q = (y[:, None] * y[None, :]) * K
def neg_dual(a): # minimize the negative of the dual objective
return 0.5 * a @ Q @ a - a.sum()
def neg_dual_grad(a):
return Q @ a - np.ones_like(a)
res = minimize(neg_dual, x0=np.zeros(len(y)), jac=neg_dual_grad, method="SLSQP",
bounds=[(0, C)] * len(y),
constraints={"type": "eq", "fun": lambda a: a @ y, "jac": lambda a: y})
alpha = res.x
sv = alpha > 1e-5
w = (alpha * y) @ X
on_margin = sv & (alpha < C - 1e-5)
b = np.mean(y[on_margin] - X[on_margin] @ w)
ref = SVC(kernel="linear", C=C).fit(X, y)
print("support vectors (ours vs sklearn):", sv.sum(), len(ref.support_))
print("w ours:", w.round(3), "| sklearn:", ref.coef_[0].round(3))
print("b ours:", round(b, 3), "| sklearn:", round(ref.intercept_[0], 3))
The two solutions should agree closely; small differences come from solver tolerances.
Why the Dual Is Used
It needs only kernel values, enabling non-linear kernels.
The solution is sparse (most alpha_i = 0).
Its size depends on n, not on the (possibly infinite) feature dimension.
Common Pitfalls
Computing b from bounded support vectors (alpha = C), which gives wrong values.
Building the full n × n kernel matrix for large n; decomposition solvers avoid this with caching.
14
How do you choose the regularization parameter C in SVM?
KnownRevise
The regularization parameter C sets how heavily the SVM penalizes margin violations: a small C favors a wide margin and tolerates misclassified training points (stronger regularization), while a large C tries hard to classify every training point correctly (weaker regularization, risk of overfitting). C should be chosen with cross-validation, searching values on a logarithmic scale, and tuned jointly with gamma when using an RBF kernel, because the two interact.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
How C Affects the Model
C
Margin
Support vectors
Bias / variance
Very small (1e-3)
Very wide
Many
High bias (underfitting)
Moderate (1)
Balanced
Fewer
Often a good starting point
Very large (1e3+)
Narrow
Few (on separable data)
High variance (overfitting)
Procedure
Put scaling and the SVM in a Pipeline so each fold scales with its own training data.
Search C on a log scale, e.g. 10^-3 ... 10^3.
For RBF, search C and gamma together.
Use a metric that matches the goal (F1 or ROC AUC for imbalanced data).
Refine around the best value, then evaluate once on a held-out test set.
Code Example
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, train_test_split, validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
# 1. Validation curve for C alone (gamma fixed at "scale")
C_range = np.logspace(-3, 3, 7)
train_scores, val_scores = validation_curve(pipe, X_train, y_train, param_name="svc__C",
param_range=C_range, cv=5, scoring="roc_auc")
for C, tr, va in zip(C_range, train_scores.mean(axis=1), val_scores.mean(axis=1)):
print(f"C={C:>8.3f} train AUC {tr:.3f} val AUC {va:.3f}")
# 2. Joint search over C and gamma
grid = GridSearchCV(pipe, {"svc__C": np.logspace(-2, 3, 6), "svc__gamma": np.logspace(-4, 0, 5)},
cv=5, scoring="roc_auc", n_jobs=-1)
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV AUC:", round(grid.best_score_, 3))
print("test AUC:", round(grid.score(X_test, y_test), 3))
A widening gap between training and validation scores as C grows is the sign of overfitting.
Tips
If the best value is at the edge of the grid, extend the grid.
Noisy labels usually favor smaller C.
In LinearSVC and SVC, C multiplies the loss; in SGDClassifier, alpha multiplies the regularization, so roughly alpha ≈ 1 / (C * n_samples).
Common Pitfalls
Choosing C by training accuracy.
Scaling the data before splitting, which leaks information into cross-validation.
Tuning C on the test set.
15
What is the hinge loss function?
KnownRevise
Hinge loss is the loss function behind SVMs: for a label y in {-1, +1} and a raw score f(x) = w·x + b, it is max(0, 1 - y f(x)). It is zero when a point is on the correct side of the boundary with a margin of at least 1, and it grows linearly as the point moves into the margin or onto the wrong side. Minimizing hinge loss plus an L2 penalty on w is exactly the soft-margin SVM, which connects SVMs to other regularized linear models.
Sparse solutions (only margin points matter); not differentiable at m = 1
Squared hinge
max(0, 1 - m)^2
Smooth; penalizes large violations more (LinearSVC default)
Logistic (log loss)
log(1 + exp(-m))
Never exactly zero; gives probabilities (logistic regression)
0-1 loss
1 if m < 0 else 0
What we care about, but not convex
Here m = y f(x). Hinge loss is a convex upper bound on the 0-1 loss, which makes it tractable.
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import hinge_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
def hinge(y_pm, scores):
return np.maximum(0, 1 - y_pm * scores)
margins = np.array([2.0, 1.0, 0.5, 0.0, -1.0])
print("losses:", hinge(np.ones(5), margins)) # [0. 0. 0.5 1. 2. ]
# A linear SVM trained with SGD on hinge loss
X, y = make_classification(n_samples=2000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = make_pipeline(StandardScaler(),
SGDClassifier(loss="hinge", alpha=1e-4, max_iter=1000, random_state=42))
model.fit(X_train, y_train)
scores = model.decision_function(X_test)
y_pm = np.where(y_test == 1, 1, -1)
print("mean hinge loss (manual):", round(hinge(y_pm, scores).mean(), 4))
print("mean hinge loss (sklearn):", round(hinge_loss(y_test, scores), 4))
print("test accuracy:", round(model.score(X_test, y_test), 3))
Subgradient
Hinge loss is not differentiable at y f(x) = 1, so gradient methods use a subgradient: -y x when y f(x) < 1, and 0 otherwise. Only points that violate the margin update the weights.
Common Pitfalls
Using labels 0/1 instead of -1/+1 in a manual implementation.
Expecting probabilities from a hinge-loss model; use log loss or calibration.
Omitting regularization, which lets the weights grow to make every margin large.
16
Why do kernel parameters like sigma in the RBF kernel matter?
KnownRevise
The RBF (Gaussian) kernel measures similarity as K(x, z) = exp(-||x - z||^2 / (2 sigma^2)), which scikit-learn writes as exp(-gamma * ||x - z||^2) with gamma = 1 / (2 sigma^2). Sigma (or gamma) sets the kernel’s reach: how far a training point’s influence extends. A small sigma (large gamma) makes each point influence only its immediate neighbourhood, producing very wiggly boundaries that overfit; a large sigma (small gamma) makes the boundary smooth, approaching a linear model, and can underfit.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Effect of Gamma
gamma (sigma)
Influence of each point
Boundary
Support vectors
Risk
Very small (large sigma)
Very broad
Almost linear
Many
Underfitting
Moderate
Local but smooth
Follows the true shape
Moderate
Good fit
Very large (small sigma)
Tiny
Islands around points
Nearly all points
Overfitting
scikit-learn’s default gamma="scale" uses 1 / (n_features * X.var()), a sensible starting point after standardization.
Interaction with C
Large gamma + large C: memorizes the training data.
Small gamma + small C: too simple.
Good models lie along a diagonal band in the (C, gamma) grid, so tune them together.
Code Example
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=400, noise=0.3, random_state=42)
for gamma in (0.01, 0.1, 1, 10, 100):
sigma = np.sqrt(1 / (2 * gamma))
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, gamma=gamma))
res = cross_validate(model, X, y, cv=5, return_train_score=True)
n_sv = len(model.fit(X, y).named_steps["svc"].support_)
print(f"gamma={gamma:>6} (sigma={sigma:.2f}): train {res['train_score'].mean():.3f}, "
f"val {res['test_score'].mean():.3f}, support vectors {n_sv}")
Look for the gamma where validation accuracy peaks; past it, training accuracy keeps rising while validation accuracy falls.
Practical Tips
Always standardize features first; gamma is meaningless if features have very different scales.
Search gamma on a log scale, for example 10^-4 to 10^1.
A useful heuristic for sigma is the median pairwise distance between training points.
Common Pitfalls
Mixing up the two parameterizations: larger gamma means smaller sigma.
Tuning gamma with C fixed at an extreme value.
Reusing a gamma tuned on one feature set after adding or removing features.
17
What is the computational complexity of training an SVM?
KnownRevise
Training a kernel SVM with standard solvers such as libsvm’s SMO takes roughly between O(n^2) and O(n^3) time in the number of samples n, plus up to O(n^2) memory for kernel values, so it becomes impractical beyond tens of thousands of samples. Linear SVMs trained with liblinear or stochastic gradient descent scale roughly linearly with the number of samples and features, and handle millions of examples. Prediction costs O(number of support vectors × d) for kernel SVMs and O(d) for linear ones.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Complexity Summary
Model / solver
Training time
Memory
Prediction per sample
SVC / SVR (libsvm, SMO)
~O(n^2 d) to O(n^3 d)
Kernel cache up to O(n^2)
O(n_SV d)
LinearSVC (liblinear, coordinate descent)
~O(n d) per pass
O(n d)
O(d)
SGDClassifier(loss="hinge")
O(n d) per epoch
O(d) for the model
O(d)
Nystroem(k) + linear SVM
O(n k d + n k^2)
O(n k)
O(k d)
The exponent depends on C, the kernel, how separable the data is and how many support vectors there are: noisy data with many support vectors is the slow case.
Practical Speed-Ups
Use LinearSVC or SGDClassifier for large or high-dimensional data (text).
Approximate kernels with Nystroem or RBFSampler and train a linear SVM.
Increase cache_size (MB) in SVC when memory allows.
Subsample, or train on cluster centroids, for model selection; retrain the final model on more data.
Use GPU implementations (e.g. cuML’s SVC) for large kernel SVMs.
Code Example: Measuring Scaling
import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
def fit_time(model, X, y):
start = time.perf_counter()
model.fit(X, y)
return time.perf_counter() - start
sizes = [2000, 4000, 8000]
times = {"SVC rbf": [], "Nystroem + LinearSVC": [], "LinearSVC": []}
for n in sizes:
X, y = make_classification(n_samples=n, n_features=30, flip_y=0.1, random_state=42)
times["SVC rbf"].append(fit_time(make_pipeline(StandardScaler(), SVC(cache_size=500)), X, y))
times["Nystroem + LinearSVC"].append(fit_time(make_pipeline(
StandardScaler(), Nystroem(n_components=300, random_state=42), LinearSVC()), X, y))
times["LinearSVC"].append(fit_time(make_pipeline(StandardScaler(), LinearSVC()), X, y))
for name, t in times.items():
growth = np.log2(t[-1] / t[0]) / np.log2(sizes[-1] / sizes[0]) # empirical exponent
print(f"{name:22s} times {[round(v, 2) for v in t]} ~O(n^{growth:.1f})")
The empirical exponent is noisy for small sizes, but kernel SVC should show clearly super-linear growth.
Interview Tips
Say “between quadratic and cubic in n” for kernel SVMs and explain why: pairwise kernel values.
Mention prediction cost, which is often overlooked: many support vectors make inference slow.
18
How does SVM maximize the margin?
KnownRevise
SVM maximizes the margin by finding the weights w and bias b that keep every training point on the correct side of the boundary with a score of at least 1, y_i (w·x_i + b) >= 1, while making ||w|| as small as possible. Since the distance between the two margin hyperplanes is 2 / ||w||, minimizing (1/2)||w||^2 is the same as maximizing the margin. This turns a geometric goal into a convex quadratic program with a unique optimum, solved in practice through its dual.
Scale fixing: w and b can be multiplied by any positive constant without moving the boundary, so fix the scale by requiring the closest points to have |w·x + b| = 1.
Margin width: the distance from a point to the boundary is |w·x + b| / ||w||, so the closest points are 1 / ||w|| away on each side, and the full margin is 2 / ||w||.
Optimization: maximize 2 / ||w|| ⇔ minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1.
Soft margin: add slack xi_i and a penalty C sum(xi_i) for data that is not separable.
Solve: convert to the dual with Lagrange multipliers; only support vectors get non-zero multipliers.
Why Maximize the Margin?
A wider margin means small changes in the inputs (noise) are less likely to flip predictions.
Statistical learning theory links larger margins to tighter generalization bounds, largely independent of the number of features.
The solution depends only on support vectors, so it is robust to points far from the boundary.
Code Example
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC
X, y = make_blobs(n_samples=80, centers=2, cluster_std=1.0, random_state=42)
svc = SVC(kernel="linear", C=1e5).fit(X, y) # near hard margin
w, b = svc.coef_[0], svc.intercept_[0]
distances = np.abs(X @ w + b) / np.linalg.norm(w)
print("margin width 2/||w||:", round(2 / np.linalg.norm(w), 4))
print("closest point distance × 2:", round(2 * distances.min(), 4)) # matches the margin width
# Any other separating line has a smaller minimum distance; compare with a perturbed boundary
w_alt = w + np.array([0.1, -0.1])
if np.all(np.sign(X @ w_alt + b) == np.where(y == 1, 1, -1)):
print("alternative still separates; its margin:", round(2 * (np.abs(X @ w_alt + b) / np.linalg.norm(w_alt)).min(), 4))
else:
print("alternative boundary misclassifies some points")
Common Pitfalls
Forgetting scaling: the margin is measured in feature units, so one large-scale feature dominates.
Believing the maximum margin is always best on noisy data: the soft margin with tuned C trades margin for errors.
19
Can you derive the optimization problem for the soft margin SVM?
KnownRevise
The soft-margin SVM starts from the hard-margin problem, which maximizes the margin while requiring every point to be correctly classified, and relaxes it with slack variables so that some points may violate the margin at a cost. The derivation below builds the primal problem from geometry, shows its equivalence to regularized hinge loss, and derives the dual through Lagrange multipliers, which reveals the box constraint 0 <= alpha_i <= C and the role of support vectors.
Distance of point i to the boundary: y_i f(x_i) / ||w||.
Fix the scale so the closest points have y_i f(x_i) = 1; the margin is then 2 / ||w||.
Problem: minimize (1/2)||w||^2 subject to y_i (w·x_i + b) >= 1 for all i.
2. Adding Slack
Allow violations: y_i (w·x_i + b) >= 1 - xi_i, with xi_i >= 0.
Penalize the total violation with weight C.
Primal soft-margin problem: minimize (1/2)||w||^2 + C sum_i xi_i subject to y_i (w·x_i + b) >= 1 - xi_i and xi_i >= 0.
3. Equivalent Unconstrained Form
At the optimum each slack is as small as allowed: xi_i = max(0, 1 - y_i (w·x_i + b)). Substituting gives minimize (1/2)||w||^2 + C sum_i max(0, 1 - y_i f(x_i)): L2-regularized hinge loss.
4. Lagrangian
L = (1/2)||w||^2 + C sum xi_i - sum alpha_i [y_i (w·x_i + b) - 1 + xi_i] - sum mu_i xi_i, with alpha_i, mu_i >= 0.
Setting the derivatives to zero: - With respect to w: w = sum_i alpha_i y_i x_i. - With respect to b: sum_i alpha_i y_i = 0. - With respect to xi_i: C - alpha_i - mu_i = 0, so 0 <= alpha_i <= C.
5. Dual Problem
Substituting back, the slack terms cancel:
maximize sum_i alpha_i - (1/2) sum_i sum_j alpha_i alpha_j y_i y_j (x_i·x_j) subject to 0 <= alpha_i <= C and sum_i alpha_i y_i = 0.
Replacing x_i·x_j with K(x_i, x_j) gives the kernel SVM.
6. KKT Interpretation
Condition
Point
alpha_i = 0
Correct side, outside the margin (y_i f(x_i) >= 1)
0 < alpha_i < C
Exactly on the margin (xi_i = 0)
alpha_i = C
Inside the margin or misclassified (xi_i > 0)
Code Check
import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=150, n_features=3, flip_y=0.1, random_state=42)
X = StandardScaler().fit_transform(X)
C = 0.5
svc = SVC(kernel="linear", C=C).fit(X, y)
y_pm = np.where(y == 1, 1, -1)
f = svc.decision_function(X)
xi = np.maximum(0, 1 - y_pm * f)
alpha = np.zeros(len(X))
alpha[svc.support_] = np.abs(svc.dual_coef_[0])
print("alpha = 0 implies y*f >= 1:", np.all(y_pm[alpha < 1e-8] * f[alpha < 1e-8] >= 1 - 1e-3))
print("xi > 0 implies alpha = C:", np.allclose(alpha[xi > 1e-3], C))
print("w = sum alpha_i y_i x_i:", np.allclose((alpha * y_pm) @ X, svc.coef_[0], atol=1e-4))
All three checks should print True (within solver tolerance).
Interview Tips
Explain where C appears in the dual (the upper bound on alpha).
Mention that the dual needs only inner products, enabling kernels.
20
What preprocessing steps should you take before training an SVM?
KnownRevise
Preprocessing for an SVM matters more than for tree models, because SVMs work with distances and dot products. The essential steps are: handle missing values, encode categorical features numerically, scale numeric features (standardization is the default), deal with outliers and class imbalance, and optionally reduce dimensionality or select features. All of these should be fitted on training data only, which is easiest inside a scikit-learn Pipeline with a ColumnTransformer.
Scaling the whole dataset before the split, which leaks test information.
Label-encoding nominal categories as 0, 1, 2, which invents an order and false distances.
Using StandardScaler on sparse TF-IDF matrices (it densifies them); use MaxAbsScaler or none.
Forgetting to apply exactly the same preprocessing at prediction time; a saved Pipeline guarantees it.
21
How do you handle categorical variables when training an SVM?
KnownRevise
SVMs need numeric inputs, and because they rely on distances and dot products, the way categories are encoded directly shapes the model. Nominal categories (colour, city) should usually be one-hot encoded, so every category is equally distant from the others; ordinal categories (small, medium, large) can be mapped to ordered numbers. High-cardinality features need grouping of rare levels, target encoding with cross-fitting, or hashing. All encoders must be fitted on training data only, which a ColumnTransformer inside a Pipeline ensures.
Avoid label-encoding nominal categories as 0, 1, 2: the SVM would treat “category 2” as twice as far from “category 0” as “category 1” is.
Code Example
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler, TargetEncoder
from sklearn.svm import SVC
rng = np.random.default_rng(42)
n = 1000
df = pd.DataFrame({
"tenure": rng.integers(1, 60, n),
"plan": rng.choice(["basic", "plus", "pro"], n),
"size": rng.choice(["small", "medium", "large"], n),
"city": rng.choice([f"city_{i}" for i in range(150)], n), # high cardinality
})
y = ((df["plan"] == "pro") | (df["tenure"] > 40)).astype(int).to_numpy()
preprocess = ColumnTransformer([
("num", StandardScaler(), ["tenure"]),
("onehot", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
("ordinal", Pipeline([("enc", OrdinalEncoder(categories=[["small", "medium", "large"]])),
("scale", StandardScaler())]), ["size"]),
("target", TargetEncoder(random_state=42), ["city"]), # cross-fitted inside fit_transform
])
model = Pipeline([("prep", preprocess), ("svc", SVC(kernel="rbf", C=1.0))])
print("CV accuracy:", round(cross_val_score(model, df, y, cv=5).mean(), 3))
model.fit(df, y)
print("encoded feature count:", model.named_steps["prep"].transform(df.head(1)).shape[1])
Scaling One-Hot Columns
One-hot columns are 0/1; numeric features are usually standardized. Mixing them is fine, but very many one-hot columns can dominate RBF distances, so consider grouping rare levels or weighting groups of features.
Common Pitfalls
Fitting encoders on the full dataset before splitting.
Unknown categories at prediction time crashing the encoder; set handle_unknown="ignore".
Thousands of one-hot columns with an RBF kernel; a linear SVM suits sparse, wide data better.
22
What methods can you use to tune SVM hyperparameters?
KnownRevise
Tuning SVM hyperparameters means choosing the kernel, the regularization strength C and kernel parameters such as gamma (RBF) or degree (polynomial) to maximize cross-validated performance. The standard methods are grid search over log-spaced values, randomized search from log-uniform distributions, successive halving to discard poor settings early, and Bayesian optimization (for example Optuna). Whatever the method, wrap scaling and the SVM in a Pipeline, use a metric that matches the goal, and keep a final test set untouched.
Evaluates many candidates cheaply on small subsets first
Small-sample results can mislead for SVMs
Bayesian optimization (Optuna)
Uses past results to choose the next trial
Extra dependency
Code Example
from scipy.stats import loguniform
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipe = Pipeline([("scale", StandardScaler()), ("svc", SVC())])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Grid search across kernels with kernel-specific parameters
param_grid = [
{"svc__kernel": ["linear"], "svc__C": [0.01, 0.1, 1, 10]},
{"svc__kernel": ["rbf"], "svc__C": [0.1, 1, 10, 100], "svc__gamma": [1e-3, 1e-2, 1e-1, "scale"]},
{"svc__kernel": ["poly"], "svc__C": [0.1, 1, 10], "svc__degree": [2, 3], "svc__coef0": [0, 1]},
]
grid = GridSearchCV(pipe, param_grid, cv=cv, scoring="roc_auc", n_jobs=-1).fit(X_train, y_train)
print("grid best:", grid.best_params_, round(grid.best_score_, 4))
# Randomized search with log-uniform distributions
rand = RandomizedSearchCV(pipe, {"svc__kernel": ["rbf"], "svc__C": loguniform(1e-2, 1e3),
"svc__gamma": loguniform(1e-4, 1e0)},
n_iter=40, cv=cv, scoring="roc_auc", random_state=42, n_jobs=-1).fit(X_train, y_train)
print("random best:", {k: (round(v, 4) if isinstance(v, float) else v) for k, v in rand.best_params_.items()})
print("test ROC AUC of grid winner:", round(grid.score(X_test, y_test), 4))
An Optuna version samples C and gamma with trial.suggest_float(..., log=True) and returns the mean of cross_val_score as the objective.
Common Pitfalls
Linear grids such as [1, 2, 3, ..., 10]; SVM parameters act on a log scale.
Scaling outside the Pipeline, which leaks fold information.
Reporting the best CV score as the final performance; use the held-out test set or nested cross-validation.
23
How do you handle an imbalanced dataset when using SVM?
KnownRevise
Class imbalance pushes an SVM’s boundary toward the minority class, because the penalty term is dominated by the many majority-class points; the result often looks accurate while missing most minority cases. The main fixes are class weights (a larger C for the minority class), resampling (oversampling the minority class or undersampling the majority), adjusting the decision threshold on the SVM’s scores, and evaluating with metrics that reflect minority-class performance, such as precision, recall, F1 and average precision.
Each class’s C is multiplied by its weight; simplest first step
Threshold tuning
Choose a cut-off on decision_function using validation data
Matches the business trade-off between false positives and negatives
Oversampling
SMOTE / random oversampling (imbalanced-learn)
Must be done inside CV folds
Undersampling
Remove majority examples
Also speeds up kernel SVM training
Metrics
Recall, precision, F1, PR AUC; confusion matrix
Accuracy is misleading
Stratified splits
StratifiedKFold
Keeps class ratios in each fold
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics import average_precision_score, classification_report, precision_recall_curve
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=4000, n_features=10, n_informative=5, weights=[0.95, 0.05],
flip_y=0.01, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.25, stratify=y_train, random_state=42)
for weight in (None, "balanced"):
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight=weight)).fit(X_tr, y_tr)
scores = model.decision_function(X_test)
print(f"class_weight={weight}: PR AUC {average_precision_score(y_test, scores):.3f}")
print(classification_report(y_test, model.predict(X_test), digits=3))
# Threshold tuning on validation data: pick the threshold with the best F1
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight="balanced")).fit(X_tr, y_tr)
prec, rec, thr = precision_recall_curve(y_val, model.decision_function(X_val))
f1 = 2 * prec[:-1] * rec[:-1] / (prec[:-1] + rec[:-1] + 1e-12)
best_thr = thr[np.argmax(f1)]
y_pred = (model.decision_function(X_test) >= best_thr).astype(int)
print(f"tuned threshold {best_thr:.3f}:")
print(classification_report(y_test, y_pred, digits=3))
With imbalanced-learn, put the sampler inside its Pipeline (make_pipeline(StandardScaler(), SMOTE(random_state=42), SVC())) so synthetic samples are created only from each training fold.
Common Pitfalls
Oversampling before the train/test split, which copies test-like points into training.
Reporting accuracy: 95% is achieved by always predicting the majority class here.
Tuning the threshold on the test set.
Treating decision_function values as probabilities.
24
Which metrics are commonly used to evaluate an SVM model?
KnownRevise
SVMs are evaluated with the same metrics as other classifiers or regressors, chosen to match the problem: accuracy, precision, recall, F1 and the confusion matrix for classification; ROC AUC and average precision (PR AUC) for ranking quality, which can be computed from the SVM’s decision_function scores without probabilities; and MAE, RMSE and R² for Support Vector Regression. SVM-specific diagnostics, such as the number of support vectors and the margin, help explain model complexity and prediction cost.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Classification Metrics
Metric
Meaning
When to use
Accuracy
Share of correct predictions
Balanced classes, equal error costs
Precision
Of predicted positives, how many are correct
False positives are costly (spam filter)
Recall (sensitivity)
Of actual positives, how many are found
False negatives are costly (disease screening)
F1
Harmonic mean of precision and recall
Imbalanced classes, single summary
ROC AUC
Probability a random positive scores above a random negative
Threshold-independent ranking
Average precision (PR AUC)
Area under precision-recall curve
Rare positive class
Confusion matrix
Counts of each error type
Always inspect
Log loss / Brier score
Probability quality
Only with calibrated probabilities
Regression (SVR) Metrics
MAE (average absolute error), RMSE (penalizes large errors), R² (variance explained), and the share of predictions within epsilon of the target.
SVM-Specific Diagnostics
len(model.support_): many support vectors mean slower predictions and possibly overlapping classes.
Margin width 2 / ||w|| for linear models.
Training vs validation score gap for overfitting.
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import (ConfusionMatrixDisplay, average_precision_score, classification_report,
confusion_matrix, roc_auc_score)
from sklearn.model_selection import cross_validate, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0))
cv = cross_validate(model, X_train, y_train, cv=5,
scoring=["accuracy", "precision", "recall", "f1", "roc_auc"])
for name in ("accuracy", "precision", "recall", "f1", "roc_auc"):
print(f"CV {name:9s}: {cv['test_' + name].mean():.3f} ± {cv['test_' + name].std():.3f}")
model.fit(X_train, y_train)
scores = model.decision_function(X_test) # no probability=True needed for AUC metrics
print("test ROC AUC:", round(roc_auc_score(y_test, scores), 3))
print("test PR AUC:", round(average_precision_score(y_test, scores), 3))
print(confusion_matrix(y_test, model.predict(X_test)))
print(classification_report(y_test, model.predict(X_test), digits=3))
print("support vectors:", len(model.named_steps["svc"].support_))
Common Pitfalls
Using accuracy alone on imbalanced data.
Computing log loss on uncalibrated decision_function outputs.
Calling predict_proba with probability=True and expecting it to agree exactly with predict; the Platt model can disagree near the boundary.
25
What is the trade-off between model complexity and generalization in SVM?
KnownRevise
In an SVM, model complexity is controlled mainly by C and the kernel parameters. A more complex model (large C, large RBF gamma, high polynomial degree) fits the training data more closely, reducing bias but increasing variance, so it may generalize worse; a simpler model (small C, small gamma, linear kernel) has a wider margin and smoother boundary, which generalizes better until it starts to underfit. The goal is the setting with the best validation performance, which cross-validation finds.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Complexity Knobs
Knob
Simpler (more regularized)
More complex
C
Small: wider margin, more violations
Large: narrow margin, fits noise
gamma (RBF)
Small: smooth, near-linear boundary
Large: local “islands” around points
degree (poly)
2
5+
Kernel
Linear
RBF, high-degree poly
Features
Fewer, selected
Many raw or engineered features
Why the Margin Helps Generalization
Statistical learning theory bounds the generalization error of margin classifiers in terms of the margin relative to the data’s spread, not the raw number of features. A wide margin means low effective capacity even in very high dimensions, which is why SVMs work well on text. The fraction of support vectors also bounds the leave-one-out error: very many support vectors can signal a poor fit.
Code Example: Train vs Validation Across Gamma and C
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=500, noise=0.3, random_state=42)
print(f"{'C':>6} {'gamma':>6} {'train':>6} {'val':>6} {'gap':>6} {'#SV':>5}")
for C in (0.1, 1, 100):
for gamma in (0.1, 1, 30):
model = make_pipeline(StandardScaler(), SVC(C=C, gamma=gamma))
res = cross_validate(model, X, y, cv=5, return_train_score=True)
tr, va = res["train_score"].mean(), res["test_score"].mean()
n_sv = len(model.fit(X, y).named_steps["svc"].support_)
print(f"{C:>6} {gamma:>6} {tr:6.3f} {va:6.3f} {tr - va:6.3f} {n_sv:5d}")
Look for low-gamma, low-C rows with low scores on both sets (underfitting) and high-gamma, high-C rows with near-perfect training scores but a large gap (overfitting); the best rows sit in between.
Practical Guidance
Start simple (linear, or RBF with default gamma) and add complexity only if validation scores improve.
Use learning curves: if both curves plateau low, add complexity; if a big gap persists, regularize or add data.
Use nested cross-validation when you need an unbiased estimate after tuning.
Common Pitfalls
Judging complexity by training accuracy.
Tuning C and gamma separately, missing their interaction.
26
How does feature scaling affect SVM performance?
KnownRevise
Feature scaling is essential for SVMs because both the margin and most kernels depend on distances or dot products between feature vectors. If one feature ranges in thousands and another between 0 and 1, the large feature dominates the distance, the margin is effectively measured along it alone, and the RBF kernel’s gamma becomes meaningless. Scaling also improves the numerical conditioning of the optimization, so training converges faster.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Effects of Not Scaling
Aspect
Effect
Linear SVM
Weights and margin biased toward large-scale features; regularization penalizes features unequally
RBF kernel
Distances dominated by one feature; gamma either flattens or explodes the kernel
Polynomial kernel
Large values raised to powers cause numerical overflow and instability
Training time
Poor conditioning: many more iterations, convergence warnings
Interpretation
coef_ values not comparable across features
Scalers
Scaler
Use when
StandardScaler
Default: mean 0, standard deviation 1
MinMaxScaler
Bounded features; required range [0, 1]
RobustScaler
Outliers present (uses median and IQR)
MaxAbsScaler
Sparse data such as TF-IDF (keeps sparsity)
Normalizer
Per-sample unit length, e.g. for cosine-like similarity
Code Example
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler
from sklearn.svm import SVC
X, y = load_wine(return_X_y=True)
print("feature ranges:", np.ptp(X, axis=0).round(1)) # from about 0.5 to over 1000
configs = {
"no scaling": SVC(kernel="rbf"),
"StandardScaler": make_pipeline(StandardScaler(), SVC(kernel="rbf")),
"MinMaxScaler": make_pipeline(MinMaxScaler(), SVC(kernel="rbf")),
"RobustScaler": make_pipeline(RobustScaler(), SVC(kernel="rbf")),
}
for name, model in configs.items():
print(f"{name:15s} CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
The wine dataset’s proline feature is in the hundreds to over a thousand while others are below 10, so the unscaled model should score clearly worse.
Common Pitfalls
Fitting the scaler on all data before cross-validation; keep it in the Pipeline.
Scaling one-hot columns unnecessarily (harmless with StandardScaler but changes their relative weight).
Forgetting to apply the same fitted scaler at prediction time.
27
How can you speed up SVM training on large datasets?
KnownRevise
Kernel SVM training scales roughly between O(n^2) and O(n^3), so on large datasets the main speed-ups come from changing the problem: use a linear SVM (LinearSVC or SGDClassifier with hinge loss), approximate the kernel with random features or the Nyström method and train a linear SVM on top, reduce the data (subsampling, removing redundant points) or move to GPU implementations. Within SVC, a bigger kernel cache, a looser tolerance and well-scaled features also help.
Linear-time solvers; often enough for high-dimensional data
Kernel approximation
Nystroem(n_components=500) or RBFSampler + LinearSVC
Explicit low-dimensional features approximate the RBF kernel
Subsampling for tuning
Tune C/gamma on 10-20% of data, refit once on more
Fewer expensive fits
Successive halving
HalvingRandomSearchCV
Discards poor settings early
Cache and tolerance
SVC(cache_size=2000, tol=1e-2)
Fewer kernel recomputations, earlier stop
Scaling
StandardScaler
Better conditioning: faster convergence
Parallelism
n_jobs=-1 in search / OvR wrappers
Uses all cores (SVC itself is single-threaded)
Feature reduction
PCA, feature selection
Cheaper kernel evaluations
GPU
cuML SVC
Large constant-factor speed-ups
Online learning
SGDClassifier.partial_fit on chunks
Data larger than memory
Code Example
import time
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = make_classification(n_samples=30000, n_features=40, n_informative=15, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
models = {
"SVC rbf (exact)": make_pipeline(StandardScaler(), SVC(kernel="rbf", cache_size=1000)),
"Nystroem(500) + LinearSVC": make_pipeline(StandardScaler(), Nystroem(gamma=0.02, n_components=500,
random_state=42), LinearSVC()),
"LinearSVC": make_pipeline(StandardScaler(), LinearSVC()),
"SGD hinge": make_pipeline(StandardScaler(), SGDClassifier(loss="hinge", random_state=42)),
}
for name, model in models.items():
start = time.perf_counter()
model.fit(X_train, y_train)
elapsed = time.perf_counter() - start
print(f"{name:28s} fit {elapsed:6.2f}s test accuracy {model.score(X_test, y_test):.3f}")
Expect the exact SVC to be the slowest by a wide margin; compare whether the approximations lose any accuracy on your data.
Common Pitfalls
Using SVC(kernel="linear") on large data instead of LinearSVC.
probability=True, which adds an internal 5-fold cross-validation.
Forgetting that prediction time grows with the number of support vectors too.
28
How do you diagnose and fix underfitting or overfitting in an SVM?
KnownRevise
Diagnosing an SVM’s fit starts with comparing training and validation scores: low scores on both indicate underfitting (too simple), while a high training score with a much lower validation score indicates overfitting (too complex). Then adjust the complexity controls (C, gamma, kernel, degree), the features and the amount of data. Validation curves, learning curves and the number of support vectors make the diagnosis concrete.
Increase C; increase gamma; switch from linear to RBF/poly; add features; check scaling
Overfitting
Train near perfect, validation much lower
Decrease C; decrease gamma; lower degree; remove noisy features; more data; check for leakage
Many support vectors (> ~50% of data)
Heavy class overlap or too small C
Better features; tune C
Unstable results across folds
High variance
More data, simpler model, repeated CV
Code Example: Validation and Learning Curves
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=1500, n_features=20, n_informative=6, flip_y=0.08, random_state=42)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0))
gammas = np.logspace(-4, 1, 6)
train, val = validation_curve(pipe, X, y, param_name="svc__gamma", param_range=gammas, cv=5)
for g, tr, va in zip(gammas, train.mean(axis=1), val.mean(axis=1)):
label = "underfit" if tr < 0.8 else ("overfit" if tr - va > 0.1 else "ok")
print(f"gamma={g:<8.4f} train {tr:.3f} val {va:.3f} -> {label}")
sizes, tr_lc, va_lc = learning_curve(make_pipeline(StandardScaler(), SVC(gamma=0.05)), X, y, cv=5,
train_sizes=np.linspace(0.1, 1.0, 5), random_state=42)
for n, tr, va in zip(sizes, tr_lc.mean(axis=1), va_lc.mean(axis=1)):
print(f"n={n:5d} train {tr:.3f} val {va:.3f}")
The labels in the first loop use simple illustrative thresholds; in practice judge from the curves. In the learning curve, a validation score still rising at full size means more data would help.
Diagnostic Checklist
Confirm features are scaled inside the Pipeline.
Check for leakage if validation scores look too good.
Tune C and gamma jointly with cross-validation.
Inspect misclassified examples for label noise or missing features.
Common Pitfalls
Fixing overfitting by lowering C alone when gamma is the real cause, or the reverse.
Drawing conclusions from a single train/validation split on small data.
29
What should you consider when deploying an SVM model in production?
KnownRevise
Deploying an SVM involves the same concerns as any model (reproducible preprocessing, versioning, monitoring) plus some SVM-specific ones: prediction latency and memory grow with the number of support vectors for kernel models, scaling and encoding must be identical to training, probabilities need calibration if decisions depend on them, and models must be retrained because standard SVMs cannot update incrementally. Packaging the whole scikit-learn Pipeline as one artifact handles most preprocessing risks.
Scheduled or drift-triggered; batch retraining on recent data
Interpretability
Linear weights or SHAP explanations for audits
Fallback
Default behaviour on invalid input or service errors
Code Example: Package, Save, Load and Serve
import time
import joblib
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
pipeline = make_pipeline(StandardScaler(),
CalibratedClassifierCV(SVC(kernel="rbf", C=1.0), method="sigmoid", cv=5))
pipeline.fit(X_train, y_train)
artifact = {"model": pipeline, "threshold": 0.5, "version": "2026-11-10",
"feature_count": X.shape[1], "sklearn": __import__("sklearn").__version__}
joblib.dump(artifact, "svm_model.joblib")
loaded = joblib.load("svm_model.joblib") # load once at service start-up
def predict(features):
x = np.asarray(features, dtype=float).reshape(1, -1)
if x.shape[1] != loaded["feature_count"] or not np.isfinite(x).all():
raise ValueError("invalid input")
proba = loaded["model"].predict_proba(x)[0, 1]
return {"probability": round(float(proba), 4), "label": int(proba >= loaded["threshold"]),
"model_version": loaded["version"]}
start = time.perf_counter()
for row in X_test[:200]:
predict(row)
print(f"mean latency {(time.perf_counter() - start) / 200 * 1000:.2f} ms")
print(predict(X_test[0]))
Common Pitfalls
Saving only the SVM without its scaler.
Loading a pickle with a different scikit-learn version.
No monitoring, so slow data drift goes unnoticed.
30
How does SVM handle incremental or online learning?
KnownRevise
Standard kernel SVM solvers are batch algorithms: scikit-learn’s SVC must retrain on all data when new examples arrive. For incremental or online learning, the practical options are a linear SVM trained with stochastic gradient descent (SGDClassifier(loss="hinge"), which supports partial_fit), the same model on top of a fixed kernel approximation for non-linear boundaries, or specialized online kernel SVM algorithms such as LASVM and budgeted SVMs. Another approach is to retrain periodically, keeping only the support vectors plus new data.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Options
Approach
How it works
Trade-offs
SGDClassifier(loss="hinge").partial_fit
Updates weights on each mini-batch with the hinge subgradient
Linear only; fast; handles streams and data larger than memory
RBFSampler / Nystroem + SGD
Fixed random/landmark features, then online linear SVM
Feature map must be fitted on an initial sample and kept fixed
Support-vector retraining
Keep the old SVs, add new data, retrain SVC
Approximate; SV set can grow
Online kernel SVMs (LASVM, budgeted SGD)
Add/remove SVs incrementally under a budget
Not in scikit-learn
Periodic batch retraining
Retrain on a sliding window
Simple; handles drift
Code Example: Streaming Linear SVM
import numpy as np
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import RBFSampler
from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=50000, n_features=20, n_informative=8, random_state=42)
X_stream, y_stream = X[:45000], y[:45000]
X_test, y_test = X[45000:], y[45000:]
classes = np.unique(y)
# Fit preprocessing on an initial chunk, then keep it fixed
scaler = StandardScaler().fit(X_stream[:2000])
rbf = RBFSampler(gamma=0.05, n_components=500, random_state=42).fit(scaler.transform(X_stream[:2000]))
linear = SGDClassifier(loss="hinge", alpha=1e-4, random_state=42)
kernelized = SGDClassifier(loss="hinge", alpha=1e-5, random_state=42)
for start in range(0, len(X_stream), 5000): # chunks arriving over time
xb = scaler.transform(X_stream[start:start + 5000])
yb = y_stream[start:start + 5000]
linear.partial_fit(xb, yb, classes=classes) # classes required on the first call
kernelized.partial_fit(rbf.transform(xb), yb, classes=classes)
Xt = scaler.transform(X_test)
print(f"after {start + 5000:5d} samples: linear {linear.score(Xt, y_test):.3f}, "
f"RBF features {kernelized.score(rbf.transform(Xt), y_test):.3f}")
Concept Drift
For streams whose distribution changes, monitor accuracy on recent labelled data, use a constant or adaptive learning rate so old data is gradually forgotten, or retrain on a sliding window.
Common Pitfalls
Refitting the scaler on each chunk, which changes the feature space under the model.
Forgetting classes= on the first partial_fit call.
Expecting SVC to have partial_fit; it does not.
31
What are the challenges of training SVMs in distributed computing environments?
KnownRevise
Distributing SVM training is hard because the kernel SVM’s optimization couples every pair of training points through the kernel matrix, so the problem does not split cleanly across machines the way gradient descent on independent mini-batches does. The main challenges are communication cost, memory for kernel values, the global equality constraint in the dual, and load imbalance, since the number and location of support vectors are unknown in advance. Practical systems therefore use linear SVMs with distributed SGD, kernel approximations, or divide-and-conquer schemes that train local SVMs and merge their support vectors.
Every alpha_i interacts with every other through K(x_i, x_j)
Kernel matrix size
n × n values; 1 million samples means 10^12 entries
Global constraint
sum alpha_i y_i = 0 ties all partitions together
Communication
Solvers such as SMO update two variables at a time: too fine-grained to distribute
Load imbalance
Support vectors concentrate in some partitions
Hyperparameter search
Each (C, gamma) setting needs a full expensive training
Non-identical partitions
Local models trained on skewed shards disagree
Approaches
Approach
Idea
Tools
Distributed linear SVM
Data-parallel gradient or L-BFGS updates of w
Spark MLlib LinearSVC, Vowpal Wabbit
Kernel approximation + linear SVM
Random Fourier features / Nyström make the problem linear
scikit-learn on Dask, Spark
Cascade SVM
Train SVMs on partitions, pass their support vectors up a tree, retrain
Custom
Divide-and-conquer kernel SVM
Cluster the data, train local SVMs, use them to warm-start a global one
Research implementations
Parallel hyperparameter search
Distribute independent fits across workers
GridSearchCV(n_jobs=-1) with joblib/Dask/Ray backends
GPU acceleration
Massively parallel kernel computation on one node
cuML SVC
Code Example: Cascade-Style Approximation
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=20000, n_features=20, n_informative=8, flip_y=0.03, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
# Layer 1: each "worker" trains on its own shard (run in parallel in a real cluster)
shards = np.array_split(np.random.default_rng(42).permutation(len(X_train)), 4)
sv_idx = []
for shard in shards:
local = SVC(kernel="rbf", C=1.0).fit(X_train[shard], y_train[shard])
sv_idx.extend(shard[local.support_]) # keep only support vectors
# Layer 2: train on the union of support vectors
sv_idx = np.array(sv_idx)
final = SVC(kernel="rbf", C=1.0).fit(X_train[sv_idx], y_train[sv_idx])
print(f"final model trained on {len(sv_idx)} of {len(X_train)} points")
print("cascade accuracy:", round(final.score(X_test, y_test), 3))
With noisy data, many points are support vectors, so the cascade’s second stage can still be large; that is the method’s main limitation.
Spark MLlib
Spark’s pyspark.ml.classification.LinearSVC trains a linear SVM with OWL-QN over partitioned data; there is no distributed kernel SVM in MLlib, which reflects the difficulty described above.
Common Pitfalls
Expecting linear speed-ups from more machines for exact kernel SVMs.
Scaling each shard separately, which gives inconsistent feature spaces.
Ignoring that a gradient-boosted tree or neural network may suit very large data better.
32
How can you reduce SVM model storage and inference time?
KnownRevise
A kernel SVM stores every support vector and evaluates the kernel against each of them for every prediction, so its size and latency grow with the number of support vectors. To shrink both, reduce the number of support vectors (tune C and gamma, clean noisy labels, reduce the training set), reduce features, or replace the kernel model with a cheaper equivalent: a linear SVM, an explicit kernel approximation, or a compact model distilled from the SVM. Batching predictions and lower-precision storage add further gains.
import os
import time
import joblib
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import Nystroem
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = make_classification(n_samples=12000, n_features=30, n_informative=10, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
models = {
"SVC C=10": make_pipeline(StandardScaler(), SVC(C=10)),
"SVC C=0.5": make_pipeline(StandardScaler(), SVC(C=0.5)),
"Nystroem(300)+LinearSVC": make_pipeline(StandardScaler(), Nystroem(n_components=300, random_state=42),
LinearSVC()),
"LinearSVC": make_pipeline(StandardScaler(), LinearSVC()),
}
for name, model in models.items():
model.fit(X_train, y_train)
path = f"{name.split('(')[0].replace(' ', '_').replace('=', '')}.joblib"
joblib.dump(model, path, compress=3)
start = time.perf_counter()
acc = model.score(X_test, y_test)
latency_ms = (time.perf_counter() - start) / len(X_test) * 1000
n_sv = len(model[-1].support_) if hasattr(model[-1], "support_") else "-"
print(f"{name:25s} acc {acc:.3f} SVs {n_sv!s:>5} size {os.path.getsize(path) / 1024:8.1f} KB "
f"{latency_ms:.4f} ms/sample")
Compare accuracy against size and latency to choose the cheapest model that meets the accuracy target.
Common Pitfalls
Reducing C so much that accuracy drops; measure both.
Benchmarking single predictions in a loop when the service could batch them.
Forgetting the scaler and encoders in size and latency measurements.
33
What is Support Vector Regression, and where is it used?
KnownRevise
Support Vector Regression (SVR) adapts the SVM idea to predicting continuous values. Instead of a margin between classes, it fits a function with an epsilon-insensitive tube around it: errors smaller than epsilon are ignored, and only points outside the tube (the support vectors) contribute to the loss, while the model is kept as flat (regularized) as possible. With kernels, SVR fits non-linear relationships, and it is used for small to medium datasets in areas such as forecasting, engineering and chemistry.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Objective
Minimize (1/2)||w||^2 + C * sum(xi_i + xi_i*) subject to each prediction being within epsilon + slack of its target.
In words: keep the function flat, ignore errors up to epsilon, and pay a linear penalty, weighted by C, for anything beyond.
Parameters
Parameter
Effect
epsilon
Tube width: larger = fewer support vectors, smoother fit, ignores more error
C
Penalty on errors outside the tube: larger = fits data more closely
kernel, gamma
Shape and flexibility of the function
Epsilon is in the units of the target, so scale the target or set epsilon relative to its noise level.
Applications
Domain
Example
Finance
Volatility or price-change estimation on engineered features
Energy
Short-term load or solar output forecasting
Engineering
Surrogate models for expensive simulations
Chemistry / QSAR
Predicting molecular properties
Healthcare
Dose-response or biomarker estimation
Code Example
import numpy as np
from sklearn.compose import TransformedTargetRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVR
X, y = fetch_california_housing(return_X_y=True) # downloads the data
X, y = X[:5000], y[:5000] # subset: kernel SVR is slow on large n
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
svr = TransformedTargetRegressor( # scales y so epsilon is in standard units
regressor=make_pipeline(StandardScaler(), SVR(kernel="rbf")),
transformer=StandardScaler())
grid = GridSearchCV(svr, {"regressor__svr__C": [1, 10, 100],
"regressor__svr__epsilon": [0.05, 0.1, 0.3],
"regressor__svr__gamma": ["scale", 0.1]},
cv=5, scoring="neg_mean_absolute_error", n_jobs=-1)
grid.fit(X_train, y_train)
pred = grid.predict(X_test)
print("best:", grid.best_params_)
print(f"test MAE {mean_absolute_error(y_test, pred):.3f}, R2 {r2_score(y_test, pred):.3f}")
print("support vectors:", len(grid.best_estimator_.regressor_[-1].support_))
Common Pitfalls
Leaving the target unscaled, so a default epsilon of 0.1 is meaningless.
Large datasets: use LinearSVR or kernel approximations.
Expecting prediction intervals; SVR gives point predictions.
34
What is the linear kernel in SVM, and when should you use it?
KnownRevise
The linear kernel is simply the dot product, K(x, z) = x·z, so the SVM learns a straight hyperplane w·x + b = 0 in the original feature space with no implicit mapping. It is the fastest and most interpretable SVM: training scales almost linearly with data size using specialized solvers, the model is just a weight per feature, and it is often as accurate as non-linear kernels on high-dimensional data such as text, where classes are usually linearly separable.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
When to Use It
Situation
Why linear works well
Many features relative to samples (text, genomics)
High-dimensional data is often linearly separable; non-linear kernels add overfitting risk
Large datasets (100k+ samples)
LinearSVC / SGDClassifier scale to millions of rows
Need for interpretability
Each weight shows a feature’s direction and strength
Low latency
Prediction is one dot product
Baseline
Always worth trying before RBF
Use a non-linear kernel when there are few features, many samples relative to features, and evidence that the boundary is curved.
Implementations
Class
Solver
Notes
LinearSVC
liblinear
Fast; squared hinge by default; supports L1 penalty
SGDClassifier(loss="hinge")
SGD
Streaming, partial_fit, huge data
SVC(kernel="linear")
libsvm
Slow for large n; gives support vectors and dual coefficients
Code Example: Text Classification
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC
cats = ["sci.space", "rec.autos", "comp.graphics"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))
model = make_pipeline(TfidfVectorizer(sublinear_tf=True, min_df=2, ngram_range=(1, 2)), LinearSVC(C=0.5))
model.fit(train.data, train.target)
print(classification_report(test.target, model.predict(test.data), target_names=train.target_names))
# Interpret: top weighted terms per class
vocab = np.array(model[0].get_feature_names_out())
for i, name in enumerate(train.target_names):
print(name, "->", ", ".join(vocab[np.argsort(model[-1].coef_[i])[-8:]]))
The data is downloaded on first use. Tens of thousands of TF-IDF features train in seconds.
Common Pitfalls
Reading weights as importance without scaling numeric features first.
Using SVC(kernel="linear") for large data instead of LinearSVC.
Convergence warnings in LinearSVC; scale features or raise max_iter.
35
What is the RBF kernel, and how does it transform the feature space?
KnownRevise
The Radial Basis Function (RBF), or Gaussian, kernel measures similarity between two points as K(x, z) = exp(-gamma * ||x - z||^2): 1 when the points coincide and decaying toward 0 as they move apart. It corresponds to an implicit mapping into an infinite-dimensional feature space, where the SVM finds a linear boundary that appears as a flexible, curved boundary in the original space. In effect, the decision function becomes a weighted sum of Gaussian “bumps” centred on the support vectors.
The squared distance in the exponent means similarity depends only on how far apart points are (it is “radial”).
Expanding exp(-gamma ||x - z||^2) as a Taylor series gives polynomial terms of every degree, so the implicit feature map has infinitely many dimensions.
All points are mapped onto a sphere of radius 1 in that space (K(x, x) = 1).
Decision function: f(x) = sum_i alpha_i y_i exp(-gamma ||x_i - x||^2) + b, so each support vector contributes a bump whose width is set by gamma.
Parameters
Parameter
Role
Too small
Too large
gamma (= 1 / (2 sigma^2))
Width of each bump
Very smooth, near-linear: underfit
Bumps around single points: overfit
C
Penalty for violations
Wide margin, underfit
Fits noise
Code Example
import numpy as np
from sklearn.datasets import make_circles
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
x = np.array([[0.0, 0.0]])
points = np.array([[0.0, 0.0], [0.5, 0.0], [1.0, 0.0], [2.0, 0.0]])
for gamma in (0.5, 2.0):
print(f"gamma={gamma}: similarities {rbf_kernel(x, points, gamma=gamma).round(3)}")
# Manual decision function equals sklearn's
X, y = make_circles(n_samples=300, factor=0.5, noise=0.1, random_state=42)
svc = SVC(kernel="rbf", gamma=1.0, C=1.0).fit(X, y)
K = rbf_kernel(X[:5], svc.support_vectors_, gamma=1.0)
manual = K @ svc.dual_coef_[0] + svc.intercept_[0]
print("matches decision_function:", np.allclose(manual, svc.decision_function(X[:5])))
for kernel in ("linear", "rbf"):
print(kernel, round(cross_val_score(SVC(kernel=kernel), X, y, cv=5).mean(), 3))
The concentric circles cannot be separated by a line, but the RBF SVM separates them easily.
When to Use It
Default non-linear kernel for low- to medium-dimensional, dense numeric data.
Less useful for very high-dimensional sparse text, where linear kernels are as accurate and faster.
Common Pitfalls
Unscaled features, which make distances (and gamma) dominated by one feature.
Tuning gamma without C.
Large datasets: exact RBF SVMs are slow; use Nystroem approximations.
36
When would you use a polynomial kernel in SVM?
KnownRevise
The polynomial kernel, K(x, z) = (gamma * x·z + coef0)^degree, lets an SVM learn boundaries built from interactions between features up to a chosen degree, as if all products of features up to that degree had been added. Use it when you expect feature interactions of a known, low order to matter, for example pairwise interactions (degree 2), and when the data has been normalized. In practice the RBF kernel is often more accurate and easier to tune, so the polynomial kernel is chosen for specific structure, not as a default.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Parameters
Parameter
Role
Notes
degree
Highest interaction order
2-3 typical; higher degrees overfit and become unstable
coef0
Weight of lower-order terms
coef0=0 keeps only terms of exactly that degree; coef0=1 includes all lower degrees
gamma
Scales the dot product
Interacts with degree and C
C
Regularization
Tune jointly
Good Use Cases
Situation
Reason
Known pairwise interactions (e.g. price × quantity)
Degree 2 captures them directly
Normalized image features (e.g. digit pixels)
Degree 2-4 polynomial SVMs historically worked well on MNIST
NLP with bag-of-words and feature conjunctions
Degree 2 captures word pairs
When you want an alternative to explicit PolynomialFeatures in high dimensions
Avoids building the huge feature matrix
Code Example
import numpy as np
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
rng = np.random.default_rng(42)
X = rng.normal(size=(800, 4))
y = (X[:, 0] * X[:, 1] + 0.5 * X[:, 2] ** 2 > 0.5).astype(int) # degree-2 structure
for kernel, params in [("linear", {}), ("poly", {"degree": 2, "coef0": 1}), ("poly", {"degree": 5, "coef0": 1}),
("rbf", {})]:
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel=kernel, **params)),
{"svc__C": [0.1, 1, 10]}, cv=5)
grid.fit(X, y)
label = f"{kernel} {params.get('degree', '')}".strip()
print(f"{label:8s} CV accuracy {grid.best_score_:.3f}")
Because the target is built from degree-2 terms, the degree-2 polynomial kernel should do well here; the linear kernel cannot represent the product x0 * x1.
Common Pitfalls
High degrees on unscaled data: values explode and training becomes slow or unstable.
Forgetting coef0: with the default coef0=0, only terms of exactly that degree are used.
Tuning degree, gamma, coef0 and C together without enough data for reliable cross-validation.
37
What is the purpose of the sigmoid kernel in SVM?
KnownRevise
The sigmoid kernel, K(x, z) = tanh(gamma * x·z + coef0), was proposed because an SVM using it has the same form as a two-layer neural network with tanh hidden units, one per support vector. In practice it is rarely the best choice: it is not positive semi-definite for many parameter values, so it is not always a valid kernel and the optimization can behave poorly, and RBF or linear kernels usually perform as well or better. It is mainly of historical and theoretical interest.
Decision function = weighted sum of tanh units over support vectors
Validity
PSD only for some (gamma, coef0) combinations, e.g. small gamma and negative coef0
Typical performance
Often below RBF; sensitive to parameters
Parameters
gamma (> 0), coef0 (often ≤ 0), C
When It Might Be Tried
As one candidate in a kernel search when the data was preprocessed to small values.
When teaching the connection between SVMs and neural networks.
In legacy systems or papers that used it.
Code Example: Checking Validity and Performance
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics.pairwise import sigmoid_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
Xs = StandardScaler().fit_transform(X)[:200]
for gamma, coef0 in [(0.01, -1.0), (0.01, 1.0), (1.0, 0.0)]:
K = sigmoid_kernel(Xs, gamma=gamma, coef0=coef0)
min_eig = np.linalg.eigvalsh(K).min()
print(f"gamma={gamma}, coef0={coef0}: smallest eigenvalue {min_eig:.3f} "
f"({'PSD' if min_eig > -1e-8 else 'not PSD'})")
for kernel, params in [("sigmoid", {"gamma": 0.01, "coef0": -1.0}), ("sigmoid", {}), ("rbf", {}), ("linear", {})]:
acc = cross_val_score(make_pipeline(StandardScaler(), SVC(kernel=kernel, **params)), X, y, cv=5).mean()
print(f"{kernel:8s} {params!s:32s} CV accuracy {acc:.3f}")
A negative smallest eigenvalue means the kernel matrix is not positive semi-definite for that data and parameter setting.
Interview Tips
Mention the neural-network analogy and the PSD problem.
Recommend RBF or linear in practice, and a real neural network if that kind of model is wanted.
38
How do you create a custom SVM kernel, and what should you consider?
KnownRevise
A custom kernel lets an SVM use a similarity measure designed for your data, such as a string kernel for DNA sequences, a histogram kernel for image descriptors, or a combination of kernels for different feature groups. In scikit-learn you either pass a Python function that returns the kernel (Gram) matrix between two sets of samples, or set kernel="precomputed" and supply the matrix yourself. The key requirement is that the kernel is positive semi-definite (PSD), so it corresponds to an inner product in some feature space and the optimization stays convex.
Valid kernels stay valid under these operations: - Sum: K1 + K2 - Positive scaling: c * K with c > 0 - Product: K1 * K2 - Exponential of a valid kernel - Applying a valid kernel to a fixed feature transform: K(phi(x), phi(z))
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.metrics.pairwise import linear_kernel, rbf_kernel
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=500, n_features=10, n_informative=6, random_state=42)
X = StandardScaler().fit_transform(X) # for brevity; in real work, fit the scaler inside each fold
# 1. Callable kernel: RBF on the first 5 features + linear on the rest
def mixed_kernel(A, B):
return rbf_kernel(A[:, :5], B[:, :5], gamma=0.2) + 0.5 * linear_kernel(A[:, 5:], B[:, 5:])
print("custom kernel CV:", round(cross_val_score(SVC(kernel=mixed_kernel), X, y, cv=5).mean(), 3))
# 2. Precomputed kernel: you build train and test Gram matrices yourself
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
K_train = mixed_kernel(X_train, X_train) # (n_train, n_train)
K_test = mixed_kernel(X_test, X_train) # (n_test, n_train): rows = test, columns = train
svc = SVC(kernel="precomputed").fit(K_train, y_train)
print("precomputed test accuracy:", round(svc.score(K_test, y_test), 3))
# 3. Check positive semi-definiteness on a sample
eigvals = np.linalg.eigvalsh(K_train)
print("smallest eigenvalue:", round(float(eigvals.min()), 6), "(PSD)" if eigvals.min() > -1e-8 else "(not PSD)")
Considerations
Consideration
Detail
Validity (PSD)
Non-PSD kernels can make the solver fail or give poor solutions; check eigenvalues
Symmetry
K(x, z) must equal K(z, x)
Normalization
Normalize so K(x, x) = 1 to avoid scale effects: K(x,z) / sqrt(K(x,x) K(z,z))
Computational cost
Callable kernels run in Python; precompute for repeated fits
Hyperparameters
Kernel parameters need tuning like gamma
Precomputed shapes
Test matrix is test × train, built against the training samples
Leakage
Kernel parameters learned from data (e.g. bandwidth) must come from training data only
Common Pitfalls
Passing a precomputed test matrix of shape test × test.
Using similarity measures that are not PSD (e.g. some edit-distance transforms) without correction.
Very slow Python kernels on large datasets.
39
Implement a basic linear SVM from scratch in Python.
KnownRevise
A linear SVM from scratch can be trained by minimizing the regularized hinge loss with (sub)gradient descent: the objective is (lambda/2)||w||^2 + mean(max(0, 1 - y_i (w·x_i + b))). For each point that violates the margin (y_i f(x_i) < 1), the gradient pushes w toward y_i x_i; for all points, the regularization term shrinks w. The implementation below uses NumPy, labels in {-1, +1}, mini-batches and the Pegasos-style decreasing learning rate, then checks the result against scikit-learn.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+ (for data and comparison) (Python 3.12)
Gradient in Words
Regularization term: gradient lambda * w.
Hinge term for a violating point: gradient -y_i x_i for w and -y_i for b; zero for points outside the margin.
Update: w ← w - eta * (lambda * w - mean over violators of y_i x_i).
Code Example
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
class LinearSVM:
def __init__(self, lam=0.01, epochs=50, batch_size=32, random_state=42):
self.lam, self.epochs, self.batch_size = lam, epochs, batch_size
self.rng = np.random.default_rng(random_state)
def fit(self, X, y):
y = np.where(y > 0, 1.0, -1.0) # labels as -1/+1
n, d = X.shape
self.w, self.b = np.zeros(d), 0.0
t = 0
for _ in range(self.epochs):
for batch in np.array_split(self.rng.permutation(n), max(1, n // self.batch_size)):
t += 1
eta = 1.0 / (self.lam * t) # Pegasos learning-rate schedule
Xb, yb = X[batch], y[batch]
violators = yb * (Xb @ self.w + self.b) < 1
grad_w = self.lam * self.w - (yb[violators, None] * Xb[violators]).sum(axis=0) / len(batch)
grad_b = -yb[violators].sum() / len(batch)
self.w -= eta * grad_w
self.b -= eta * grad_b
return self
def decision_function(self, X):
return X @ self.w + self.b
def predict(self, X):
return (self.decision_function(X) >= 0).astype(int)
def objective(self, X, y):
y = np.where(y > 0, 1.0, -1.0)
return 0.5 * self.lam * self.w @ self.w + np.maximum(0, 1 - y * self.decision_function(X)).mean()
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
svm = LinearSVM(lam=0.01, epochs=50).fit(X_train, y_train)
print("objective:", round(svm.objective(X_train, y_train), 4))
print("from scratch accuracy:", round(accuracy_score(y_test, svm.predict(X_test)), 3))
# Equivalent scikit-learn model: C = 1 / (lambda * n_samples) with plain hinge loss
ref = LinearSVC(C=1 / (0.01 * len(X_train)), loss="hinge", max_iter=10000).fit(X_train, y_train)
print("LinearSVC accuracy: ", round(ref.score(X_test, y_test), 3))
The two accuracies should be close; the from-scratch version regularizes the bias differently and stops after a fixed number of epochs.
Common Pitfalls
Using 0/1 labels in the hinge formula.
Not scaling features, which makes gradient descent slow and the margin distorted.
A constant large learning rate, which makes the weights oscillate; decay it or tune it.
40
Write a Python function to select an SVM's C parameter using cross-validation.
KnownRevise
Selecting C with cross-validation means training the SVM for a range of C values, scoring each with k-fold cross-validation on the training data only, and picking the value with the best mean score, optionally preferring a simpler (smaller C) model when scores are within one standard error. Because C acts multiplicatively, search it on a logarithmic grid. Scaling must happen inside each fold, so the function below builds a Pipeline.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def select_C(X, y, C_values=None, kernel="rbf", cv=5, scoring="roc_auc", one_se_rule=True, random_state=42):
"""Return the chosen C and a table of cross-validated scores.
With one_se_rule=True, picks the smallest C whose mean score is within one
standard error of the best mean score (a simpler, more regularized model).
"""
if C_values is None:
C_values = np.logspace(-3, 3, 13)
folds = StratifiedKFold(n_splits=cv, shuffle=True, random_state=random_state)
results = []
for C in C_values:
model = make_pipeline(StandardScaler(), SVC(kernel=kernel, C=C))
scores = cross_val_score(model, X, y, cv=folds, scoring=scoring, n_jobs=-1)
results.append((C, scores.mean(), scores.std() / np.sqrt(cv)))
best_C, best_mean, best_se = max(results, key=lambda r: r[1])
if one_se_rule:
chosen = min(C for C, mean, _ in results if mean >= best_mean - best_se)
else:
chosen = best_C
return chosen, results
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
C, table = select_C(X_train, y_train)
for c, mean, se in table:
print(f"C={c:>9.4f} ROC AUC {mean:.4f} ± {se:.4f}")
print("chosen C:", C)
final = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=C)).fit(X_train, y_train)
print("test accuracy:", round(final.score(X_test, y_test), 3))
The same search can be written as GridSearchCV(pipe, {"svc__C": C_values}, cv=folds); the hand-written version makes the one-standard-error rule explicit.
Notes
With an RBF kernel, gamma also matters; search C and gamma together for the final model.
Use stratified folds for classification so each fold keeps the class balance.
If the best C is at the edge of the grid, widen the grid.
Common Pitfalls
Using the test set inside the selection function.
Scaling before cross-validation, which leaks fold information.
Choosing by training accuracy, which always prefers the largest C.
41
Code an SVM in scikit-learn to classify text using TF-IDF features.
KnownRevise
Text classification with an SVM usually pairs TF-IDF features with a linear SVM: TF-IDF turns each document into a sparse vector weighting words that are frequent in the document but rare in the corpus, and a linear SVM separates classes well in that high-dimensional space while training in seconds. Putting the vectorizer and classifier in one Pipeline keeps vocabulary building inside each cross-validation fold and makes the model accept raw strings.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Code Example
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
categories = ["comp.graphics", "rec.sport.hockey", "sci.med", "talk.politics.guns"]
train = fetch_20newsgroups(subset="train", categories=categories, remove=("headers", "footers", "quotes"),
random_state=42) # downloads on first use
test = fetch_20newsgroups(subset="test", categories=categories, remove=("headers", "footers", "quotes"),
random_state=42)
pipe = Pipeline([
("tfidf", TfidfVectorizer(sublinear_tf=True, stop_words="english", min_df=2)),
("svm", LinearSVC(class_weight="balanced")),
])
grid = GridSearchCV(pipe, {"tfidf__ngram_range": [(1, 1), (1, 2)], "svm__C": [0.1, 0.5, 1, 5]},
cv=5, scoring="f1_macro", n_jobs=-1)
grid.fit(train.data, train.target)
print("best parameters:", grid.best_params_, "CV macro F1:", round(grid.best_score_, 3))
pred = grid.predict(test.data)
print(classification_report(test.target, pred, target_names=train.target_names, digits=3))
print(confusion_matrix(test.target, pred))
# Most indicative terms per class
model = grid.best_estimator_
terms = np.array(model.named_steps["tfidf"].get_feature_names_out())
for i, name in enumerate(train.target_names):
top = np.argsort(model.named_steps["svm"].coef_[i])[-8:][::-1]
print(f"{name:20s} {', '.join(terms[top])}")
print(grid.predict(["The goalie made thirty saves in the playoff game"]))
Removing headers, footers and quotes stops the model from relying on metadata such as email addresses, which would inflate the scores.
Design Choices
Choice
Reason
LinearSVC instead of RBF SVC
Sparse, high-dimensional text is usually linearly separable; much faster
sublinear_tf=True
Uses 1 + log(tf) so repeated words don’t dominate
min_df=2
Drops typos and very rare tokens
Bigrams
Capture short phrases (“not good”)
class_weight="balanced"
Handles uneven class sizes
Macro F1
Treats every class equally
Common Pitfalls
Fitting the vectorizer on all documents before splitting, which leaks vocabulary and IDF statistics.
Scaling TF-IDF with StandardScaler, which destroys sparsity; TF-IDF rows are already L2-normalized.
Using SVC(probability=True) when only labels are needed; use CalibratedClassifierCV if probabilities are required.
42
Build a multi-class SVM classifier using the one-vs-one strategy.
KnownRevise
The one-vs-one (OvO) strategy trains a separate binary SVM for every pair of classes, K(K-1)/2 classifiers for K classes, and predicts by majority vote among them. Each classifier sees only the data of its two classes, so the individual problems are small, which suits kernel SVMs whose cost grows faster than linearly with the number of samples. scikit-learn’s SVC uses OvO internally; the example below also builds it explicitly with OneVsOneClassifier and by hand to show how voting works.
from itertools import combinations
import numpy as np
from sklearn.datasets import load_wine
from sklearn.metrics import accuracy_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsOneClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_wine(return_X_y=True) # 3 classes -> 3 pairwise classifiers
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
# 1. scikit-learn's wrapper
ovo = make_pipeline(StandardScaler(), OneVsOneClassifier(SVC(kernel="rbf", C=1.0)))
ovo.fit(X_train, y_train)
print("pairwise models:", len(ovo[-1].estimators_))
print(classification_report(y_test, ovo.predict(X_test), digits=3))
# 2. Manual one-vs-one with voting
scaler = StandardScaler().fit(X_train)
Xtr, Xte = scaler.transform(X_train), scaler.transform(X_test)
classes = np.unique(y_train)
votes = np.zeros((len(Xte), len(classes)))
scores = np.zeros((len(Xte), len(classes))) # summed confidences break ties
for a, b in combinations(range(len(classes)), 2):
mask = np.isin(y_train, [classes[a], classes[b]])
clf = SVC(kernel="rbf", C=1.0).fit(Xtr[mask], (y_train[mask] == classes[b]).astype(int))
d = clf.decision_function(Xte) # > 0 means class b wins
votes[:, b] += d > 0
votes[:, a] += d <= 0
scores[:, b] += d
scores[:, a] -= d
pred = classes[np.argmax(votes + 1e-3 * np.tanh(scores), axis=1)]
print("manual OvO accuracy:", round(accuracy_score(y_test, pred), 3))
# 3. SVC does the same internally
svc = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, decision_function_shape="ovo")).fit(X_train, y_train)
print("SVC accuracy:", round(svc.score(X_test, y_test), 3),
"| pairwise scores per sample:", svc.decision_function(X_test[:1]).shape[1])
The tiny confidence term only breaks ties in the vote; it never overrides a clear majority.
OvO Versus OvR
Aspect
One-vs-one
One-vs-rest
Number of models
K(K-1)/2
K
Data per model
Two classes only
All data
Class balance per model
Roughly balanced
Imbalanced (one vs many)
Best with
Kernel SVMs, moderate K
Linear models, many classes
Common Pitfalls
Scaling inside each pair with different statistics; scale once on the training data.
Ignoring ties in voting, which are common with few classes.
Many classes (e.g. 100 → 4,950 models): OvR or a linear model may be more practical.
43
Show how different kernels change SVM decision boundaries on 2D data.
KnownRevise
Plotting SVM decision boundaries on 2D data is the clearest way to see what each kernel can represent: a linear kernel draws a straight line, a polynomial kernel draws curves built from feature interactions, and an RBF kernel draws flexible, local boundaries whose smoothness depends on gamma. The script below trains each kernel on a non-linear dataset, colours the predicted regions, marks the support vectors and prints cross-validated accuracy, so the visual impression can be checked against numbers.
import matplotlib
matplotlib.use("Agg") # render to a file without a display
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=300, noise=0.25, random_state=42)
configs = [
("linear", SVC(kernel="linear", C=1)),
("poly, degree 3", SVC(kernel="poly", degree=3, coef0=1, C=1)),
("rbf, gamma 0.5", SVC(kernel="rbf", gamma=0.5, C=1)),
("rbf, gamma 50", SVC(kernel="rbf", gamma=50, C=1)),
]
fig, axes = plt.subplots(1, len(configs), figsize=(18, 4.5))
for ax, (name, svc) in zip(axes, configs):
model = make_pipeline(StandardScaler(), svc)
cv_acc = cross_val_score(model, X, y, cv=5).mean()
model.fit(X, y)
DecisionBoundaryDisplay.from_estimator(model, X, response_method="predict", alpha=0.3, ax=ax,
cmap="coolwarm")
DecisionBoundaryDisplay.from_estimator(model, X, response_method="decision_function", plot_method="contour",
levels=[-1, 0, 1], linestyles=["--", "-", "--"], colors="k", ax=ax)
ax.scatter(X[:, 0], X[:, 1], c=y, cmap="coolwarm", edgecolors="k", s=20)
sv = model.named_steps["standardscaler"].inverse_transform(model.named_steps["svc"].support_vectors_)
ax.scatter(sv[:, 0], sv[:, 1], s=80, facecolors="none", edgecolors="k", linewidths=1)
ax.set_title(f"{name}\nCV acc {cv_acc:.2f}, {len(sv)} SVs")
print(f"{name:15s} CV accuracy {cv_acc:.3f}, support vectors {len(sv)}")
plt.tight_layout()
plt.savefig("svm_kernels.png", dpi=120)
The solid line is the decision boundary (f(x) = 0) and the dashed lines are the margins (f(x) = ±1); circled points are support vectors.
What to Look For
Kernel
Expected boundary on moons
Linear
Straight line; misses the interlocking shape
Polynomial (degree 3)
Smooth S-like curve
RBF, moderate gamma
Follows the moons smoothly; usually best
RBF, very large gamma
Tight islands around points; many SVs; overfits
Common Pitfalls
Plotting in scaled space but scattering raw points (or the reverse); keep the pipeline and inverse-transform the support vectors as above.
Judging the best kernel from the picture alone; check cross-validated scores.
44
Implement an SVM in Python using stochastic gradient descent.
KnownRevise
Training an SVM with stochastic gradient descent (SGD) minimizes the primal objective, L2 regularization plus hinge loss, by updating the weights one example (or mini-batch) at a time using the hinge subgradient. It scales to millions of samples and streaming data because each step costs only O(d). Below is a from-scratch NumPy version with per-sample updates and a learning-rate schedule, compared with scikit-learn’s SGDClassifier(loss="hinge"), which implements the same idea.
For each example (x, y) with y ∈ {-1, +1} and learning rate eta: - If y (w·x + b) < 1 (inside the margin or wrong): w ← w - eta (lambda w - y x) and b ← b + eta y. - Otherwise: w ← w - eta lambda w (only shrinkage from regularization).
Code Example
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
def sgd_svm(X, y, lam=1e-4, epochs=10, eta0=0.01, random_state=42):
"""Linear SVM by per-sample SGD on L2-regularized hinge loss. y in {0, 1}."""
rng = np.random.default_rng(random_state)
y_pm = np.where(y == 1, 1.0, -1.0)
n, d = X.shape
w, b, t = np.zeros(d), 0.0, 0
history = []
for epoch in range(epochs):
for i in rng.permutation(n):
t += 1
eta = eta0 / (1 + eta0 * lam * t) # slowly decaying step size
if y_pm[i] * (X[i] @ w + b) < 1:
w -= eta * (lam * w - y_pm[i] * X[i])
b += eta * y_pm[i]
else:
w -= eta * lam * w
hinge = np.maximum(0, 1 - y_pm * (X @ w + b)).mean()
history.append(0.5 * lam * w @ w + hinge)
return w, b, history
X, y = make_classification(n_samples=20000, n_features=30, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
w, b, history = sgd_svm(X_train, y_train)
print("objective per epoch:", [round(h, 4) for h in history])
print("from-scratch accuracy:", round(((X_test @ w + b >= 0).astype(int) == y_test).mean(), 3))
ref = SGDClassifier(loss="hinge", alpha=1e-4, max_iter=10, tol=None, random_state=42).fit(X_train, y_train)
print("SGDClassifier accuracy:", round(ref.score(X_test, y_test), 3))
The objective should fall over the first epochs and then level off; both accuracies should be similar.
Practical Notes
Shuffle every epoch; ordered data (e.g. sorted by label) breaks SGD.
SGDClassifier adds averaging (average=True), early stopping, partial_fit for streams and class_weight.
For non-linear boundaries, apply RBFSampler or Nystroem features first.
Common Pitfalls
Unscaled features, which make one step size wrong for every feature.
Too large a learning rate, which makes the loss oscillate.
Comparing alpha in SGDClassifier directly with C in SVC; roughly alpha = 1 / (C * n_samples).
45
Write a script to visualize the support vectors of a trained SVM.
KnownRevise
Visualizing support vectors shows which training points actually define an SVM: the points on the margin lines and those violating the margin. The script below trains a linear SVM on 2D data, draws the decision boundary and both margins, circles the support vectors, distinguishes margin support vectors (0 < alpha < C) from bounded ones (alpha = C), and, for higher-dimensional data, projects onto two principal components to show where the support vectors lie.
In the PCA view, support vectors should cluster where the two classes meet; points far from that region are not needed by the model.
Reading the Plot
Green circles lie exactly on the dashed margin lines.
Orange circles are inside the margin or misclassified; their count grows as C decreases.
Re-running with a smaller C widens the margin and adds support vectors.
Common Pitfalls
Plotting support_vectors_ from a scaled pipeline on unscaled axes; use indices (support_) into the original data as above.
Reading the PCA plot as the true boundary: it is a projection of a 30-dimensional model.
46
Write a grid-search function to find the best SVM kernel and parameters.
KnownRevise
A grid search over SVM kernels must account for the fact that each kernel has its own parameters: gamma for RBF, degree and coef0 for polynomial, and only C for linear. scikit-learn’s GridSearchCV accepts a list of parameter grids, one per kernel, so no meaningless combinations are tried. The function below wraps this with a scaling Pipeline, stratified folds, a configurable metric and a summary of the best result for each kernel.
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def svm_kernel_search(X, y, scoring="f1", cv=5, n_jobs=-1, random_state=42):
"""Grid-search kernel, C and kernel-specific parameters for an SVC.
Returns the fitted GridSearchCV object and a DataFrame with the best
cross-validated score per kernel.
"""
pipe = Pipeline([("scale", StandardScaler()), ("svc", SVC())])
C_values = np.logspace(-2, 2, 5)
param_grid = [
{"svc__kernel": ["linear"], "svc__C": C_values},
{"svc__kernel": ["rbf"], "svc__C": C_values, "svc__gamma": ["scale", 1e-3, 1e-2, 1e-1]},
{"svc__kernel": ["poly"], "svc__C": C_values, "svc__degree": [2, 3],
"svc__gamma": ["scale"], "svc__coef0": [0.0, 1.0]},
{"svc__kernel": ["sigmoid"], "svc__C": C_values, "svc__gamma": [1e-3, 1e-2], "svc__coef0": [-1.0, 0.0]},
]
folds = StratifiedKFold(n_splits=cv, shuffle=True, random_state=random_state)
search = GridSearchCV(pipe, param_grid, scoring=scoring, cv=folds, n_jobs=n_jobs)
search.fit(X, y)
results = pd.DataFrame(search.cv_results_)
results["kernel"] = results["param_svc__kernel"]
summary = (results.sort_values("mean_test_score", ascending=False)
.groupby("kernel", as_index=False).first()[["kernel", "mean_test_score", "std_test_score", "params"]]
.sort_values("mean_test_score", ascending=False))
return search, summary
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
search, summary = svm_kernel_search(X_train, y_train)
pd.set_option("display.max_colwidth", 120)
print(summary.to_string(index=False))
print("overall best:", search.best_params_)
print("test F1 of the refitted best model:", round(search.score(X_test, y_test), 3))
Notes
GridSearchCV refits the best configuration on all training data (refit=True), so search.predict is ready to use.
The grid has 5 + 20 + 20 + 20 = 65 candidates × 5 folds = 325 fits; for larger grids use RandomizedSearchCV or HalvingGridSearchCV.
Compare kernels by their best scores and standard deviations; if the linear kernel is within noise of RBF, prefer it for speed and simplicity.
Common Pitfalls
One flat grid with gamma and degree for every kernel, which wastes fits on ignored parameters.
Evaluating on the training data used for the search; keep a held-out test set.
Scaling outside the pipeline.
47
How are quasi-Newton methods used for SVM training?
KnownRevise
Quasi-Newton methods such as BFGS and its memory-efficient variant L-BFGS approximate second-order (curvature) information from successive gradients, converging in far fewer iterations than plain gradient descent. For SVMs they are applied to the primal problem of linear SVMs, usually with a smooth loss such as the squared hinge or a smoothed (Huberized) hinge, since the standard hinge is not differentiable at the margin. L-BFGS variants are used in large-scale and distributed linear SVM training, for example OWL-QN in Spark MLlib’s LinearSVC.
| Aspect | Detail | |—|—| | Objective | (1/2)||w||^2 + C sum max(0, 1 - y_i w·x_i)^2 (squared hinge) is once differentiable | | Advantage | Superlinear convergence; few passes over the data; easy to parallelize gradient computation | | Memory | L-BFGS stores only the last m gradient pairs (e.g. m = 10) | | Limitation | Plain hinge is non-smooth; kernel SVMs are usually solved in the dual with SMO instead | | Variants | OWL-QN for L1 regularization; truncated Newton (as in liblinear’s TRON for L2-loss SVMs) |
Code Example: Linear SVM via L-BFGS
import numpy as np
from scipy.optimize import minimize
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
y_pm = np.where(y_train == 1, 1.0, -1.0)
C = 0.1
Xa = np.hstack([X_train, np.ones((len(X_train), 1))]) # bias as an extra feature
def objective(theta):
margins = 1 - y_pm * (Xa @ theta)
active = margins > 0
loss = 0.5 * theta @ theta + C * np.sum(margins[active] ** 2)
grad = theta - 2 * C * (y_pm[active] * margins[active]) @ Xa[active]
return loss, grad
res = minimize(objective, x0=np.zeros(Xa.shape[1]), jac=True, method="L-BFGS-B")
theta = res.x
pred = (np.hstack([X_test, np.ones((len(X_test), 1))]) @ theta >= 0).astype(int)
print("L-BFGS converged:", res.success, "in", res.nit, "iterations")
print("L-BFGS accuracy:", round((pred == y_test).mean(), 3))
ref = LinearSVC(C=C, loss="squared_hinge", max_iter=10000).fit(X_train, y_train)
print("LinearSVC accuracy:", round(ref.score(X_test, y_test), 3))
print("weight correlation:", round(np.corrcoef(theta[:-1], ref.coef_[0])[0, 1], 4))
The bias is regularized here, as in liblinear’s default, so the weights should be nearly identical to LinearSVC‘s.
Compared with Other Solvers
Solver
Problem form
Typical use
SMO (libsvm)
Dual, any kernel
SVC
Dual coordinate descent (liblinear)
Dual, linear
LinearSVC default
Trust-region Newton (liblinear)
Primal, squared hinge
Dense, well-conditioned problems
L-BFGS / OWL-QN
Primal, smooth loss
Distributed linear SVMs
SGD
Primal, hinge
Streaming and huge data
Common Pitfalls
Applying L-BFGS to the plain hinge loss and getting poor convergence because of the kink.
Unscaled features, which make the curvature approximation poor.
48
What is Sequential Minimal Optimization (SMO), and why does it matter for SVMs?
KnownRevise
Sequential Minimal Optimization (SMO), introduced by John Platt in 1998, solves the SVM dual problem by repeatedly optimizing just two Lagrange multipliers at a time while keeping the others fixed. Two is the minimum, because the constraint sum alpha_i y_i = 0 means changing one multiplier forces a change in another. Each two-variable subproblem has a closed-form solution, so SMO needs no general quadratic programming solver and only a kernel cache instead of the full n × n matrix. It made kernel SVMs practical and is the basis of libsvm, which scikit-learn’s SVC and SVR use.
Pick a multiplier alpha_i that violates the KKT conditions (heuristics choose the worst violator).
Pick a second alpha_j that will give a large step (e.g. maximizing |E_i - E_j|, where E is the prediction error).
Compute the unconstrained optimum for alpha_j along the constraint line, using eta = K_ii + K_jj - 2 K_ij.
Clip alpha_j to the box [L, H] implied by 0 <= alpha <= C and the equality constraint; update alpha_i to compensate.
Update b and the error cache.
Repeat until all KKT conditions hold within a tolerance.
Code Example: Simplified SMO
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.svm import SVC
def simplified_smo(X, y, C=1.0, tol=1e-3, max_passes=20, random_state=42):
"""Simplified SMO for a linear kernel; y in {-1, +1}. Picks the second index at random."""
rng = np.random.default_rng(random_state)
n = len(y)
K = X @ X.T
alpha, b, passes = np.zeros(n), 0.0, 0
while passes < max_passes:
changed = 0
for i in range(n):
E_i = (alpha * y) @ K[:, i] + b - y[i]
if (y[i] * E_i < -tol and alpha[i] < C) or (y[i] * E_i > tol and alpha[i] > 0):
j = rng.choice([k for k in range(n) if k != i])
E_j = (alpha * y) @ K[:, j] + b - y[j]
a_i, a_j = alpha[i], alpha[j]
if y[i] != y[j]:
L, H = max(0, a_j - a_i), min(C, C + a_j - a_i)
else:
L, H = max(0, a_i + a_j - C), min(C, a_i + a_j)
eta = 2 * K[i, j] - K[i, i] - K[j, j]
if L == H or eta >= 0:
continue
alpha[j] = np.clip(a_j - y[j] * (E_i - E_j) / eta, L, H)
if abs(alpha[j] - a_j) < 1e-5:
continue
alpha[i] = a_i + y[i] * y[j] * (a_j - alpha[j])
b1 = b - E_i - y[i] * (alpha[i] - a_i) * K[i, i] - y[j] * (alpha[j] - a_j) * K[i, j]
b2 = b - E_j - y[i] * (alpha[i] - a_i) * K[i, j] - y[j] * (alpha[j] - a_j) * K[j, j]
b = b1 if 0 < alpha[i] < C else b2 if 0 < alpha[j] < C else (b1 + b2) / 2
changed += 1
passes = passes + 1 if changed == 0 else 0
return alpha, b
X, y01 = make_blobs(n_samples=80, centers=2, cluster_std=1.5, random_state=42)
y = np.where(y01 == 1, 1.0, -1.0)
alpha, b = simplified_smo(X, y)
w = (alpha * y) @ X
ref = SVC(kernel="linear", C=1.0).fit(X, y)
print("SMO w:", w.round(3), "b:", round(b, 3), "| SVs:", int((alpha > 1e-5).sum()))
print("sklearn w:", ref.coef_[0].round(3), "b:", round(ref.intercept_[0], 3), "| SVs:", len(ref.support_))
print("prediction agreement:", (np.sign(X @ w + b) == ref.predict(X)).mean())
The simplified version picks the second multiplier at random, so it is slower than libsvm’s heuristics, but it should reach nearly the same solution.
Why It Matters
Memory: no full kernel matrix needed; libsvm caches the rows it uses (cache_size).
Simplicity: analytic updates, no external QP solver.
Speed: practical training for tens of thousands of samples.
Common Pitfalls
Expecting SMO to scale to millions of samples; it is still roughly quadratic or worse.
Loose tolerance (tol) settings that stop early with an inaccurate solution.
49
What is Platt scaling, and why use probabilistic outputs from SVMs?
KnownRevise
Platt scaling converts an SVM’s raw decision scores into probabilities by fitting a logistic (sigmoid) function to them: P(y = 1 | x) = 1 / (1 + exp(A * f(x) + B)), where f(x) is the SVM score and A, B are learned on held-out data. SVMs naturally output distances to the boundary, not probabilities, but many applications need calibrated probabilities: risk-based decisions, cost-sensitive thresholds, combining models, and reporting confidence to users.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Why Probabilities Matter
Need
Example
Decision with costs
Approve a loan if expected profit = P(repay) × gain − (1 − P) × loss > 0
Ranking by risk
Prioritize patients by probability of disease
Model combination
Average probabilities across models in an ensemble
Metrics
Log loss, Brier score and calibration curves need probabilities
In scikit-learn
Option
How
Notes
SVC(probability=True)
Internal 5-fold CV fits Platt’s sigmoid
Slower training; predict_proba can disagree with predict near the boundary
Needs more data (roughly 1,000+ calibration samples)
Code Example
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.datasets import make_classification
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
X, y = make_classification(n_samples=6000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
for method in ("sigmoid", "isotonic"):
model = make_pipeline(StandardScaler(), CalibratedClassifierCV(LinearSVC(C=0.1), method=method, cv=5))
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
frac_pos, mean_pred = calibration_curve(y_test, proba, n_bins=10)
print(f"{method:8s} log loss {log_loss(y_test, proba):.4f} Brier {brier_score_loss(y_test, proba):.4f}")
print(" predicted vs observed per bin:",
[(round(p, 2), round(o, 2)) for p, o in zip(mean_pred, frac_pos)][:5], "...")
A well-calibrated model has observed frequencies close to the predicted probabilities in each bin.
Common Pitfalls
Fitting the sigmoid on the same data used to train the SVM, which gives over-confident probabilities (scikit-learn uses cross-validation to avoid this).
Treating decision_function values as probabilities.
Using probability=True when only labels or rankings (ROC AUC) are needed; it only adds training time.
50
What is the "Reese kernel" in SVM, and is it a standard kernel?
KnownRevise
There is no standard SVM kernel called the “Reese kernel”. It does not appear among the kernels in the main SVM libraries (libsvm, scikit-learn, LIBLINEAR) or in the standard kernel-methods literature, so the name is most likely a misremembering or a mix-up with another kernel. In an interview, the best answer is to say so plainly, then explain which kernels are commonly used and how you would evaluate an unfamiliar kernel if someone proposed one.
Find the definition: the formula and its parameters, and a reference.
Check validity: the kernel must be symmetric and positive semi-definite (PSD); test the smallest eigenvalue of its Gram matrix on real data.
Implement it as a callable or precomputed kernel.
Compare it with linear and RBF baselines under the same cross-validation.
Code Example: Testing a Proposed Kernel
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def proposed_kernel(A, B, gamma=0.05, alpha=1.0):
"""Example 'unfamiliar' kernel: rational quadratic, (1 + gamma * d^2 / (2 * alpha)) ** -alpha."""
sq = (A ** 2).sum(1)[:, None] + (B ** 2).sum(1)[None, :] - 2 * A @ B.T
return (1 + gamma * np.maximum(sq, 0) / (2 * alpha)) ** -alpha
X, y = load_breast_cancer(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
K = proposed_kernel(Xs[:300], Xs[:300])
print("symmetric:", np.allclose(K, K.T), "| smallest eigenvalue:", round(float(np.linalg.eigvalsh(K).min()), 6))
for name, svc in [("proposed", SVC(kernel=proposed_kernel)), ("rbf", SVC(kernel="rbf")),
("linear", SVC(kernel="linear"))]:
score = cross_val_score(make_pipeline(StandardScaler(), svc), X, y, cv=5).mean()
print(f"{name:8s} CV accuracy {score:.3f}")
Interview Tips
It is better to say “I’m not aware of a standard kernel by that name” than to invent a description.
Show you can still reason about any kernel: validity (PSD), parameters, cost and empirical comparison.
51
How would you build an anomaly detection system with a one-class SVM?
KnownRevise
A one-class SVM learns a boundary around normal data only: it maps the data with a kernel (usually RBF) and finds the smallest region, separated from the origin with maximum margin, that contains most of the training points. New points outside the region are flagged as anomalies. It suits problems where anomalies are rare, varied or unlabelled, such as equipment faults, fraud or intrusions. The parameter nu sets an upper bound on the fraction of training points treated as outliers (and a lower bound on the fraction of support vectors), and gamma controls how tightly the boundary wraps the data.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
System Design
Step
Details
1. Data
Collect mostly normal operating data; remove known faults from training
2. Features
Domain features (rolling means, variances, frequency bands for sensors); scale them
3. Model
OneClassSVM(kernel="rbf", nu=..., gamma=...) in a Pipeline
4. Tuning
If a few labelled anomalies exist, tune nu/gamma on a validation set for precision/recall; otherwise set nu to the expected anomaly rate
5. Scoring
decision_function: negative = outside the boundary; rank alerts by score
6. Thresholding
Adjust the threshold to the alert budget
7. Monitoring
Track alert rate and drift; retrain on recent normal data
Code Example: Machine Sensor Monitoring
import numpy as np
from sklearn.metrics import average_precision_score, classification_report
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import OneClassSVM
rng = np.random.default_rng(42)
# Normal operation: temperature and vibration are correlated
temp = rng.normal(70, 3, 2000)
vib = 0.05 * temp + rng.normal(0, 0.2, 2000)
normal = np.column_stack([temp, vib])
# Faults: overheating, or vibration unusually high for the temperature
faults = np.vstack([np.column_stack([rng.normal(85, 2, 30), rng.normal(4.3, 0.2, 30)]),
np.column_stack([rng.normal(70, 3, 30), rng.normal(5.5, 0.3, 30)])])
X_train = normal[:1500] # normal data only
X_test = np.vstack([normal[1500:], faults])
y_test = np.r_[np.zeros(500), np.ones(len(faults))] # 1 = fault (for evaluation only)
detector = make_pipeline(StandardScaler(), OneClassSVM(kernel="rbf", nu=0.01, gamma=0.5))
detector.fit(X_train)
scores = -detector.decision_function(X_test) # higher = more anomalous
pred = (detector.predict(X_test) == -1).astype(int)
print(classification_report(y_test, pred, target_names=["normal", "fault"], digits=3))
print("average precision of anomaly scores:", round(average_precision_score(y_test, scores), 3))
print("training points flagged:", round((detector.predict(X_train) == -1).mean(), 3), "(about nu or less)")
The second fault type has a normal temperature and a vibration level that is only abnormal relative to temperature, which a one-class SVM can catch because it models the joint shape of the data.
Alternatives
Isolation Forest (fast, scales well), Local Outlier Factor, autoencoders for high-dimensional signals, and SGDOneClassSVM with kernel approximation for large datasets.
Common Pitfalls
Training data contaminated with anomalies, which the model then learns as normal.
Unscaled features, which distort the RBF boundary.
A very large gamma that wraps each training point individually and flags all new data.
52
How can SVMs be used for feature selection?
KnownRevise
SVMs support feature selection in three main ways: SVM-RFE (recursive feature elimination), which repeatedly trains a linear SVM and removes the features with the smallest absolute weights; L1-regularized linear SVMs, whose penalty drives many weights exactly to zero so only useful features remain; and wrapper or model-agnostic methods such as permutation importance with a kernel SVM. Feature selection reduces noise and cost and improves interpretability, which is especially valuable in high-dimensional data such as gene expression or text.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Methods
Method
How
Notes
SVM-RFE
RFE / RFECV with LinearSVC or SVC(kernel="linear"); drop lowest ‘
Weights are only comparable when features are scaled, so always standardize first.
Code Example
from sklearn.datasets import make_classification
from sklearn.feature_selection import RFECV, SelectFromModel
from sklearn.inspection import permutation_importance
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = make_classification(n_samples=600, n_features=40, n_informative=6, n_redundant=4,
shuffle=False, random_state=42) # first 10 columns carry the signal
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# 1. SVM-RFE with cross-validation to choose the number of features
rfe = make_pipeline(StandardScaler(), RFECV(LinearSVC(C=0.1, max_iter=5000), step=2, cv=cv, scoring="accuracy"))
rfe.fit(X, y)
selected = rfe[-1].support_.nonzero()[0]
print("RFECV kept", len(selected), "features:", selected)
# 2. L1-penalized LinearSVC
l1 = make_pipeline(StandardScaler(), SelectFromModel(LinearSVC(penalty="l1", dual=False, C=0.05, max_iter=5000)))
l1.fit(X, y)
print("L1 kept:", l1[-1].get_support().nonzero()[0])
# 3. Compare accuracy with and without selection (selection inside the pipeline, so no leakage)
full = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
selected_model = make_pipeline(StandardScaler(),
SelectFromModel(LinearSVC(penalty="l1", dual=False, C=0.05, max_iter=5000)),
SVC(kernel="rbf"))
for name, model in [("all features", full), ("L1-selected", selected_model)]:
print(f"{name:13s} CV accuracy {cross_val_score(model, X, y, cv=cv).mean():.3f}")
# 4. Permutation importance for an RBF SVM
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)
full.fit(X_train, y_train)
imp = permutation_importance(full, X_test, y_test, n_repeats=10, random_state=42)
print("top features by permutation importance:", imp.importances_mean.argsort()[::-1][:8])
Because shuffle=False, the informative and redundant features are columns 0-9, so the selected indices should mostly fall in that range.
Common Pitfalls
Selecting features on the full dataset and then cross-validating the SVM on them, which gives optimistic scores.
Interpreting weights of unscaled features.
Dropping correlated features one at a time: RFE may keep one of a correlated pair arbitrarily.
53
How are SVMs used in bioinformatics and computational biology?
KnownRevise
SVMs have been widely used in bioinformatics because biological datasets often have many features and few samples, such as tens of thousands of gene expression values for a few hundred patients, where margin-based models generalize well. Classic applications include cancer subtype classification from microarray or RNA-seq data, protein function and structure prediction, splice-site and promoter detection in DNA, protein-protein interaction prediction, and drug activity prediction. Specialized string kernels (spectrum, mismatch kernels) let SVMs work directly on DNA and protein sequences.
Linear SVM, often with feature selection (SVM-RFE)
Splice sites, promoters
DNA sequences
String / spectrum kernels
Protein family / remote homology
Protein sequences
Mismatch or profile kernels
Secondary structure
Sequence windows
Multi-class SVM
Protein-protein interaction
Pair features
Pairwise kernels
Drug activity (QSAR)
Molecular fingerprints
Tanimoto kernel SVM
Mass spectrometry diagnostics
Spectra peaks
Linear or RBF SVM
Code Example: Spectrum Kernel for DNA Sequences
The spectrum kernel counts shared k-mers (substrings of length k) between two sequences.
from itertools import product
import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
rng = np.random.default_rng(42)
MOTIF = "TATAAT" # promoter-like motif
def random_seq(n=60, with_motif=False):
seq = "".join(rng.choice(list("ACGT"), n))
if with_motif:
pos = rng.integers(0, n - len(MOTIF))
seq = seq[:pos] + MOTIF + seq[pos + len(MOTIF):]
return seq
seqs = [random_seq(with_motif=i % 2 == 0) for i in range(400)]
y = np.array([int(i % 2 == 0) for i in range(400)])
K_MER = 3
kmers = ["".join(p) for p in product("ACGT", repeat=K_MER)]
index = {k: i for i, k in enumerate(kmers)}
def spectrum_features(seq):
v = np.zeros(len(kmers))
for i in range(len(seq) - K_MER + 1):
v[index[seq[i:i + K_MER]]] += 1
return v / np.linalg.norm(v) # normalize so long sequences don't dominate
X = np.array([spectrum_features(s) for s in seqs]) # explicit k-mer counts: K(a, b) = X_a · X_b
print("spectrum kernel SVM CV accuracy:", round(cross_val_score(SVC(kernel="linear", C=1.0), X, y, cv=5).mean(), 3))
A linear SVM on k-mer count vectors is exactly a spectrum-kernel SVM; for long k, sparse vectors or tries keep it efficient.
Practical Concerns
p >> n: use strong regularization and nested cross-validation; select features inside the folds.
Batch effects: samples from different labs or runs differ systematically; correct or validate across batches.
Interpretability: linear weights or SVM-RFE rankings point to candidate genes or motifs.
Common Pitfalls
Selecting genes on the whole dataset before cross-validation, which gives wildly optimistic accuracy.
Homologous sequences in both training and test sets, which inflates protein-prediction results.
54
How would you apply SVMs to image classification tasks?
KnownRevise
Applying SVMs to image classification requires turning each image into a fixed-length feature vector, because SVMs cannot learn spatial features from raw pixels as CNNs do. Historically this meant hand-crafted descriptors (HOG, SIFT with bag-of-visual-words, colour histograms) fed to RBF or chi-squared kernel SVMs. Today the most effective approach is to extract embeddings from a pretrained CNN or vision transformer and train a linear SVM on them, which gives strong accuracy with few labelled images and little compute.
import numpy as np
from skimage.color import rgb2gray
from skimage.feature import hog
from sklearn.metrics import classification_report
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
rng = np.random.default_rng(42)
def make_image(label):
"""Synthetic 32x32 RGB images: 0 = horizontal stripes (bluish), 1 = vertical stripes (reddish)."""
img = rng.normal(0.4, 0.1, (32, 32, 3))
stripes = (np.arange(32) // 4) % 2
if label == 0:
img[stripes == 1, :, 2] += 0.3
else:
img[:, stripes == 1, 0] += 0.3
return np.clip(img, 0, 1)
labels = rng.integers(0, 2, 600)
images = [make_image(label) for label in labels]
def features(img):
shape = hog(rgb2gray(img), orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2))
colour = np.concatenate([np.histogram(img[..., c], bins=8, range=(0, 1))[0] / img[..., c].size
for c in range(3)])
return np.concatenate([shape, colour])
X = np.array([features(img) for img in images])
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.25, stratify=labels, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC()), {"svc__C": [0.1, 1, 10], "svc__kernel": ["linear", "rbf"]},
cv=5).fit(X_train, y_train)
print(grid.best_params_)
print(classification_report(y_test, grid.predict(X_test), target_names=["horizontal", "vertical"]))
For real photos, replace features with embeddings from a pretrained network (for example ResNet or a ViT with its classification head removed) and use LinearSVC.
Common Pitfalls
Raw pixels of large images: huge dimensions, no translation invariance, slow RBF training.
Inconsistent resizing and colour handling between training and inference.
Too few images per class without augmentation or pretrained features.
55
Describe a financial application where SVMs are used for forecasting.
KnownRevise
A typical financial forecasting application of SVMs is predicting the direction or size of short-term changes, for example whether a stock index or currency pair will rise over the next day (SVM classifier), or next-day volatility (Support Vector Regression), from lagged returns, volatility measures, volume and macro indicators. Another well-established use is credit risk: predicting the probability that a borrower will default. SVMs handle non-linear relationships with modest data, but financial series are noisy and non-stationary, so the design must center on leakage-free features, time-ordered validation and comparison with naive baselines.
The last baseline converts the 20-day standard deviation into an expected absolute return (for normal returns, the mean absolute value is about 0.8 times the standard deviation).
Common Pitfalls
Look-ahead bias: features or scaling that use future data; random splits instead of time-ordered ones.
Claiming trading profits without transaction costs and realistic execution.
Overfitting through repeated testing on the same historical period.
Regime changes (crises) that break relationships learned in calm periods.
56
How can SVMs be used for sentiment analysis on social media data?
KnownRevise
SVM-based sentiment analysis on social media classifies posts as positive, negative or neutral using a linear SVM on text features. Social media text is short, informal and noisy, full of hashtags, mentions, emojis, slang, elongated words (“sooo good”) and sarcasm, so preprocessing and feature design matter: normalizing URLs and user names, keeping emojis and negations, and adding character n-grams that are robust to misspellings. A linear SVM on these features is fast and a strong baseline, though fine-tuned transformers usually do better when enough labelled data and compute are available.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Feature Design
Challenge
Handling
URLs, @mentions
Replace with tokens URL, USER
Hashtags
Keep the word (#happy → happy plus the hashtag token)
Emojis and emoticons
Keep them; they carry strong sentiment
Elongations (“sooo”)
Reduce repeated characters to two (“soo”)
Negation (“not good”)
Word bigrams capture it
Misspellings, slang
Character n-grams (2-5)
Class imbalance
class_weight="balanced"
Code Example
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import FeatureUnion, make_pipeline
from sklearn.svm import LinearSVC
def normalize(text):
text = text.lower()
text = re.sub(r"https?://\S+", " URL ", text)
text = re.sub(r"@\w+", " USER ", text)
text = re.sub(r"#(\w+)", r" \1 HASHTAG_\1 ", text)
return re.sub(r"(.)\1{2,}", r"\1\1", text) # "sooo" -> "soo"
posts = ["Loving the new update!!! 😍 #happy", "@support this app keeps crashing, sooo annoying 😡",
"not bad at all, works fine", "Worst customer service ever http://x.co/abc",
"Great battery life, really impressed 👍", "meh, it's ok I guess",
"Totally disappointed, not worth the money", "Best purchase this year!! #win"]
labels = ["pos", "neg", "pos", "neg", "pos", "neu", "neg", "pos"]
features = FeatureUnion([
("words", TfidfVectorizer(preprocessor=normalize, ngram_range=(1, 2), token_pattern=r"[^\s]+")),
("chars", TfidfVectorizer(preprocessor=normalize, analyzer="char_wb", ngram_range=(2, 5))),
])
model = make_pipeline(features, LinearSVC(C=0.5, class_weight="balanced"))
model.fit(posts, labels)
for post in ["sooo happy with this!!! 😍", "app crashed again, not good", "it is fine"]:
print(f"{post!r} -> {model.predict([post])[0]}")
The tiny dataset is only for illustration; a real model needs thousands of labelled posts (for example from public benchmarks such as SemEval Twitter sentiment tasks or your own annotated data) and evaluation with macro F1 on a held-out set.
Common Pitfalls
Removing emojis, punctuation and stop words such as “not”, which carry sentiment.
Training on one platform or time period and deploying on another, where slang differs.
Ignoring sarcasm and mixed sentiment; review errors to see where the model fails.
Accuracy as the only metric when neutral posts dominate.
57
How are SVMs applied to text categorization?
KnownRevise
Text categorization assigns documents to predefined categories, such as topics, departments, languages or spam vs ham, and linear SVMs on TF-IDF features have been a top-performing approach since Joachims’ work in 1998. Text produces very high-dimensional, sparse vectors in which categories are usually close to linearly separable, exactly where margin-based linear models shine. SVMs handle single-label, multi-label (one binary SVM per label) and hierarchical categorization, and they remain strong, fast baselines next to fine-tuned transformers.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Why SVMs Fit Text
Property of text
Why SVMs handle it well
Tens of thousands of features
Margin-based generalization doesn’t depend on dimension directly
Sparse vectors
Linear solvers (liblinear) exploit sparsity
Few irrelevant features overall
Weights spread over many informative terms
Linearly separable categories
Linear kernel is enough; fast
Task Variants
Variant
Approach
Single-label, many classes
LinearSVC (one-vs-rest)
Multi-label (document can have several tags)
OneVsRestClassifier(LinearSVC()) with a binary indicator matrix
Hierarchical (topic → subtopic)
Classifier per level or per parent node
Short texts (titles, tweets)
Add character n-grams
Code Example: Multi-Label Tagging
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import classification_report
from sklearn.multiclass import OneVsRestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.svm import LinearSVC
docs = [
"Central bank raises interest rates to curb inflation",
"New vaccine trial shows strong immune response",
"Tech company reports record quarterly profit from cloud services",
"Hospital adopts AI system to read chest X-rays",
"Stock markets fall as inflation data surprises investors",
"Startup raises funding for medical imaging software",
"Government announces budget with new healthcare spending",
"Chipmaker shares jump after earnings beat expectations",
]
tags = [["economy"], ["health"], ["tech", "business"], ["health", "tech"], ["economy", "business"],
["business", "health", "tech"], ["economy", "health"], ["business", "tech"]]
mlb = MultiLabelBinarizer()
Y = mlb.fit_transform(tags)
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
OneVsRestClassifier(LinearSVC(C=1.0)))
model.fit(docs, Y)
new_docs = ["Investors worry about inflation and interest rates",
"Software helps doctors analyse scans faster"]
for doc, row in zip(new_docs, model.predict(new_docs)):
print(doc, "->", list(mlb.classes_[row.astype(bool)]))
The training set here is tiny for illustration; real systems use thousands of labelled documents per category and evaluate with micro- and macro-averaged F1 on a held-out set (classification_report supports multi-label indicator matrices).
Common Pitfalls
Using an RBF kernel on sparse TF-IDF features: much slower, rarely more accurate.
Fitting the vectorizer on test documents.
Ignoring label imbalance in multi-label data; use class_weight="balanced" and per-label thresholds.
Reporting only micro-F1, which hides poor performance on rare categories.
58
How can SVMs be used for handwriting recognition?
KnownRevise
SVMs were among the most accurate handwriting recognizers before deep learning: on digit benchmarks such as MNIST, RBF and polynomial SVMs on raw or lightly processed pixels reached error rates around 1-2%, and SVMs on engineered features such as HOG did even better. The approach is to normalize each character image (size, position, slant), convert it to a feature vector (pixels, HOG or other descriptors), and train a multi-class SVM. Today CNNs are more accurate, but SVMs remain a strong baseline for small datasets and constrained devices.
Split a word or form into individual characters (or use whole-word models)
Normalization
Centre, resize to a fixed grid, correct slant, binarize or scale to [0, 1]
Features
Raw pixels, HOG (stroke directions), zoning, projection histograms
Classifier
Multi-class SVM (one-vs-one in SVC) with RBF kernel
Post-processing
Dictionary or language model for words
Code Example: Digits with Pixels vs HOG Features
import numpy as np
from skimage.feature import hog
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
digits = load_digits() # 1,797 images of 8x8 pixels
X_pix = digits.data / 16.0
X_hog = np.array([hog(img, orientations=8, pixels_per_cell=(4, 4), cells_per_block=(1, 1))
for img in digits.images])
for name, X in [("pixels", X_pix), ("pixels + HOG", np.hstack([X_pix, X_hog]))]:
X_train, X_test, y_train, y_test = train_test_split(X, digits.target, test_size=0.25,
stratify=digits.target, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf")),
{"svc__C": [1, 10, 100], "svc__gamma": ["scale", 0.001, 0.01]}, cv=5, n_jobs=-1)
grid.fit(X_train, y_train)
pred = grid.predict(X_test)
print(f"{name:13s} test accuracy {accuracy_score(y_test, pred):.4f} best {grid.best_params_}")
print(confusion_matrix(y_test, pred)) # which digits get confused
Improving Accuracy
Data augmentation with small shifts and rotations (the “virtual support vector” method added transformed copies of support vectors).
Deskewing images before feature extraction.
Larger images and richer descriptors for full characters and words.
Common Pitfalls
Inconsistent normalization between training and new handwriting samples.
Using raw pixels from large images with an RBF SVM on huge datasets: training becomes very slow; reduce dimensions or use CNNs.
Evaluating on writers who also appear in training; split by writer for realistic results.
59
How would you use SVMs for intrusion detection in cybersecurity?
KnownRevise
SVMs are used for intrusion detection in two ways: as supervised classifiers that learn to separate normal traffic from known attack types using labelled network-flow or log features, and as one-class SVMs that learn the boundary of normal behaviour and flag anything outside it, which can catch previously unseen attacks. A practical system extracts features from flows (durations, byte counts, flags, rates), scales and encodes them, trains the model on representative traffic, and tunes thresholds to keep false alarms manageable for analysts.
nu is an upper bound on the share of training points treated as outliers, so it roughly sets the false-alarm rate on normal traffic.
Common Pitfalls
Random train/test splits of time-ordered traffic; split by time to mimic deployment.
Training on outdated datasets whose attacks no longer look like current ones.
Ignoring alert volume: a 1% false-positive rate on millions of flows overwhelms analysts.
Adversaries adapting to the model; combine with rules and other detectors.
60
Propose an SVM application in healthcare for disease diagnosis.
KnownRevise
A good SVM application in healthcare is a diagnostic support model that flags patients at high risk of a disease from routinely collected data, for example detecting chronic kidney disease or diabetes from lab results and vital signs, or classifying tumours as malignant or benign from imaging-derived features. SVMs suit these problems because clinical datasets are often small to medium, have many correlated numeric features, and need good generalization from limited labels. The model supports clinicians rather than replacing them, so evaluation focuses on sensitivity, specificity, calibration and external validation.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Proposal: Malignancy Screening from Biopsy Image Features
Element
Design
Goal
Flag likely malignant fine-needle aspirate samples for priority pathologist review
Data
Numeric features of cell nuclei (radius, texture, concavity…) per sample
Model
Standardized features + RBF SVM, tuned with stratified CV
Outputs
Calibrated probability of malignancy, plus a threshold chosen with clinicians
Primary metric
Sensitivity (recall) at a fixed acceptable specificity; ROC AUC
Validation
Hold-out test set; later, external data from other hospitals
Governance
Privacy (de-identified data), bias checks across groups, documentation, regulatory review if used clinically
Code Example
import numpy as np
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import confusion_matrix, roc_auc_score
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
data = load_breast_cancer()
X, y = data.data, (data.target == 0).astype(int) # 1 = malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
grid = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel="rbf", class_weight="balanced")),
{"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.001, 0.01, 0.1]},
cv=cv, scoring="roc_auc")
grid.fit(X_train, y_train)
print("best:", grid.best_params_, "CV ROC AUC:", round(grid.best_score_, 3))
calibrated = CalibratedClassifierCV(grid.best_estimator_, method="sigmoid", cv=cv).fit(X_train, y_train)
proba = calibrated.predict_proba(X_test)[:, 1]
print("test ROC AUC:", round(roc_auc_score(y_test, proba), 3))
# Choose the threshold for high sensitivity using training-fold predictions, not the test set
train_proba = calibrated.predict_proba(X_train)[:, 1]
thresholds = np.linspace(0.05, 0.5, 10)
sens = [((train_proba >= t) & (y_train == 1)).sum() / (y_train == 1).sum() for t in thresholds]
threshold = max((t for t, s in zip(thresholds, sens) if s >= 0.98), default=thresholds[0])
tn, fp, fn, tp = confusion_matrix(y_test, (proba >= threshold).astype(int)).ravel()
print(f"threshold {threshold:.2f}: sensitivity {tp / (tp + fn):.3f}, specificity {tn / (tn + fp):.3f}")
For a stricter threshold choice, use out-of-fold predictions (cross_val_predict) instead of in-sample probabilities.
Risks to Address
Dataset shift: different scanners, labs or populations change feature distributions.
Label quality: diagnoses must be confirmed (e.g. by histopathology).
Patient-level splits: several samples from one patient must stay in the same split.
Explainability: provide feature attributions (e.g. permutation importance or SHAP) for clinician review.
Common Pitfalls
Optimizing accuracy when missed disease is far more costly than a false alarm.
Reporting only internal validation results.
61
What are recent advances in SVMs, and what do they mean for ML?
KnownRevise
SVM research has matured, and recent progress is mostly about making kernel methods scale, combining them with learned representations, and connecting them to deep learning theory, rather than new core formulations. Key developments include fast large-scale solvers and kernel approximations (random features, Nyström), GPU implementations, linear SVMs on deep embeddings, theoretical links between wide neural networks and kernels (the neural tangent kernel), and quantum kernel research. For practitioners, this means SVMs are now a strong option on top of pretrained features and on small-to-medium structured problems, while deep learning dominates raw perception tasks.
Code Example: Random Fourier Features vs Exact RBF
import time
from sklearn.datasets import make_classification
from sklearn.kernel_approximation import RBFSampler
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC
X, y = make_classification(n_samples=25000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
for name, model in [
("exact RBF SVC", make_pipeline(StandardScaler(), SVC(kernel="rbf", gamma=0.05))),
("RFF(2000) + LinearSVC", make_pipeline(StandardScaler(), RBFSampler(gamma=0.05, n_components=2000,
random_state=42), LinearSVC(C=1.0))),
]:
start = time.perf_counter()
model.fit(X_train, y_train)
print(f"{name:24s} {time.perf_counter() - start:6.1f}s accuracy {model.score(X_test, y_test):.3f}")
Random Fourier features approximate the RBF kernel with explicit cosine features, so a fast linear solver can be used; accuracy approaches the exact kernel as the number of components grows.
Implications for ML Practice
Use SVMs on embeddings from foundation models for small labelled datasets.
For tabular data, compare with gradient-boosted trees; SVMs win mainly on smaller, cleaner, high-dimensional problems.
Kernel theory remains valuable for understanding generalization in deep networks.
Interview Tips
Distinguish practical advances (scaling, embeddings, GPUs) from research directions (NTK, quantum kernels).
Be ready to explain when you would still pick an SVM today.
62
How can deep learning techniques be integrated with SVMs?
KnownRevise
Deep learning and SVMs are combined mainly by using a neural network as a feature extractor and an SVM as the classifier: a pretrained CNN or transformer turns raw images or text into embeddings, and a linear or RBF SVM is trained on those embeddings. This works especially well with small labelled datasets, where fine-tuning the whole network would overfit. Other integrations include replacing the softmax layer with a hinge-loss (SVM) output layer trained end to end, and deep kernel learning, where a network learns the input representation for a kernel.
Freeze a pretrained network, extract features, train an SVM
Small datasets, fast experiments
Hinge-loss output layer
Use nn.MultiMarginLoss (multi-class hinge) instead of cross-entropy
End-to-end training with margin objective
Deep kernel learning
Network learns features that feed an RBF kernel
Research; Gaussian-process style models
Ensemble
Combine CNN probabilities with SVM scores
Diversity in errors
Code Example: CNN Embeddings + SVM
import numpy as np
import torch
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
from torchvision import datasets, models
torch.manual_seed(42)
weights = models.ResNet18_Weights.DEFAULT
backbone = models.resnet18(weights=weights) # downloads pretrained weights
backbone.fc = torch.nn.Identity() # output 512-d embeddings
backbone.eval()
preprocess = weights.transforms()
data = datasets.CIFAR10(root="data", train=True, download=True) # downloads CIFAR-10
idx = np.random.default_rng(42).choice(len(data), 1000, replace=False) # small labelled set
with torch.no_grad():
feats, labels = [], []
for start in range(0, len(idx), 100):
batch = [data[i] for i in idx[start:start + 100]]
x = torch.stack([preprocess(img) for img, _ in batch])
feats.append(backbone(x).numpy())
labels.extend(label for _, label in batch)
X, y = np.vstack(feats), np.array(labels)
svm = make_pipeline(StandardScaler(), LinearSVC(C=0.01, max_iter=5000))
print("5-fold accuracy, ResNet18 features + linear SVM:", round(cross_val_score(svm, X, y, cv=5).mean(), 3))
Code Example: Hinge-Loss Output Layer
import torch
import torch.nn as nn
torch.manual_seed(42)
model = nn.Sequential(nn.Linear(512, 128), nn.ReLU(), nn.Linear(128, 10))
loss_fn = nn.MultiMarginLoss(margin=1.0) # Crammer-Singer-style multi-class hinge
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4) # weight decay ≈ the SVM's L2 term
x, y = torch.randn(64, 512), torch.randint(0, 10, (64,))
loss = loss_fn(model(x), y)
loss.backward()
optimizer.step()
print("hinge loss:", round(loss.item(), 4))
Common Pitfalls
Using the backbone’s training-time augmentations or forgetting eval(), which makes embeddings noisy.
Mismatched preprocessing between feature extraction and the network’s pretraining.
Fine-tuning a large network on a few hundred samples when frozen embeddings + SVM would generalize better.
63
How are SVMs used in reinforcement learning contexts?
KnownRevise
SVMs are not a standard component of modern reinforcement learning (RL), where neural networks dominate, but they have been used in several supporting roles: as function approximators for value functions (support vector regression for Q-values in batch methods such as fitted Q-iteration), as policy classifiers in classification-based policy iteration (predict the best action for a state), in imitation learning to copy an expert’s actions, and for safety or feasibility classification of states. Their appeal is good generalization from small amounts of data; their limitations are batch training and poor scaling to the large, changing datasets that RL generates.
An expert balances a simple 1D system: the action pushes left or right depending on position and velocity. The SVM learns the expert’s policy from logged states.
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
rng = np.random.default_rng(42)
def expert_action(pos, vel):
return int(pos + 0.5 * vel < 0) # 1 = push right, 0 = push left
def step(pos, vel, action):
vel = vel + (0.1 if action == 1 else -0.1) + rng.normal(0, 0.02)
return pos + vel * 0.1, vel
# Collect expert demonstrations
states, actions = [], []
for _ in range(200):
pos, vel = rng.uniform(-1, 1), rng.uniform(-1, 1)
for _ in range(30):
a = expert_action(pos, vel)
states.append([pos, vel]); actions.append(a)
pos, vel = step(pos, vel, a)
X, y = np.array(states), np.array(actions)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
policy = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=10)).fit(X_train, y_train)
print("action agreement with expert:", round(policy.score(X_test, y_test), 3))
# Roll out the learned policy and measure how close it keeps the system to 0
def rollout(policy_fn, episodes=50):
errors = []
for _ in range(episodes):
pos, vel = rng.uniform(-1, 1), rng.uniform(-1, 1)
for _ in range(50):
pos, vel = step(pos, vel, policy_fn(pos, vel))
errors.append(abs(pos))
return np.mean(errors)
print("mean |position| expert:", round(rollout(expert_action), 3))
print("mean |position| SVM: ", round(rollout(lambda p, v: int(policy.predict([[p, v]])[0])), 3))
Limitations
RL data is non-stationary and grows continuously; kernel SVMs must be retrained in batch.
Behaviour cloning suffers from compounding errors when the agent visits states the expert never did (DAgger addresses this).
Common Pitfalls
Judging an imitation policy only by action accuracy; evaluate it by rollouts.
Using SVMs where deep RL with experience replay is clearly the better tool.
64
What role have SVMs played in developing self-driving cars?
KnownRevise
SVMs played an important role in early autonomous-driving research, before deep learning became dominant in the mid-2010s. The classic example is pedestrian detection with HOG features and a linear SVM (Dalal and Triggs, 2005), which became a standard baseline; SVMs were also used for vehicle and traffic-sign classification, lane and road-surface classification, and obstacle classification from LiDAR features. Today, convolutional neural networks and transformers handle perception, while SVMs remain useful for small, well-defined subproblems, fast baselines and resource-constrained components.
Colour and texture features + SVM (e.g. off-road robots)
Segmentation networks
LiDAR object classification
Hand-crafted point-cloud features + SVM
Point-cloud networks (PointNet-style, BEV)
Driver monitoring
Eye/face features + SVM for drowsiness
CNNs on camera input
Code Example: HOG + Linear SVM Classifier
The example classifies small grayscale patches as “pedestrian-like” (vertical body structure) or background, using synthetic images so it runs anywhere.
import numpy as np
from skimage.feature import hog
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
rng = np.random.default_rng(42)
def synthetic_patch(person):
img = rng.normal(0.5, 0.15, (64, 32))
if person:
c = rng.integers(12, 20)
img[8:16, c - 3:c + 3] += 0.4 # head
img[16:44, c - 5:c + 5] += 0.3 # torso
img[44:62, c - 5:c - 1] += 0.3 # legs
img[44:62, c + 1:c + 5] += 0.3
return np.clip(img, 0, 1)
patches = [synthetic_patch(i % 2 == 0) for i in range(1200)]
labels = np.array([int(i % 2 == 0) for i in range(1200)])
features = np.array([hog(p, orientations=9, pixels_per_cell=(8, 8), cells_per_block=(2, 2)) for p in patches])
X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.25, stratify=labels,
random_state=42)
clf = make_pipeline(StandardScaler(), LinearSVC(C=0.01)).fit(X_train, y_train)
print("HOG feature length:", features.shape[1])
print(classification_report(y_test, clf.predict(X_test), target_names=["background", "pedestrian"]))
Real systems train on datasets such as INRIA Person or Caltech Pedestrians and slide the detector over image pyramids, followed by non-maximum suppression.
Why Deep Learning Replaced SVMs Here
CNNs learn features directly from pixels, outperforming hand-crafted HOG features.
Detection, segmentation and tracking can be learned jointly.
GPUs make large models fast enough for real time.
Interview Tips
Cite HOG + SVM as the classic pedestrian detector and explain sliding windows.
Note that safety-critical systems still value simple, verifiable components.
65
How can SVMs be used for domain adaptation in transfer learning?
KnownRevise
Domain adaptation trains a model on a labelled source domain and applies it to a related target domain whose data distribution differs, for example reviews of books (source) and electronics (target). With SVMs, the main approaches are: reweighting source samples so they resemble the target (importance weighting via sample_weight), aligning feature distributions before training (e.g. CORAL or a shared feature space), adapting a source SVM with a small number of labelled target samples (adaptive SVM, which regularizes the target model toward the source model), and transductive or self-training SVMs that use unlabelled target data.
Target labels are used here only for evaluation; in practice, keep a small labelled target set for validation, because which method helps depends on the kind of shift.
Common Pitfalls
Tuning hyperparameters on source validation data only, which ignores the shift.
Extreme importance weights that let a few points dominate; clip or normalize them.
Assuming adaptation always helps: if the label relationship changes between domains, reweighting cannot fix it.
66
What are the potential uses of SVMs in recommendation systems?
KnownRevise
SVMs can play several roles in recommendation systems, though they are rarely the core engine today: they can classify whether a user will engage with an item from user, item and context features (content-based or hybrid scoring), rank candidates with Ranking SVM (learning from pairwise preferences such as “clicked A over B”), and filter or re-rank results, for example detecting spam or inappropriate items. Their strengths are high-dimensional sparse features and good generalization from limited data; their weakness is scaling to the billions of user-item pairs that collaborative-filtering and deep models handle.
One linear SVM per user on item text features (TF-IDF)
Recommend news articles similar to those a user liked
Hybrid engagement model
Features from user, item, context, plus embeddings
Predict click on a recommended product
Ranking SVM
Train on feature differences x_A - x_B labelled +1 if A was preferred
Learning-to-rank for search results
Cold-start
Uses item and user attributes instead of interaction history
New items with no ratings
Filtering
Classify items as spam, low quality or unsafe
Clean candidate lists
Code Example: Pairwise Ranking SVM
import numpy as np
from sklearn.svm import LinearSVC
rng = np.random.default_rng(42)
n_items, n_features = 200, 6
items = rng.normal(size=(n_items, n_features))
true_pref = np.array([1.5, -1.0, 0.8, 0.0, 0.3, -0.5]) # hidden user taste
utility = items @ true_pref + rng.normal(0, 0.3, n_items)
# Build pairwise training data from observed preferences
pairs_X, pairs_y = [], []
for _ in range(2000):
a, b = rng.choice(n_items, 2, replace=False)
diff = items[a] - items[b]
label = 1 if utility[a] > utility[b] else -1
pairs_X.extend([diff, -diff]) # symmetric pairs keep classes balanced
pairs_y.extend([label, -label])
ranker = LinearSVC(C=1.0, fit_intercept=False).fit(np.array(pairs_X), np.array(pairs_y))
scores = items @ ranker.coef_[0] # score items with the learned weights
top10_learned = set(np.argsort(scores)[-10:])
top10_true = set(np.argsort(utility)[-10:])
print("learned weights:", ranker.coef_[0].round(2))
print("overlap of top-10 lists:", len(top10_learned & top10_true), "of 10")
print("rank correlation:", round(np.corrcoef(np.argsort(np.argsort(scores)),
np.argsort(np.argsort(utility)))[0, 1], 3))
The learned weights should point in the same direction as the hidden preference vector, so ranking by them recovers most of the best items.
Limitations
Pairwise training data grows quadratically; sample pairs carefully.
Collaborative signals (who liked what) are better captured by matrix factorization or two-tower neural models.
Kernel SVMs are too slow for large candidate sets; use linear SVMs.
Common Pitfalls
Evaluating with accuracy instead of ranking metrics (NDCG, MAP, recall@k).
Leaking future interactions into training features.
67
How might quantum machine learning research affect SVM algorithms?
KnownRevise
Quantum machine learning research touches SVMs mainly through quantum kernels: a quantum circuit maps each data point to a quantum state, and the kernel value is the overlap between two such states, estimated on quantum hardware or a simulator. The resulting kernel matrix is then used by an ordinary classical SVM solver. The hope is that some quantum feature maps produce kernels that are hard to compute classically yet useful for certain data. So far, no practical advantage over classical kernels on real-world datasets has been established; the field faces noise, sampling cost and the problem that many quantum kernels concentrate toward constant values as the number of qubits grows.
Read full answerHide full answer
Applies toQuantum kernel research as of 2026; Qiskit Machine Learning (optional), scikit-learn 1.5+ (Python 3.12)
How a Quantum Kernel SVM Works
Encode each feature vector into a quantum state with a parameterized circuit (the feature map).
EstimateK(x, z) = |<phi(x)|phi(z)>|^2 for every pair by running circuits and counting measurement outcomes.
Train a classical SVM on the precomputed kernel matrix.
Predict by estimating kernel values between new points and the support vectors.
Potential Impacts and Open Problems
Topic
Status
Expressive feature maps
Kernels classical computers may struggle to evaluate; benefit on practical data unproven
Provable advantage
Shown for specially constructed problems (e.g. based on discrete logarithms), not typical datasets
Exponential concentration
Kernel values become nearly identical for many qubits, so the SVM cannot distinguish points
Sampling cost
Each kernel entry needs many circuit runs; n^2 entries for training
Hardware noise
Current devices add errors to kernel estimates
Quantum-inspired classical algorithms
Some proposed speed-ups were matched by classical “dequantized” methods
Code Example: Classical Simulation of a Simple Quantum-Style Kernel
The kernel below is the fidelity of product states produced by single-qubit rotations, which can be computed classically in closed form; it illustrates the workflow without quantum hardware.
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC
def product_state_fidelity_kernel(A, B):
"""Each feature x_k is encoded as a qubit rotated by angle x_k; K = prod_k cos^2((a_k - b_k) / 2)."""
diff = A[:, None, :] - B[None, :, :]
return np.prod(np.cos(diff / 2) ** 2, axis=2)
X, y = make_moons(n_samples=300, noise=0.2, random_state=42)
X = MinMaxScaler(feature_range=(0, np.pi)).fit_transform(X) # angles in [0, pi]
K = product_state_fidelity_kernel(X, X)
print("kernel PSD:", np.linalg.eigvalsh(K).min() > -1e-9)
for name, clf in [("fidelity kernel", SVC(kernel=product_state_fidelity_kernel)),
("classical RBF", SVC(kernel="rbf"))]:
print(f"{name:16s} CV accuracy {cross_val_score(clf, X, y, cv=5).mean():.3f}")
Entangling feature maps (as in Qiskit’s ZZFeatureMap with FidelityQuantumKernel and QSVC) cannot in general be simulated this cheaply for many qubits; that is where any quantum advantage would have to come from.
Interview Tips
Explain that the SVM itself stays classical; only the kernel computation changes.
Mention benchmarking against tuned classical kernels and the concentration problem.
Avoid claiming practical speed-ups that have not been demonstrated.
68
How can SVMs be combined with other models in an ensemble?
KnownRevise
SVMs combine well with other models because they make different kinds of errors from tree ensembles, linear models and neural networks. The main options are voting (average the predictions or probabilities of several models), stacking (train a meta-model on the out-of-fold predictions of base models such as an SVM, a random forest and gradient boosting), and pipelines where another model produces features for the SVM (for example neural-network embeddings or tree-based leaf encodings). Ensembles help most when the members are individually good and their errors are not strongly correlated.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Combination Strategies
Strategy
How the SVM participates
Notes
Hard voting
Contributes a class label
Simple; no probabilities needed
Soft voting
Contributes a probability
Needs calibrated SVM outputs
Stacking
Its decision_function becomes a meta-feature
Meta-model learns how much to trust each member
Feature pipeline
SVM trained on learned features (CNN embeddings, PCA)
Often strong for small labelled datasets
Cascade
Cheap model filters easy cases; SVM handles hard ones
Reduces latency
Code Example: Voting and Stacking
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import (HistGradientBoostingClassifier, RandomForestClassifier, StackingClassifier,
VotingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
svm = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, probability=True, random_state=42))
rf = RandomForestClassifier(n_estimators=300, random_state=42)
gb = HistGradientBoostingClassifier(random_state=42)
lr = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
voting = VotingClassifier([("svm", svm), ("rf", rf), ("gb", gb)], voting="soft")
stacking = StackingClassifier(
estimators=[("svm", svm), ("rf", rf), ("gb", gb)],
final_estimator=LogisticRegression(),
stack_method="auto", # uses predict_proba where available
cv=5, # meta-features come from out-of-fold predictions
)
for name, model in [("SVM", svm), ("RandomForest", rf), ("GradientBoosting", gb), ("LogReg", lr),
("Soft voting", voting), ("Stacking", stacking)]:
scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
print(f"{name:17s} ROC AUC {scores.mean():.4f} ± {scores.std():.4f}")
On an easy dataset the differences may be within noise; ensembles pay off more on harder problems with diverse members.
Practical Tips
Check that members disagree on some examples (low error correlation); identical models add nothing.
Use out-of-fold predictions for stacking (scikit-learn does this) to avoid leakage.
Weigh the gain against the extra latency and maintenance of several models.
Common Pitfalls
Soft voting with uncalibrated SVM scores.
Stacking evaluated on the same data used to create meta-features.
Adding the SVM without scaling inside its own pipeline.
69
How do bagging and boosting work with SVM classifiers?
KnownRevise
Bagging trains several SVMs on different bootstrap samples (and optionally feature subsets) and combines them by voting or averaging, which mainly reduces variance and allows training on large data in parallel pieces. Boosting trains SVMs sequentially, each focusing on the examples the previous ones got wrong (through sample weights in AdaBoost), and combines them with weights, which mainly reduces bias. Because a tuned SVM is already a strong, low-variance learner, the gains from either are usually modest; they help most with large or noisy data, or with deliberately weakened SVMs.
Read full answerHide full answer
Applies toscikit-learn 1.6+ (Python 3.12)
Comparison
Aspect
Bagging SVMs
Boosting SVMs (AdaBoost)
Training
Parallel, independent
Sequential
Data per model
Bootstrap or subsample
All data with updated weights
Main effect
Lower variance, faster on large n
Lower bias
Base SVM
Normal or subsampled
Should be weak (small C, linear, or few features)
Risk
Little
Overfitting noisy labels; strong SVMs leave little to correct
Train an SVM with those weights (fit(X, y, sample_weight=w)), which scales each point’s penalty C.
Increase the weights of misclassified points, decrease the others.
Give each SVM a vote proportional to its accuracy and repeat.
Code Example
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, BaggingClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=3000, n_features=20, n_informative=8, flip_y=0.05, random_state=42)
X = StandardScaler().fit_transform(X) # for brevity; fit inside each fold in real projects
models = {
"single SVC rbf": SVC(kernel="rbf", C=1.0),
"bagging (15 SVCs, 30% samples)": BaggingClassifier(SVC(kernel="rbf", C=1.0), n_estimators=15,
max_samples=0.3, n_jobs=-1, random_state=42),
"weak linear SVC": SVC(kernel="linear", C=0.01),
"AdaBoost (30 weak linear SVCs)": AdaBoostClassifier(SVC(kernel="linear", C=0.01), n_estimators=30,
random_state=42),
}
for name, model in models.items():
print(f"{name:32s} CV accuracy {cross_val_score(model, X, y, cv=5, n_jobs=-1).mean():.3f}")
Since scikit-learn 1.6, AdaBoostClassifier uses the SAMME algorithm, which needs only class predictions, so the SVMs do not need probability=True (older versions defaulted to SAMME.R, which requires probabilities).
When to Use
Bagging: large datasets (train subsets in parallel), noisy data, or when you need an ensemble’s variance reduction.
Boosting: rarely the first choice for SVMs; gradient-boosted trees are usually better boosted learners.
Common Pitfalls
Boosting a strong RBF SVM: after the first round there is little left to correct, and later rounds overfit noise.
Expecting large gains over a single well-tuned SVM; always compare against it.
70
Describe a scenario where an SVM is a weak learner in an ensemble.
KnownRevise
An SVM can serve as a weak learner when it is deliberately constrained, for example trained on a small random subset of samples or features, heavily regularized, or limited to a linear kernel, so that each model is only modestly accurate but different from the others. A realistic scenario is a large text or sensor dataset where one exact kernel SVM is too slow: training many small SVMs on random subsets (bagging or random subspaces) and combining their votes gives a faster, often more robust model. SVMs are less common in boosting because they are strong, stable learners, which limits what boosting can add.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Scenario: Fraud Detection on 200,000 Transactions
One RBF SVM on all rows would take a very long time to train.
Instead, train 20 RBF SVMs, each on 5% of the rows and 70% of the features, in parallel.
Each model is individually weaker, but their errors differ; averaging their scores reduces variance.
Training cost per model grows roughly quadratically with its sample size, so 20 models on 5% each cost far less than one model on 100%.
Ways to Make SVMs “Weak”
Constraint
Effect
Small random sample per model
Fast; diverse boundaries
Random feature subspace
Different views of the data
Small C
Simple, high-bias boundaries
Linear kernel on non-linear data
Limited capacity
Code Example: Bagged SVMs with Random Subspaces
import time
from sklearn.datasets import make_classification
from sklearn.ensemble import BaggingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_classification(n_samples=40000, n_features=30, n_informative=12, weights=[0.9, 0.1],
flip_y=0.02, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
weak_svm = make_pipeline(StandardScaler(), SVC(kernel="rbf", C=1.0, class_weight="balanced"))
ensemble = BaggingClassifier(
estimator=weak_svm,
n_estimators=20,
max_samples=0.05, # each SVM sees 5% of the rows
max_features=0.7, # and 70% of the features
n_jobs=-1,
random_state=42,
)
start = time.perf_counter()
ensemble.fit(X_train, y_train)
print(f"ensemble trained in {time.perf_counter() - start:.1f}s")
single_small = weak_svm.fit(X_train[:1500], y_train[:1500]) # one weak learner alone
print("single weak SVM ROC AUC:", round(roc_auc_score(y_test, single_small.decision_function(X_test)), 3))
print("bagged SVMs ROC AUC: ", round(roc_auc_score(y_test, ensemble.decision_function(X_test)), 3))
BaggingClassifier.decision_function averages the members’ scores, which works for ranking metrics without needing probabilities.
When It Helps (and When It Doesn’t)
Helps: large datasets, noisy labels, high variance between subsets, parallel hardware.
Doesn’t help much: small, clean datasets where a single tuned SVM is already stable; trees benefit more from bagging because they are less stable.
Common Pitfalls
Fitting the scaler outside the bagged estimator; keep it inside so each member scales its own sample.
Using probability=True in every member, which multiplies training time.