Principal Component Analysis (PCA) is an unsupervised, linear technique that finds new axes, called principal components, along which the data varies the most: the first component is the direction of maximum variance, the second is the direction of maximum remaining variance at a right angle to the first, and so on, so that projecting the data onto the first few components keeps most of its information in far fewer dimensions. Mathematically, the components are the eigenvectors of the data’s covariance matrix (equivalently, the right singular vectors of the centred data matrix), and each eigenvalue is the variance captured by its component. The steps are: centre the data (and usually standardize it), compute the covariance matrix or the SVD, sort components by eigenvalue, keep the top k, and project. PCA is used for dimensionality reduction, visualization, noise removal, compression, and decorrelating features before modelling.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Steps
Step
What happens
1. Standardize
Subtract the mean of each feature; divide by its standard deviation if units differ
2. Covariance or SVD
C = XᵀX / (n − 1), or the SVD X = UΣVᵀ
3. Eigen-decomposition
Eigenvectors = components; eigenvalues = variance along them
4. Sort and choose k
Keep the components with the largest eigenvalues
5. Project
Z = X·W_k (n × k scores)
Code Example: From Scratch vs scikit-learn on Iris
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X = StandardScaler().fit_transform(load_iris().data) # 150 x 4
Xc = X - X.mean(axis=0)
cov = Xc.T @ Xc / (len(Xc) - 1)
eigvals, eigvecs = np.linalg.eigh(cov) # ascending order for symmetric matrices
order = np.argsort(eigvals)[::-1]
eigvals, eigvecs = eigvals[order], eigvecs[:, order]
print("explained variance ratio:", (eigvals / eigvals.sum()).round(4)) # [0.7296 0.2285 0.0367 0.0052]
Z_mine = Xc @ eigvecs[:, :2]
pca = PCA(n_components=2).fit(X)
Z_sk = pca.transform(X)
signs = np.sign(np.sum(Z_mine * Z_sk, axis=0)) # components are defined up to sign
print("same projection up to sign:", np.allclose(Z_mine * signs, Z_sk)) # True
print("first two components keep", round(float(pca.explained_variance_ratio_.sum()), 4)) # 0.9581
Two components keep about 96% of the variance of the four standardized Iris features.
Properties
Components are orthogonal, and the projected scores are uncorrelated.
Components are ordered by the variance they explain.
Signs are arbitrary: a component and its negative are equally valid.
PCA is linear and sensitive to feature scale and outliers.
Common Pitfalls
Skipping standardization when features have different units.
Fitting PCA on all data before the train/test split (leakage); put it in a Pipeline.
Assuming the highest-variance directions are the most predictive for a target.
Interview Tips
Describe the steps and the eigenvector/eigenvalue meaning.
Mention scaling, SVD and the explained variance ratio.
2
How is PCA used for dimensionality reduction?
KnownRevise
PCA reduces dimensionality by replacing the original d features with the first k principal components, the k orthogonal directions that capture the most variance, and representing each sample by its k projections (scores); because later components usually carry little variance, often mostly noise, dropping them loses little information while making the data smaller, less redundant and easier to model or plot. k is chosen from the cumulative explained variance ratio (for example the smallest k reaching 95%), a scree plot’s elbow, or downstream validation performance. In practice you fit PCA on the training data only, inside a Pipeline with scaling, then transform both training and test data; inverse_transform maps scores back to the original space and its reconstruction error shows how much was lost.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: Digits from 64 to Fewer Dimensions
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True) # 1797 images, 64 pixel features
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
scaler = StandardScaler().fit(X_train)
pca = PCA(n_components=0.95).fit(scaler.transform(X_train)) # keep 95% of the variance
print(f"components kept: {pca.n_components_} of 64") # about 40
Z_test = pca.transform(scaler.transform(X_test))
X_back = pca.inverse_transform(Z_test)
mse = np.mean((scaler.transform(X_test) - X_back) ** 2)
print(f"test reconstruction MSE (standardized units): {mse:.3f}")
for k in [10, 20, 40, None]:
steps = [StandardScaler()] + ([PCA(n_components=k)] if k else []) + [LogisticRegression(max_iter=2000)]
score = cross_val_score(make_pipeline(*steps), X_train, y_train, cv=5).mean()
print(f"k={k or 'all 64':>6}: CV accuracy {score:.3f}")
Accuracy typically stays close to the full-feature model from around 20 to 40 components while the model trains on far fewer inputs.
Benefits and Costs
Benefit
Cost
Fewer features, faster training
Components are harder to interpret
Removes correlated redundancy
Linear only
Filters some noise
Variance is not the same as predictive value
2-D/3-D plots
Needs scaling and is sensitive to outliers
Common Pitfalls
Fitting the scaler and PCA on the full dataset before splitting.
Choosing k by variance alone when downstream accuracy is what matters.
Applying PCA to sparse, high-dimensional text without care (centring makes it dense; use TruncatedSVD).
Interview Tips
Explain projection onto the top-k components and choosing k.
Mention pipelines, inverse_transform and TruncatedSVD for sparse data.
3
What are eigenvalues and eigenvectors in PCA?
KnownRevise
An eigenvector of a square matrix A is a non-zero vector v whose direction doesn’t change when A is applied, only its length: Av = λv, where the scalar λ is the eigenvalue; in PCA the matrix is the covariance matrix of the centred data, its eigenvectors are the principal components (directions in feature space), and each eigenvalue equals the variance of the data along its eigenvector. Because the covariance matrix is symmetric and positive semi-definite, its eigenvalues are real and non-negative and its eigenvectors can be chosen orthonormal, so they form a new coordinate system. Sorting by eigenvalue ranks the directions by how much variance they explain, the explained variance ratio of component i is λᵢ / Σλ, and the sum of all eigenvalues equals the total variance (the trace of the covariance matrix).
Read full answerHide full answer
Applies toNumPy 2.x
Worked Example (2 × 2)
Take the covariance matrix C = [[3, 1], [1, 3]].
Solve det(C − λI) = (3 − λ)² − 1 = 0, so λ = 4 or λ = 2.
For λ = 4: (C − 4I)v = 0 gives v = [1, 1]/√2.
For λ = 2: v = [1, −1]/√2.
The first component, along [1, 1], explains 4 / (4 + 2) = 66.7% of the variance, and the total variance 6 equals the trace 3 + 3.
Code Example
import numpy as np
C = np.array([[3.0, 1.0], [1.0, 3.0]])
vals, vecs = np.linalg.eigh(C) # symmetric: eigh, ascending eigenvalues
print(vals) # [2. 4.]
v = vecs[:, 1]
print(np.allclose(C @ v, 4 * v), np.abs(v).round(4)) # True [0.7071 0.7071]
rng = np.random.default_rng(42)
X = rng.multivariate_normal([0, 0], C, size=50_000)
Xc = X - X.mean(axis=0)
sample_vals, sample_vecs = np.linalg.eigh(np.cov(Xc, rowvar=False))
print("sample eigenvalues:", sample_vals.round(2)) # close to [2, 4]
proj = Xc @ sample_vecs[:, ::-1] # project onto components, largest first
print("variance along each component:", proj.var(axis=0, ddof=1).round(2)) # matches the eigenvalues
print("correlation of the scores:", round(float(np.corrcoef(proj.T)[0, 1]), 4)) # about 0: uncorrelated
print("trace equals total variance:", np.isclose(np.trace(np.cov(Xc, rowvar=False)), sample_vals.sum())) # True
Summary
Term
Meaning in PCA
Eigenvector
Principal component direction (a loading vector)
Eigenvalue
Variance along that direction
λᵢ / Σλ
Explained variance ratio
Σλ
Total variance = trace of the covariance matrix
Orthogonality
Components are perpendicular; scores are uncorrelated
Common Pitfalls
Using np.linalg.eig on a symmetric matrix and getting tiny complex parts or unsorted output; eigh is designed for it.
Forgetting that eigenvectors are defined up to sign.
Computing eigenvectors on uncentred data.
Interview Tips
Write Av = λv and work a 2 × 2 example.
Connect eigenvalues to variance and the trace.
4
What role does the covariance matrix play in PCA?
KnownRevise
The covariance matrix summarizes how every pair of features varies together, C = XcᵀXc / (n − 1) for centred data Xc: its diagonal holds each feature’s variance and its off-diagonal entries the covariances, and PCA is exactly the eigen-decomposition of this matrix. The variance of the data projected onto any unit direction w is wᵀCw, so finding the direction of maximum variance means maximizing wᵀCw subject to ‖w‖ = 1, whose solution is the top eigenvector of C; the remaining eigenvectors give the next orthogonal directions, and rotating the data into this eigenbasis produces uncorrelated components (the covariance matrix of the scores is diagonal, holding the eigenvalues). Because C depends on units, standardized data is usually used, which makes C the correlation matrix.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
What the Matrix Tells You
Part
Meaning
Cᵢᵢ
Variance of feature i
Cᵢⱼ
Covariance of features i and j (positive: rise together)
Large off-diagonal values
Redundant, correlated features that PCA can combine
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X = StandardScaler().fit_transform(load_iris().data)
Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / (len(X) - 1)
print(np.allclose(C, np.cov(X, rowvar=False))) # True
print("correlation of petal length and petal width:", round(float(C[2, 3]), 3)) # about 0.97: very redundant
vals, vecs = np.linalg.eigh(C)
w = vecs[:, -1] # top eigenvector
rng = np.random.default_rng(42)
random_dirs = rng.normal(size=(1000, 4))
random_dirs /= np.linalg.norm(random_dirs, axis=1, keepdims=True)
random_var = np.einsum("ij,jk,ik->i", random_dirs, C, random_dirs)
print(f"variance along top eigenvector {w @ C @ w:.3f}, best random direction {random_var.max():.3f}")
# No random direction beats the top eigenvector (its eigenvalue, about 2.94)
scores = PCA().fit_transform(X)
print(np.cov(scores, rowvar=False).round(3)) # diagonal: components are uncorrelated
Standardized Iris has total variance 4 (one per feature, slightly more with ddof = 1), and the top eigenvalue of about 2.94 is about 73% of it.
Covariance vs Correlation
Use
When
Covariance matrix (raw data)
Features share units and their scale is meaningful
Correlation matrix (standardized data)
Features have different units or ranges
Common Pitfalls
Using raw covariance with mixed units, so the feature with the largest numbers dominates.
Forgetting to centre before computing XᵀX.
Building a d × d covariance matrix when d is huge; use SVD or randomized methods instead.
Interview Tips
Show that projected variance is wᵀCw and why the top eigenvector maximizes it.
Mention diagonal covariance of the scores and covariance vs correlation.
5
What is the variance explained by a principal component?
KnownRevise
The variance explained by a principal component is the variance of the data along that component’s direction, which equals its eigenvalue λᵢ of the covariance matrix; the explained variance ratio λᵢ / Σⱼ λⱼ is the share of the total variance that component captures, and the cumulative ratio over the first k components tells you how much of the data’s spread a k-dimensional projection keeps. In scikit-learn these are explained_variance_ and explained_variance_ratio_; from an SVD of the centred data they are sᵢ² / (n − 1) and sᵢ² / Σ s². They are used to choose k (a threshold such as 90–95%, or the elbow of a scree plot) and to judge how much structure is low-dimensional, keeping in mind that “variance explained” describes the inputs, not how well a target can be predicted.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Example (Standardized Iris)
Component
Explained variance ratio
Cumulative
PC1
0.7296
0.7296
PC2
0.2285
0.9581
PC3
0.0367
0.9948
PC4
0.0052
1.0000
Code Example
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X = StandardScaler().fit_transform(load_iris().data)
pca = PCA().fit(X)
print("explained variance:", pca.explained_variance_.round(4))
print("ratio:", pca.explained_variance_ratio_.round(4)) # [0.7296 0.2285 0.0367 0.0052]
print("cumulative:", np.cumsum(pca.explained_variance_ratio_).round(4))
Xc = X - X.mean(axis=0)
s = np.linalg.svd(Xc, compute_uv=False)
print(np.allclose(s**2 / (len(X) - 1), pca.explained_variance_)) # True
print(np.allclose(s**2 / np.sum(s**2), pca.explained_variance_ratio_)) # True
scores = pca.transform(X)
print(np.allclose(scores.var(axis=0, ddof=1), pca.explained_variance_)) # True: variance of each score
k = int(np.searchsorted(np.cumsum(pca.explained_variance_ratio_), 0.95) + 1)
print("components needed for 95%:", k) # 2
Choosing k from Explained Variance
Method
Rule
Threshold
Smallest k with cumulative ratio ≥ 0.90 or 0.95
Scree plot
Keep components before the curve flattens
Kaiser rule (standardized data)
Eigenvalue > 1
Downstream
k with the best validation score
Common Pitfalls
Comparing explained variance across datasets with different scaling.
Treating a low-variance component as useless for prediction; it can still separate classes.
Forgetting that unscaled data makes the ratios reflect units (raw Iris PC1 is about 92%).
Interview Tips
Equate explained variance with eigenvalues and the ratio with λᵢ / Σλ.
Show how the cumulative curve is used to pick k.
6
How does feature scaling affect PCA?
KnownRevise
PCA looks for directions of maximum variance, and variance depends on units, so without scaling the features with the largest numeric ranges dominate the first components regardless of how informative they are: a salary in rupees (variance in the billions) would swamp age in years, and the “first component” would simply be salary. Standardizing each feature to mean 0 and standard deviation 1 (equivalent to running PCA on the correlation matrix) gives every feature an equal starting weight so components reflect shared structure rather than units; scaling is the right default when features have different units, while leaving data unscaled can be correct when all features share a meaningful unit (pixel intensities, spectra) and their relative variances matter. Scaling also changes the explained variance ratios and the loadings, so always say which version you used, and fit the scaler on training data only, inside a Pipeline.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n = 1000
age = rng.normal(40, 10, n) # years
income = 20_000 + 900 * age + rng.normal(0, 15_000, n) # rupees, depends on age
score = rng.normal(600, 50, n) # credit score, unrelated
X = np.column_stack([age, income, score])
names = ["age", "income", "score"]
raw = PCA().fit(X)
print("unscaled ratios:", raw.explained_variance_ratio_.round(4))
print("unscaled PC1 loadings:", dict(zip(names, raw.components_[0].round(4))))
# PC1 is almost purely income, because its variance is millions of times larger
Xs = StandardScaler().fit_transform(X)
scaled = PCA().fit(Xs)
print("scaled ratios:", scaled.explained_variance_ratio_.round(3))
print("scaled PC1 loadings:", dict(zip(names, scaled.components_[0].round(3))))
# PC1 now combines age and income (they are correlated) and score gets its own component
When to Scale
Situation
Scale?
Mixed units (age, income, counts)
Yes, standardize
Same unit, variance meaningful (pixels, spectra)
Often no, just centre
Heavy outliers
Consider RobustScaler or robust PCA
Sparse text features
Use TruncatedSVD without centring
Common Pitfalls
Running PCA on raw data with mixed units and interpreting the first component as “most important”.
Fitting the scaler on the full dataset before splitting.
Comparing loadings between scaled and unscaled analyses.
Interview Tips
Give a concrete example of one feature dominating.
Mention correlation-matrix PCA and when not to scale.
7
What is the difference between PCA and factor analysis?
KnownRevise
PCA is a data-summarizing technique that finds orthogonal linear combinations of the observed variables explaining the most total variance, while factor analysis (FA) is a statistical model that assumes the observed variables are generated by a smaller number of hidden factors plus feature-specific noise, x = μ + Lf + ε, and tries to explain the shared variance (covariance) between variables. PCA’s components are functions of the data; FA’s factors are latent causes behind the data. FA estimates a separate noise variance (uniqueness) for each variable, so it doesn’t spend components on variance that only one variable has, and its loadings can be rotated (varimax, promax) to make factors easier to interpret, which is why FA is used in psychology and survey analysis to discover constructs such as “satisfaction”. PCA is preferred for compression, preprocessing and visualization; when uniquenesses are small and equal, the two give similar results (probabilistic PCA is FA with equal noise variance for every feature).
Read full answerHide full answer
Applies toscikit-learn 1.5+
Comparison
PCA
Factor analysis
Goal
Summarize total variance
Model shared variance with latent factors
Model
No generative model (except probabilistic PCA)
x = μ + Lf + ε, ε with per-feature variance
Noise
Treated as part of the variance
Explicit uniqueness per variable
Solution
Eigenvectors, unique up to sign
Loadings unique up to rotation
Rotation
Not used
Varimax/promax for interpretability
Typical use
Compression, preprocessing, plots
Surveys, psychometrics, latent constructs
Code Example
import numpy as np
from sklearn.decomposition import PCA, FactorAnalysis
rng = np.random.default_rng(42)
n = 2000
factors = rng.normal(size=(n, 2)) # two latent factors
L = np.array([[0.9, 0.0], [0.8, 0.0], [0.7, 0.0], # items 0-2 load on factor 1
[0.0, 0.9], [0.0, 0.8], [0.0, 0.7]]) # items 3-5 load on factor 2
noise_sd = np.array([0.3, 0.5, 0.7, 0.3, 0.5, 2.0]) # item 5 is very noisy
X = factors @ L.T + rng.normal(size=(n, 6)) * noise_sd
fa = FactorAnalysis(n_components=2, rotation="varimax", random_state=42).fit(X)
print("FA loadings:\n", fa.components_.T.round(2))
print("FA noise variances:", fa.noise_variance_.round(2)) # close to noise_sd**2, item 5 about 4.0
pca = PCA(n_components=2).fit(X)
print("PCA loadings:\n", pca.components_.T.round(2))
# PCA gives the noisy item 5 a large loading because its total variance is high;
# FA attributes that variance to the item's own noise
Common Pitfalls
Calling PCA “factor analysis” in reports; they answer different questions.
Interpreting rotated factors as proven causes.
Running FA with too few observations per variable.
Interview Tips
Contrast total variance vs shared variance and the generative model.
Mention uniquenesses, rotation and probabilistic PCA.
8
Why is PCA considered an unsupervised technique?
KnownRevise
PCA is unsupervised because it uses only the input features X and never a target label y: it finds directions of maximum variance in X, and those directions are the same whether the data is labelled spam/not spam, cat/dog, or not labelled at all. That is useful for exploring data, compressing it and removing redundancy, but it also means PCA has no guarantee of keeping the information that matters for a particular prediction task: a low-variance direction can be exactly the one that separates classes, and PCA might drop it. When labels are available and the goal is prediction, supervised alternatives such as linear discriminant analysis (maximizes class separation), partial least squares (maximizes covariance with the target) or simply selecting k by validation performance inside a Pipeline make sure the reduction serves the task.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: Variance Is Not the Same as Class Separation
import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(42)
n = 1000
y = rng.integers(0, 2, n)
big_noise = rng.normal(0, 10, n) # high variance, unrelated to y
small_signal = (2 * y - 1) * 0.5 + rng.normal(0, 0.2, n) # low variance, separates the classes
X = np.column_stack([big_noise, small_signal])
pca = PCA(n_components=1).fit(X) # never sees y
print("PCA direction:", pca.components_[0].round(3)) # about [1, 0]: the noise axis
acc_pca = cross_val_score(make_pipeline(PCA(n_components=1), LogisticRegression()), X, y, cv=5).mean()
acc_lda = cross_val_score(make_pipeline(LinearDiscriminantAnalysis(n_components=1), LogisticRegression()), X, y, cv=5).mean()
print(f"accuracy with 1 PCA component {acc_pca:.3f}, with 1 LDA component {acc_lda:.3f}")
# about 0.5 (chance) vs nearly 1.0
Unsupervised vs Supervised Reduction
Method
Uses y?
Objective
PCA
No
Maximize variance
t-SNE / UMAP
No
Preserve neighbourhoods
LDA
Yes
Maximize between-class vs within-class variance
PLS
Yes
Maximize covariance with the target
Feature selection by model
Yes
Predictive importance
Common Pitfalls
Assuming the top components are the most predictive.
Choosing k without checking downstream performance.
Thinking “unsupervised” means PCA can be fitted on test data; it still leaks information.
Interview Tips
Say PCA ignores labels and give the low-variance-signal counterexample.
Mention LDA and PLS as supervised alternatives.
9
Derive PCA as minimizing reconstruction error
KnownRevise
PCA can be derived as finding the k-dimensional linear subspace that minimizes the average squared distance between each centred data point and its projection onto that subspace; because the squared length of each point splits into the part inside the subspace plus the part outside it (Pythagoras), minimizing the reconstruction error is the same as maximizing the variance of the projections, and the solution is the subspace spanned by the top k eigenvectors of the covariance matrix. The minimum error equals the sum of the discarded eigenvalues, (n − 1)·Σᵢ₌ₖ₊₁ λᵢ in total, which is the Eckart-Young theorem: the truncated SVD is the best rank-k approximation of the centred data in the Frobenius norm.
Read full answerHide full answer
Applies tolinear algebra; check with NumPy 2.x
Derivation
Setup. Centred data x₁, …, xₙ ∈ ℝᵈ. Choose an orthonormal basis W ∈ ℝᵈˣᵏ (WᵀW = I). The projection of x is WWᵀx, and the reconstruction error is J(W) = Σᵢ ‖xᵢ − WWᵀxᵢ‖².
Expand. Since WWᵀ is a projection (symmetric and idempotent), ‖x − WWᵀx‖² = xᵀx − xᵀWWᵀx. So J(W) = Σᵢ ‖xᵢ‖² − Σᵢ ‖Wᵀxᵢ‖².
Convert to variance. The first term doesn’t depend on W, so minimizing J means maximizing Σᵢ ‖Wᵀxᵢ‖² = tr(WᵀXᵀXW) = (n − 1)·tr(WᵀCW), where C is the covariance matrix.
One direction (k = 1). Maximize wᵀCw subject to wᵀw = 1. The Lagrangian wᵀCw − λ(wᵀw − 1) has gradient 2Cw − 2λw = 0, so Cw = λw: w is an eigenvector, and the objective equals λ, so the best choice is the top eigenvector.
k directions. Maximizing tr(WᵀCW) with orthonormal columns gives the top k eigenvectors (by the Ky Fan theorem, or by induction adding one orthogonal direction at a time), and the maximum is λ₁ + … + λₖ.
import numpy as np
rng = np.random.default_rng(42)
A = rng.normal(size=(5, 5))
X = rng.normal(size=(500, 5)) @ A # correlated data
Xc = X - X.mean(axis=0)
n, k = len(Xc), 2
C = Xc.T @ Xc / (n - 1)
vals, vecs = np.linalg.eigh(C)
vals, vecs = vals[::-1], vecs[:, ::-1]
def recon_error(W):
return float(np.sum((Xc - Xc @ W @ W.T) ** 2))
W_pca = vecs[:, :k]
print(f"PCA error {recon_error(W_pca):.2f}, predicted (n-1)*sum(discarded) {(n - 1) * vals[k:].sum():.2f}") # equal
best_random = min(recon_error(np.linalg.qr(rng.normal(size=(5, k)))[0]) for _ in range(2000))
print(f"best of 2000 random 2-D subspaces: {best_random:.2f}") # never below the PCA error
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
rank_k = U[:, :k] * s[:k] @ Vt[:k]
print(np.isclose(np.sum((Xc - rank_k) ** 2), recon_error(W_pca))) # True: Eckart-Young
Common Pitfalls
Forgetting to centre, which makes the “best subspace” pass through the origin instead of the mean.
Mixing up the totals: error sums over points, so the factor n − 1 (or n) matters when comparing to eigenvalues.
Treating minimum reconstruction error as optimal for a downstream prediction task.
Interview Tips
Use the Pythagoras split to show the equivalence with maximum variance.
Solve the one-direction case with a Lagrange multiplier and state the discarded-eigenvalue error.
10
What is SVD and how does it relate to PCA?
KnownRevise
Singular Value Decomposition factors any n × d matrix as X = UΣVᵀ, where U (n × r) and V (d × r) have orthonormal columns and Σ is diagonal with non-negative singular values s₁ ≥ s₂ ≥ … ; for centred data, the columns of V are exactly the principal component directions, the variance along component i is sᵢ² / (n − 1), and the scores (projections) are XV = UΣ. That follows from XᵀX = VΣ²Vᵀ, so the eigenvectors of the covariance matrix XᵀX/(n − 1) are V and its eigenvalues are sᵢ²/(n − 1). Computing PCA through the SVD of X is what scikit-learn does because it never forms XᵀX, which squares the condition number and loses precision, and truncated or randomized SVD makes PCA practical for large or sparse data (TruncatedSVD skips centring, which is the basis of latent semantic analysis for text).
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Correspondence
SVD of centred X
PCA
Columns of V (rows of Vᵀ)
Principal components (components_)
sᵢ² / (n − 1)
Explained variance (explained_variance_)
sᵢ² / Σ s²
Explained variance ratio
UΣ
Scores (transform output)
U_kΣ_kV_kᵀ
Best rank-k reconstruction
Code Example
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
X = load_digits().data # 1797 x 64
Xc = X - X.mean(axis=0)
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
pca = PCA(n_components=10).fit(X)
print(np.allclose(s[:10] ** 2 / (len(X) - 1), pca.explained_variance_)) # True
signs = np.sign(np.sum(Vt[:10] * pca.components_, axis=1))
print(np.allclose(Vt[:10] * signs[:, None], pca.components_)) # True (up to sign)
print(np.allclose((U[:, :10] * s[:10]) * signs, pca.transform(X))) # True: scores = U * S
C = Xc.T @ Xc / (len(X) - 1)
eigvals = np.linalg.eigvalsh(C)[::-1]
print(np.allclose(eigvals[:10], s[:10] ** 2 / (len(X) - 1))) # True: same eigenvalues
k = 10
approx = U[:, :k] * s[:k] @ Vt[:k] + X.mean(axis=0)
print(f"relative error of rank-{k} reconstruction: {np.linalg.norm(X - approx) / np.linalg.norm(Xc):.3f}")
SVD Variants
Variant
Use
Full SVD
Small or medium dense data
Truncated SVD (ARPACK)
Top k components only
Randomized SVD
Large data, approximate top k fast
TruncatedSVD without centring
Sparse matrices such as TF-IDF (LSA)
Common Pitfalls
Forgetting to centre before the SVD (then it is not PCA, unless intended as with TruncatedSVD).
Comparing signs of components between libraries.
Building the covariance matrix for very wide data instead of using the SVD directly.
Interview Tips
Derive XᵀX = VΣ²Vᵀ to connect SVD and PCA.
Mention numerical stability and truncated/randomized SVD.
11
How do you choose the number of principal components?
KnownRevise
Choose the number of components k from what the reduction is for: for compression or noise removal, take the smallest k whose cumulative explained variance ratio reaches a target such as 90–95%; for exploration, look for the elbow in a scree plot where extra components add little; for prediction, treat k as a hyperparameter and pick it by cross-validated performance of the whole pipeline; and for a statistically principled cut-off, use parallel analysis (keep components whose eigenvalues beat those from random data of the same shape) or probabilistic PCA’s likelihood, which scikit-learn’s n_components="mle" uses. Rules of thumb like the Kaiser criterion (eigenvalue > 1 on standardized data) are quick but unreliable. For visualization, k is simply 2 or 3, and you should report how much variance those keep.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Methods
Method
Rule
Best for
Cumulative variance
Smallest k with ≥ 90–95%
Compression, general use
Scree / elbow
Bend in the eigenvalue curve
Exploration
Cross-validation
Best downstream score
Predictive pipelines
Parallel analysis
Eigenvalue above the random-data 95th percentile
Finding real structure
Kaiser rule
λ > 1 (standardized data)
Quick look only
PPCA likelihood
PCA(n_components="mle")
Automatic choice
Reconstruction error
Error below a tolerance
Compression with quality targets
Code Example
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
pca = PCA().fit(Xs)
cum = np.cumsum(pca.explained_variance_ratio_)
for target in [0.80, 0.90, 0.95, 0.99]:
print(f"{target:.0%} variance needs k = {int(np.searchsorted(cum, target) + 1)}")
rng = np.random.default_rng(42) # parallel analysis
random_eigs = np.array([
PCA().fit(StandardScaler().fit_transform(rng.normal(size=Xs.shape))).explained_variance_
for _ in range(20)
])
threshold = np.percentile(random_eigs, 95, axis=0)
print("parallel analysis keeps", int(np.sum(pca.explained_variance_ > threshold)), "components")
print("Kaiser rule keeps", int(np.sum(pca.explained_variance_ > 1)), "components")
pipe = Pipeline([("scale", StandardScaler()), ("pca", PCA()), ("clf", LogisticRegression(max_iter=3000))])
search = GridSearchCV(pipe, {"pca__n_components": [10, 20, 30, 40, 50]}, cv=5).fit(X, y)
print("best k by cross-validation:", search.best_params_["pca__n_components"],
f"accuracy {search.best_score_:.3f}")
Different rules give different answers; that is normal, and the right one depends on the goal.
Common Pitfalls
Using a fixed 95% rule for a predictive model without validating it.
Choosing k on the full dataset before splitting, then reporting test scores.
Comparing thresholds between scaled and unscaled analyses.
Interview Tips
Name several methods and match each to a goal.
Recommend cross-validation in a Pipeline when there is a target.
12
What does "loading" mean in PCA?
KnownRevise
A loading describes how strongly an original feature contributes to a principal component: in the most common usage it is the feature’s coefficient (weight) in the component’s eigenvector, so component 1 = 0.52·sepal length − 0.27·sepal width + 0.58·petal length + 0.56·petal width for standardized Iris, and in the stricter statistical usage it is that weight scaled by the square root of the eigenvalue, which for standardized data equals the correlation between the feature and the component’s scores. Large absolute loadings show which features define a component, the sign shows the direction (features with the same sign move together along it), and squared loadings across components show how well each feature is represented. In scikit-learn the unit-length weights are pca.components_ (one row per component), and the correlation-style loadings are components_.T * sqrt(explained_variance_).
Read full answerHide full answer
Applies toscikit-learn 1.5+
Two Conventions
Name
Formula
Meaning
Weights / coefficients
Eigenvector entries vⱼₖ (components_)
Recipe for building the score from features
Loadings (correlation-style)
vⱼₖ·√λₖ
Correlation of feature j with component k (standardized data)
Code Example
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
iris = load_iris()
X = StandardScaler().fit_transform(iris.data)
names = ["sepal length", "sepal width", "petal length", "petal width"]
pca = PCA(n_components=2).fit(X)
for k, row in enumerate(pca.components_, start=1):
print(f"PC{k} weights:", dict(zip(names, row.round(3))))
# PC1 about [0.52, -0.27, 0.58, 0.56] up to sign: petal size and sepal length together
# PC2 is dominated by sepal width
loadings = pca.components_.T * np.sqrt(pca.explained_variance_)
scores = pca.transform(X)
corr = np.array([[np.corrcoef(X[:, j], scores[:, k])[0, 1] for k in range(2)] for j in range(4)])
print("correlation-style loadings:\n", loadings.round(3))
print("match correlations (up to the ddof factor):", np.allclose(loadings, corr, atol=0.01)) # True
print("share of each feature's variance kept by 2 PCs:", np.sum(corr**2, axis=1).round(3))
The small mismatch between loadings and correlations comes from StandardScaler using ddof = 0 while PCA’s explained variance uses ddof = 1; with 150 rows the difference is about 0.3%.
Interpreting Loadings
Look at the largest absolute loadings per component and name the theme (PC1 here: “overall flower size”).
Compare signs to see which features trade off.
Use a biplot to show scores and loading vectors together.
Common Pitfalls
Interpreting loadings from unscaled data with mixed units.
Reading small loadings as meaningful without checking stability (bootstrap the PCA).
Forgetting that the overall sign of a component is arbitrary.
Interview Tips
Distinguish eigenvector weights from correlation-style loadings.
Show how to read a component from its largest loadings.
13
Explain eigenvalue decomposition in PCA
KnownRevise
Eigenvalue decomposition writes a symmetric matrix as C = VΛVᵀ, where V’s columns are orthonormal eigenvectors and Λ is a diagonal matrix of eigenvalues; in PCA you compute the covariance (or correlation) matrix of the centred data, decompose it, sort the eigenpairs from largest to smallest eigenvalue, and the sorted eigenvectors are the principal components while the eigenvalues are the variances they explain. Projecting the data with Z = XcV rotates it into a coordinate system where the covariance is the diagonal Λ, so the new features are uncorrelated. Because covariance matrices are symmetric positive semi-definite, the spectral theorem guarantees real, non-negative eigenvalues and orthogonal eigenvectors; numerically you use a symmetric solver (np.linalg.eigh), or skip forming C and use the SVD of Xc, which gives the same result more stably.
Read full answerHide full answer
Applies toNumPy 2.x
Process
Step
Operation
Cost
1
Centre (and standardize) X (n × d)
O(nd)
2
C = XcᵀXc / (n − 1)
O(nd²)
3
C = VΛVᵀ with a symmetric solver
O(d³)
4
Sort eigenvalues descending, reorder V
O(d log d)
5
Keep the top k columns, Z = XcV_k
O(ndk)
Code Example
import numpy as np
rng = np.random.default_rng(42)
X = rng.normal(size=(300, 4)) @ np.array([[2, 0, 0, 0],
[1.5, 1, 0, 0],
[0, 0, 0.5, 0],
[0.5, 0.3, 0, 0.2]])
Xc = X - X.mean(axis=0)
C = np.cov(Xc, rowvar=False)
eigvals, V = np.linalg.eigh(C) # ascending
order = np.argsort(eigvals)[::-1]
eigvals, V = eigvals[order], V[:, order]
print("eigenvalues:", eigvals.round(3))
print("C rebuilt from V Lambda V^T:", np.allclose(V @ np.diag(eigvals) @ V.T, C)) # True
print("V orthonormal:", np.allclose(V.T @ V, np.eye(4))) # True
Z = Xc @ V
print("covariance of the rotated data:\n", np.cov(Z, rowvar=False).round(3)) # diagonal = eigenvalues
print("explained variance ratio:", (eigvals / eigvals.sum()).round(3))
k = 2
X_approx = Z[:, :k] @ V[:, :k].T + X.mean(axis=0)
print(f"rank-{k} reconstruction keeps {eigvals[:k].sum() / eigvals.sum():.1%} of the variance")
Eigen-decomposition vs SVD
Eigen-decomposition of C
SVD of Xc
Forms d × d matrix
Yes
No
Precision
Squares the condition number
Better
Good when
d small, n huge
General default
Library
np.linalg.eigh
np.linalg.svd, sklearn’s PCA
Common Pitfalls
Using np.linalg.eig and forgetting to sort or to take the real part.
Decomposing the uncentred XᵀX.
Building C when d is very large (memory d²).
Interview Tips
Write C = VΛVᵀ and explain sorting, projection and diagonal covariance.
Mention the spectral theorem and why SVD is often preferred.
Show its role in the explained variance ratio and the PCA objective.
15
What are PCA's limitations with non-linear relationships?
KnownRevise
PCA can only find flat, linear subspaces: each component is a weighted sum of the original features, so when the data lies on a curved structure (a spiral, a circle, a “Swiss roll”, clusters arranged in rings) the directions of maximum variance cut straight across it, mixing points that are far apart along the curve and needing many components to describe something that is really low-dimensional. PCA also captures only second-order statistics (variances and covariances), so it misses non-linear dependence such as y = x², where the correlation is zero but the relationship is perfect. When non-linear structure matters, use kernel PCA (PCA in a feature space defined by a kernel), manifold learning (Isomap, locally linear embedding), t-SNE or UMAP for visualization, or autoencoders, which are non-linear generalizations of PCA, and still try PCA first because it is fast, stable and invertible.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: Concentric Circles
import numpy as np
from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X, y = make_circles(n_samples=1000, factor=0.3, noise=0.05, random_state=42)
z_pca = PCA(n_components=1).fit_transform(X)
z_kpca = KernelPCA(n_components=1, kernel="rbf", gamma=10).fit_transform(X)
for name, z in [("linear PCA", z_pca), ("kernel PCA (RBF)", z_kpca)]:
acc = cross_val_score(LogisticRegression(), z, y, cv=5).mean()
print(f"{name:17s}: accuracy from 1 component {acc:.3f}")
# about 0.5 for linear PCA (the rings overlap on any straight line) vs close to 1.0 for kernel PCA
rng = np.random.default_rng(42)
x = rng.uniform(-1, 1, 5000)
print(f"corr(x, x^2) = {np.corrcoef(x, x**2)[0, 1]:.3f}") # about 0: PCA sees no relationship
Alternatives
Method
Non-linear?
Invertible
Out-of-sample
Best for
PCA
No
Yes
Yes
Baseline, compression
Kernel PCA
Yes
Approximate
Yes
Curved structure, moderate n
Isomap / LLE
Yes
No
Limited
Smooth manifolds
t-SNE
Yes
No
No
2-D visualization of clusters
UMAP
Yes
Approximate
Yes
Visualization and features, large n
Autoencoder
Yes
Yes (decoder)
Yes
Large data, images
Common Pitfalls
Concluding there’s no structure because PCA plots look like a blob.
Using t-SNE output as model features (no transform for new data, unstable distances).
Kernel PCA on very large n (an n × n kernel matrix).
Interview Tips
Give a concrete failure (circles, Swiss roll, x vs x²).
List kernel PCA, manifold methods and autoencoders with their trade-offs.
16
How is PCA used in image processing?
KnownRevise
In image processing PCA treats each image (or each small patch) as a long vector of pixel values and finds the few directions that capture most of the variation across images; this gives compression (store k scores per image instead of every pixel), denoising (reconstruct from the top components, which keep structure, and drop the rest, which are mostly noise), face recognition with eigenfaces (compare faces by their component scores), and fast low-dimensional features for classifiers. Other uses are colour-space decorrelation (PCA on RGB values, used in AlexNet-style “PCA colour augmentation”), aligning or detecting objects with PCA on shapes, and analysing hyperspectral images with hundreds of bands. Modern deep networks outperform PCA for recognition, but it remains a fast, interpretable baseline and a common preprocessing step.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Applications
Application
How PCA is used
Compression
Keep k component scores, reconstruct with inverse_transform
Denoising
Reconstruct from the top k components
Eigenfaces
Faces as combinations of “eigenface” images
Feature extraction
Scores as classifier inputs
Colour augmentation
Perturb images along RGB principal axes
Hyperspectral imaging
Reduce hundreds of bands to a few
Shape models
Active shape models of landmark points
Code Example: Compression and Denoising of Digits
import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
X = load_digits().data / 16.0 # 8x8 images, pixels scaled to 0..1
X_train, X_test = train_test_split(X, test_size=0.3, random_state=42)
for k in [5, 10, 20, 40]:
pca = PCA(n_components=k).fit(X_train)
recon = pca.inverse_transform(pca.transform(X_test))
mse = np.mean((X_test - recon) ** 2)
print(f"k={k:>2}: {k} numbers instead of 64 ({64 / k:.1f}x smaller), MSE {mse:.4f}")
rng = np.random.default_rng(42)
noisy = np.clip(X_test + rng.normal(0, 0.25, X_test.shape), 0, 1)
pca = PCA(n_components=12).fit(X_train) # fit on clean training images
denoised = pca.inverse_transform(pca.transform(noisy))
print(f"error vs clean: noisy {np.mean((noisy - X_test) ** 2):.4f}, denoised {np.mean((denoised - X_test) ** 2):.4f}")
# The denoised images are much closer to the clean ones than the noisy input
Storing the mean image and the k components is a fixed cost shared by every image, so the savings grow with the number of images compressed.
Common Pitfalls
Applying PCA to unaligned images; faces must be cropped and aligned for eigenfaces to make sense.
Using global PCA on large images (millions of pixels); use patches or randomized PCA.
Expecting PCA to match modern codecs (JPEG, learned compression) for photo quality.
Interview Tips
Explain images as vectors and reconstruction from k components.
Give compression, denoising and eigenfaces as examples.
17
What is the curse of dimensionality and how does PCA help?
KnownRevise
The curse of dimensionality is the set of problems that appear as the number of features grows: the volume of the space grows exponentially, so data becomes sparse and you need exponentially more samples to cover it; distances between points become nearly equal, which weakens nearest-neighbour methods and clustering; and models have more parameters to fit, so they overfit more easily and cost more to train. PCA helps when the data actually lies close to a lower-dimensional linear subspace, which is common because real features are correlated: projecting onto the top k components keeps most of the structure, removes redundant and noisy directions, makes distances more meaningful, and reduces model size and training time. It cannot help if the information is spread evenly across many dimensions or lies on a non-linear manifold, and it doesn’t use the target, so check that downstream performance actually improves.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Symptoms
Symptom
Example
Data sparsity
Covering [0, 1]ᵈ with 10 bins per axis needs 10ᵈ cells
Distance concentration
Nearest and farthest neighbours almost equally far
Overfitting
More features than samples
Computation
Cost grows with d
Code Example
import numpy as np
from sklearn.decomposition import PCA
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
for d in [2, 10, 100, 1000]: # distance concentration
pts = rng.random((500, d))
dist = np.linalg.norm(pts[1:] - pts[0], axis=1)
print(f"d={d:>4}: farthest/nearest distance ratio {dist.max() / dist.min():.2f}")
# The ratio falls toward 1 as d grows, so "nearest" means less and less
n, latent_dim, noise_dims = 400, 5, 195
y = rng.integers(0, 2, n)
latent = rng.normal(size=(n, latent_dim)) + y[:, None] * 0.8 # class signal in 5 dimensions
mix = rng.normal(size=(latent_dim, 50))
X = np.hstack([latent @ mix, rng.normal(size=(n, noise_dims)) * 0.5]) # 245 features
knn = KNeighborsClassifier(n_neighbors=15)
raw = cross_val_score(make_pipeline(StandardScaler(), knn), X, y, cv=5).mean()
reduced = cross_val_score(make_pipeline(StandardScaler(), PCA(n_components=5), knn), X, y, cv=5).mean()
print(f"kNN on 245 features: {raw:.3f}; after PCA to 5: {reduced:.3f}")
# PCA keeps the correlated signal directions and drops most of the noise dimensions
Common Pitfalls
Expecting PCA to help when every feature carries independent information.
Fitting PCA before cross-validation splits.
Ignoring non-linear structure that PCA can’t capture.
Interview Tips
Name sparsity, distance concentration and overfitting.
Explain that PCA works by exploiting correlation between features.
18
How does PCA handle missing values?
KnownRevise
Standard PCA does not handle missing values at all: scikit-learn’s PCA raises an error on NaN, and the covariance and SVD computations need complete data, so you must deal with gaps first. Simple options are dropping rows (fine when few rows are affected and values are missing at random) or imputing with the mean or median inside a Pipeline (fast, but it shrinks variances and correlations, which shrinks and distorts the components). Better options treat PCA itself as the imputation model: iterative PCA (fill gaps, fit a low-rank PCA, replace the gaps with the reconstruction, repeat until it stabilizes), probabilistic PCA fitted with the EM algorithm, which handles missing entries in the likelihood, or a multivariate iterative imputer; matrix completion methods (soft-impute, nuclear-norm minimization) do the same at scale. Whichever you choose, fit the imputer on training data only, and add missing-value indicators if missingness itself is informative.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Options
Approach
Pros
Cons
Drop rows
Simple
Loses data; biased if not MCAR
Mean/median imputation
Fast, pipeline-friendly
Shrinks variance and correlations
KNN imputation
Uses similar rows
Slow for large data
Iterative PCA (low-rank imputation)
Uses the structure PCA finds
Needs k chosen; iterative
Probabilistic PCA with EM
Principled likelihood
More complex
Iterative imputer
Flexible models per feature
Slower
Code Example: Mean Imputation vs Iterative PCA
import numpy as np
from sklearn.decomposition import PCA
rng = np.random.default_rng(42)
n, d, k = 500, 8, 2
X_true = rng.normal(size=(n, k)) @ rng.normal(size=(k, d)) + rng.normal(0, 0.1, (n, d)) # rank-2 data
mask = rng.random((n, d)) < 0.2 # 20% missing at random
X_obs = np.where(mask, np.nan, X_true)
def iterative_pca_impute(X, k, iters=100, tol=1e-6):
missing = np.isnan(X)
filled = np.where(missing, np.nanmean(X, axis=0), X) # start from column means
for _ in range(iters):
pca = PCA(n_components=k).fit(filled)
recon = pca.inverse_transform(pca.transform(filled))
change = np.sqrt(np.mean((filled[missing] - recon[missing]) ** 2))
filled[missing] = recon[missing] # only replace the gaps
if change < tol:
break
return filled
mean_filled = np.where(mask, np.nanmean(X_obs, axis=0), X_obs)
pca_filled = iterative_pca_impute(X_obs, k)
for name, Xf in [("mean imputation", mean_filled), ("iterative PCA", pca_filled)]:
rmse = np.sqrt(np.mean((Xf[mask] - X_true[mask]) ** 2))
ratio = PCA(n_components=2).fit(Xf).explained_variance_ratio_.sum()
print(f"{name:16s}: RMSE on missing cells {rmse:.3f}, variance in 2 PCs {ratio:.3f}")
# Iterative PCA recovers the gaps far better and restores the true low-rank structure (about 0.99)
Common Pitfalls
Fitting imputers on the full dataset before splitting.
Mean imputation with a lot of missing data, which flattens components.
Ignoring why values are missing (MNAR data biases every method).
Interview Tips
Say PCA needs complete data and list the options.
Describe iterative PCA or PPCA with EM as model-based imputation.
19
How is PCA used in feature engineering?
KnownRevise
In feature engineering PCA turns many correlated raw features into a few uncorrelated summary features: it can compress groups of related columns (dozens of sensor readings, survey items or embedding dimensions) into a handful of components, decorrelate inputs for models that dislike multicollinearity (linear and logistic regression), remove noise directions, speed up distance-based models such as kNN and SVMs, and create new informative features such as the reconstruction error, which is a good anomaly score. A practical pattern is to apply PCA to a block of similar features rather than to everything, keep the original important features alongside, and choose the number of components by cross-validation inside a Pipeline so the transformation is learned from training data only. Tree-based models usually gain little from PCA, because rotating features can make their axis-aligned splits harder, while linear and distance-based models gain the most.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Patterns
Pattern
Example
Block-wise PCA
40 sensor columns → 3 components, other features unchanged
Decorrelation
Fix multicollinearity before linear regression
Embedding compression
768-d text embeddings → 64 components
Reconstruction error feature
Anomaly score for fraud or defects
Whitening
Unit-variance, uncorrelated inputs for some algorithms
Code Example: PCA on One Block of Features
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n = 1500
machine_state = rng.normal(size=(n, 2)) # hidden drivers
sensors = machine_state @ rng.normal(size=(2, 40)) + rng.normal(0, 1.0, (n, 40)) # 40 noisy sensors
age, load = rng.uniform(0, 10, n), rng.uniform(0, 1, n)
logit = 1.2 * machine_state[:, 0] - 0.8 * machine_state[:, 1] + 0.3 * age - 3
y = rng.random(n) < 1 / (1 + np.exp(-logit))
X = np.column_stack([sensors, age, load])
sensor_cols, other_cols = list(range(40)), [40, 41]
def model(n_components):
if n_components is None:
prep = StandardScaler()
else:
prep = ColumnTransformer([
("sensors", Pipeline([("scale", StandardScaler()), ("pca", PCA(n_components=n_components))]), sensor_cols),
("other", StandardScaler(), other_cols),
])
return Pipeline([("prep", prep), ("clf", LogisticRegression(C=0.1, max_iter=2000))])
for k in [None, 1, 2, 5, 10]:
score = cross_val_score(model(k), X, y, cv=5, scoring="roc_auc").mean()
label = "raw 42 features" if k is None else f"{k} sensor PCs + 2 others"
print(f"{label:24s}: AUC {score:.3f}")
# Two components summarize the 40 sensors about as well as the raw columns, with far fewer inputs
Common Pitfalls
Applying PCA to all features, mixing unrelated groups and losing interpretability.
Fitting PCA outside the cross-validation loop.
Using PCA in front of gradient-boosted trees by default.
Forgetting to scale each block first.
Interview Tips
Mention block-wise PCA, decorrelation and reconstruction error as a feature.
Say which models benefit and how k is chosen.
20
What is the difference between PCA and t-SNE?
KnownRevise
PCA is a linear, deterministic projection that keeps the directions of maximum global variance, so distances and large-scale structure are preserved as well as a linear map can, and it can transform new data and be inverted; t-SNE is a non-linear, stochastic method for visualization that converts distances into neighbour probabilities and arranges points in 2-D or 3-D so that each point’s close neighbours stay close, which reveals clusters and local structure that PCA can miss. The price is that t-SNE distorts global geometry: the distances between clusters, the sizes of clusters and the empty space around them mean little, results change with the perplexity setting and random seed, it is slow on large data, and it has no transform for new points, so it is used for exploratory plots rather than as model features. A common workflow runs PCA first (to about 50 dimensions) to remove noise and speed up t-SNE, and UMAP is a faster alternative that keeps somewhat more global structure.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Comparison
PCA
t-SNE
Type
Linear
Non-linear
Preserves
Global variance and structure
Local neighbourhoods
Deterministic
Yes (up to sign)
No; depends on seed and perplexity
New data
transform
Not supported
Invertible
Yes
No
Speed
Fast
Slow for large n
Output use
Features, compression, plots
Plots only
Interpretable axes
Yes (loadings)
No
Code Example
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE, trustworthiness
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
pca_2d = PCA(n_components=2).fit_transform(Xs)
tsne_2d = TSNE(n_components=2, perplexity=30, init="pca", random_state=42).fit_transform(
PCA(n_components=30).fit_transform(Xs))
for name, emb in [("PCA", pca_2d), ("t-SNE", tsne_2d)]:
trust = trustworthiness(Xs, emb, n_neighbors=10)
knn = cross_val_score(KNeighborsClassifier(5), emb, y, cv=5).mean()
print(f"{name:5s}: trustworthiness {trust:.3f}, kNN accuracy in 2-D {knn:.3f}")
# t-SNE keeps neighbourhoods much better: digit classes form clear separate groups in 2-D
Cross-validated kNN on a t-SNE embedding is only a way to measure how well classes separate in the plot; the embedding was computed on all points, so it is not a valid modelling pipeline.
Common Pitfalls
Reading cluster sizes or distances between clusters from a t-SNE plot.
Drawing conclusions from one t-SNE run; try several perplexities and seeds.
Using t-SNE coordinates as model features.
Interview Tips
Contrast global vs local structure, linear vs non-linear, transform vs none.
Mention PCA-then-t-SNE and UMAP.
21
How can PCA be used to reduce noise?
KnownRevise
PCA reduces noise by assuming the real signal lives in a few high-variance directions while random noise spreads its variance thinly over all directions: project the data onto the top k components and map it back with the inverse transform, x̂ = μ + W_kW_kᵀ(x − μ), and the reconstruction keeps the structured signal while discarding the low-variance components that are mostly noise. It works best when the signal is approximately low-rank and the noise is roughly independent and similar in size across features (white noise); the number of components is chosen where the eigenvalue spectrum flattens into a “noise floor”, by cross-validation on a denoising target, or by random matrix theory thresholds. Typical uses are cleaning sensor and spectral data, image and patch denoising, removing noise from gene expression or financial returns, and stabilizing features before modelling.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Code Example
import numpy as np
from sklearn.decomposition import PCA
rng = np.random.default_rng(42)
n, d, rank = 1000, 50, 3
t = np.linspace(0, 1, d)
basis = np.vstack([np.sin(2 * np.pi * t), np.cos(2 * np.pi * t), t]) # 3 smooth patterns
clean = rng.normal(size=(n, rank)) @ basis # low-rank signal
noisy = clean + rng.normal(0, 0.5, (n, d))
pca_full = PCA().fit(noisy)
print("top eigenvalues:", pca_full.explained_variance_[:6].round(3))
# Three large values, then a flat floor near the noise variance 0.25
for k in [1, 3, 10, 50]:
pca = PCA(n_components=k).fit(noisy)
denoised = pca.inverse_transform(pca.transform(noisy))
print(f"k={k:>2}: MSE vs clean {np.mean((denoised - clean) ** 2):.4f}")
print(f"noisy input MSE vs clean: {np.mean((noisy - clean) ** 2):.4f}") # about 0.25
# k=3 gives the lowest error: about 3/50 of the noise is left (roughly 0.015)
With k = 3, the reconstruction keeps only the noise that falls inside the 3-dimensional signal subspace, about 3/50 of it, so the error falls from about 0.25 to roughly 0.015.
Choosing k for Denoising
Method
Idea
Spectrum inspection
Keep components above the flat noise floor
Known noise level
Keep eigenvalues clearly above σ²
Marchenko-Pastur bound
Random-noise eigenvalues fall below σ²(1 + √(d/n))²
Validation
Minimize error on held-out clean or repeated measurements
Common Pitfalls
Too many components, which keep noise; too few, which remove real signal.
Non-uniform noise across features; standardize or whiten the noise first.
Fitting on the data you evaluate on when the goal is a reusable denoiser.
Assuming small components are always noise; rare but important signals can hide there.
Interview Tips
Explain signal in a low-rank subspace and noise spread across all directions.
Show how to read the noise floor from the spectrum.
22
How would you use PCA for visualization?
KnownRevise
To visualize high-dimensional data with PCA, standardize the features, fit PCA, and plot each sample’s scores on the first two (or three) components, coloured by a label or a variable of interest; because PCA is a linear projection that preserves as much variance as possible, the plot shows the main global structure, such as clusters, gradients, outliers and groups that overlap, along axes you can explain from the loadings. Always report how much variance the plotted components explain (a 2-D plot that keeps 30% of the variance hides a lot), add a scree plot, and use a biplot (scores plus loading arrows) when the audience needs to see which features drive the axes. If the structure is non-linear, follow up with t-SNE or UMAP, often run on the top PCA components.
Read full answerHide full answer
Applies toscikit-learn 1.5+, Matplotlib 3.8+
Code Example: Scores Plot, Biplot and Scree Plot
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
wine = load_wine()
X = StandardScaler().fit_transform(wine.data)
pca = PCA().fit(X)
scores = pca.transform(X)[:, :2]
ratio = pca.explained_variance_ratio_
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
ax = axes[0]
for cls, marker in zip(range(3), ["o", "s", "^"]):
pts = scores[wine.target == cls]
ax.scatter(pts[:, 0], pts[:, 1], marker=marker, alpha=0.7, label=wine.target_names[cls])
scale = np.abs(scores).max() * 0.8
for j in np.argsort(-np.linalg.norm(pca.components_[:2].T, axis=1))[:5]: # 5 strongest features
vx, vy = pca.components_[0, j] * scale, pca.components_[1, j] * scale
ax.arrow(0, 0, vx, vy, color="black", width=0.01, head_width=0.1)
ax.annotate(wine.feature_names[j], (vx * 1.1, vy * 1.1), fontsize=8)
ax.set(xlabel=f"PC1 ({ratio[0]:.1%})", ylabel=f"PC2 ({ratio[1]:.1%})", title="Wine: PCA biplot")
ax.legend()
ax = axes[1]
ax.bar(range(1, len(ratio) + 1), ratio, label="per component")
ax.plot(range(1, len(ratio) + 1), np.cumsum(ratio), marker="o", color="black", label="cumulative")
ax.set(xlabel="component", ylabel="explained variance ratio", title="Scree plot")
ax.legend()
fig.tight_layout()
fig.savefig("pca_visualization.png", dpi=120)
print(f"the 2-D plot shows {ratio[:2].sum():.1%} of the variance") # about 55% for standardized wine
Checklist
Standardize unless features share a meaningful unit.
Label axes with the explained variance of each component.
Use colour, marker shape and an accessible palette for groups.
Show the scree plot so readers know what is missing.
Use equal axis scales so distances aren’t distorted.
Common Pitfalls
Concluding that groups overlap when they may separate in later components.
Unscaled data making the plot about one feature’s units.
Over-reading small differences in a 2-D view that keeps little variance.
Interview Tips
Describe scores plots, biplots and scree plots.
Say you would report explained variance and try non-linear methods if needed.
23
How are PCA results affected by outliers, and how can you fix it?
KnownRevise
PCA is sensitive to outliers because it maximizes variance, a sum of squared deviations, so a few extreme points can contribute most of the variance and pull the first components toward themselves, rotating the axes away from the structure of the bulk of the data and inflating the explained variance of a misleading direction. To address it, first detect outliers (high Mahalanobis distance, large scores on some components, or large reconstruction error), then decide whether they are errors to remove or real rare events to keep; use robust scaling (median and IQR) instead of mean and standard deviation; and use robust PCA variants: PCA on a robust covariance estimate such as the Minimum Covariance Determinant (MinCovDet), projection-pursuit methods like ROBPCA, or Principal Component Pursuit, which splits the data into a low-rank part plus a sparse outlier part.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Code Example: Outliers Rotate the First Component
import numpy as np
from sklearn.covariance import MinCovDet
from sklearn.decomposition import PCA
rng = np.random.default_rng(42)
n = 300
x = rng.normal(0, 3, n)
y = 0.5 * x + rng.normal(0, 0.5, n) # main direction about 26.6 degrees
X = np.column_stack([x, y])
outliers = np.array([[2, 30], [3, 28], [1, 32], [2.5, 29], [1.5, 31]]) # 5 points far up the y-axis
X_dirty = np.vstack([X, outliers])
def angle(v):
v = v if v[0] >= 0 else -v
return float(np.degrees(np.arctan2(v[1], v[0])))
print(f"clean PC1 angle {angle(PCA(1).fit(X).components_[0]):.1f} degrees") # about 26.6
print(f"with 5 outliers {angle(PCA(1).fit(X_dirty).components_[0]):.1f} degrees") # pulled up to roughly 70
mcd = MinCovDet(random_state=42).fit(X_dirty) # robust covariance ignores the outliers
vals, vecs = np.linalg.eigh(mcd.covariance_)
print(f"robust (MCD) PC1 angle {angle(vecs[:, -1]):.1f} degrees") # back near 26.6
d2 = mcd.mahalanobis(X_dirty) # flag outliers
print("flagged rows:", np.where(d2 > np.percentile(d2, 98.5))[0]) # includes 300-304
Remedies
Remedy
How
Inspect and clean
Fix data-entry errors; justify any removal
Robust scaling
RobustScaler (median, IQR)
Robust covariance PCA
Eigenvectors of the MCD covariance
ROBPCA
Projection pursuit plus MCD, for high dimensions
Principal Component Pursuit
X = L (low rank) + S (sparse outliers)
Transformations
Log or rank transforms for heavy-tailed features
Common Pitfalls
Deleting outliers automatically when they may be the interesting cases (fraud, faults).
Detecting outliers with the same non-robust PCA they distort (masking).
Standardizing with mean and standard deviation that the outliers have inflated.
Interview Tips
Explain why squared deviations make PCA sensitive.
Mention MCD-based PCA, robust PCA (L + S) and careful outlier handling.
24
Can PCA be applied to categorical data?
KnownRevise
Standard PCA is designed for continuous numeric variables, because it relies on means, variances and Euclidean distances, which have no natural meaning for categories such as colour or city; you can technically run PCA on one-hot encoded columns and it will produce something, but the result is distorted: dummy columns of rare categories have small variances and get little weight, the columns of one variable are dependent (they sum to 1), and distances reflect the coding rather than real similarity. The proper tools are Multiple Correspondence Analysis (MCA) for purely categorical data, which is PCA on a suitably weighted indicator matrix using chi-squared distances; Factor Analysis of Mixed Data (FAMD) for a mix of numeric and categorical variables; and categorical PCA (CATPCA) for ordinal data. For prediction tasks, embeddings or target encoding are often better than any linear reduction, and ordinal codes like 1, 2, 3 should only be used when the order and spacing are meaningful.
Read full answerHide full answer
Applies toNumPy 2.x, pandas 2.x, scikit-learn 1.5+ (the prince library implements MCA and FAMD)
Options
Data
Method
Idea
Nominal categories only
MCA
Correspondence analysis of the indicator matrix
Two categorical variables
Correspondence analysis
SVD of standardized contingency residuals
Mixed numeric and categorical
FAMD
Scale numeric columns, weight dummies, then PCA
Ordinal
CATPCA, or careful integer codes
Optimal scaling of levels
High-cardinality categories
Embeddings, target encoding
Learned or supervised representations
Code Example: Correspondence Analysis from Scratch
import numpy as np
import pandas as pd
rng = np.random.default_rng(42)
n = 2000
region = rng.choice(["north", "south", "east"], n, p=[0.5, 0.3, 0.2])
pref = np.where(region == "north", rng.choice(["tea", "coffee", "juice"], n, p=[0.7, 0.2, 0.1]),
np.where(region == "south", rng.choice(["tea", "coffee", "juice"], n, p=[0.2, 0.7, 0.1]),
rng.choice(["tea", "coffee", "juice"], n, p=[0.3, 0.3, 0.4])))
table = pd.crosstab(region, pref)
print(table)
P = table.to_numpy() / table.to_numpy().sum()
r, c = P.sum(axis=1), P.sum(axis=0)
S = (P - np.outer(r, c)) / np.sqrt(np.outer(r, c)) # standardized residuals
U, s, Vt = np.linalg.svd(S, full_matrices=False)
inertia = s**2 / np.sum(s**2)
print("share of inertia per dimension:", inertia.round(3)) # a 3x3 table has at most 2 dimensions
row_coords = (U[:, :2] * s[:2]) / np.sqrt(r)[:, None]
col_coords = (Vt[:2].T * s[:2]) / np.sqrt(c)[:, None]
print(pd.DataFrame(row_coords, index=table.index, columns=["dim1", "dim2"]).round(3))
print(pd.DataFrame(col_coords, index=table.columns, columns=["dim1", "dim2"]).round(3))
# North sits near tea and south near coffee on dimension 1; east sits nearer juice
chi2 = n * np.sum(s**2) # total inertia times n is the chi-squared statistic
print(f"chi-squared = {chi2:.1f}")
Running plain PCA on one-hot data and over-interpreting it.
Forgetting that rare categories can dominate MCA dimensions; merge very rare levels.
Interview Tips
Explain why variance and distance don’t fit nominal data.
Name MCA, FAMD and CATPCA, and embeddings for modelling.
25
What are the pros and cons of kernel PCA versus linear PCA?
KnownRevise
Kernel PCA performs PCA in a high-dimensional feature space φ(x) without ever computing φ explicitly: it builds the n × n kernel matrix Kᵢⱼ = k(xᵢ, xⱼ) (for example an RBF kernel exp(−γ‖xᵢ − xⱼ‖²)), centres it, and takes its top eigenvectors, which lets it capture non-linear structure such as concentric circles or curved manifolds that linear PCA cannot. The drawbacks follow from the same construction: memory and time grow with n² and n³ rather than with the number of features, results depend heavily on the kernel and its parameters (γ, degree) which must be tuned, components are no longer combinations of the original features so there are no interpretable loadings, mapping back to the input space only gives an approximate “pre-image”, and explained variance in feature space is not comparable with linear PCA’s. Linear PCA stays the default for speed, interpretability and exact reconstruction; kernel PCA is for moderate-sized data with clear non-linear structure, with Nyström approximations for larger n.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Comparison
Linear PCA
Kernel PCA
Structure captured
Linear
Non-linear (via kernel)
Cost
O(nd² ) or O(n d k) with SVD
O(n²) memory, O(n³) time
Hyperparameters
k
k, kernel, γ/degree
Interpretability
Loadings on original features
None directly
Inverse transform
Exact (for kept components)
Approximate pre-image (learned)
New data
Fast projection
Kernel against all training points
Scale
Millions of rows
Thousands (Nyström for more)
Code Example
import time
import numpy as np
from sklearn.datasets import make_moons
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
X, y = make_moons(n_samples=1000, noise=0.08, random_state=42)
linear = Pipeline([("pca", PCA(n_components=2)), ("clf", LogisticRegression())])
kernel = Pipeline([("kpca", KernelPCA(n_components=2, kernel="rbf")), ("clf", LogisticRegression())])
search = GridSearchCV(kernel, {"kpca__gamma": [0.5, 2, 5, 15]}, cv=5).fit(X, y)
print(f"linear PCA + logistic: {cross_val_score(linear, X, y, cv=5).mean():.3f}") # about 0.88
print(f"kernel PCA + logistic: {search.best_score_:.3f} with gamma={search.best_params_['kpca__gamma']}")
rng = np.random.default_rng(42)
for n in [1000, 4000]: # cost grows quickly with n
Xn = rng.normal(size=(n, 10))
start = time.perf_counter()
KernelPCA(n_components=2, kernel="rbf", eigen_solver="dense").fit(Xn)
print(f"n={n}: kernel PCA fit {time.perf_counter() - start:.2f}s, kernel matrix {n * n * 8 / 1e6:.0f} MB")
kpca = KernelPCA(n_components=2, kernel="rbf", gamma=15, fit_inverse_transform=True).fit(X)
X_back = kpca.inverse_transform(kpca.transform(X))
print(f"approximate pre-image error: {np.mean((X - X_back) ** 2):.4f}")
The 4000-point kernel matrix takes about 128 MB, sixteen times the 1000-point one, while linear PCA’s cost barely changes.
Common Pitfalls
Using kernel PCA on large n without approximation (memory errors).
Not tuning γ; too small behaves like linear PCA, too large overfits.
Expecting loadings or exact reconstructions.
Interview Tips
Explain the kernel trick and the n × n kernel matrix.
Weigh non-linearity against cost, tuning and interpretability.
26
Write a Python function to perform PCA from scratch with NumPy
KnownRevise
A from-scratch PCA needs four steps: centre the data by subtracting each feature’s mean (and optionally divide by its standard deviation), compute the singular value decomposition of the centred matrix (or the eigen-decomposition of its covariance), take the top k right singular vectors as components, and project the centred data onto them; the explained variances are the squared singular values divided by n − 1. A good implementation stores the mean so new data can be transformed the same way, fixes the arbitrary signs of the components so results are reproducible, offers an inverse transform for reconstruction, and is checked against sklearn.decomposition.PCA, which should agree up to the sign of each component.
import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA as SklearnPCA
from sklearn.preprocessing import StandardScaler
class PCA:
"""Minimal PCA via SVD of the centred data."""
def __init__(self, n_components=None):
self.n_components = n_components
def fit(self, X):
X = np.asarray(X, dtype=float)
n, d = X.shape
self.mean_ = X.mean(axis=0)
Xc = X - self.mean_
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
max_abs_cols = np.argmax(np.abs(U), axis=0) # deterministic signs (like sklearn's svd_flip)
signs = np.sign(U[max_abs_cols, range(U.shape[1])])
Vt = Vt * signs[:, None]
k = self.n_components or min(n, d)
self.components_ = Vt[:k]
self.explained_variance_ = (s[:k] ** 2) / (n - 1)
self.explained_variance_ratio_ = (s[:k] ** 2) / np.sum(s ** 2)
self.singular_values_ = s[:k]
return self
def transform(self, X):
return (np.asarray(X, dtype=float) - self.mean_) @ self.components_.T
def fit_transform(self, X):
return self.fit(X).transform(X)
def inverse_transform(self, Z):
return Z @ self.components_ + self.mean_
X = StandardScaler().fit_transform(load_iris().data)
mine = PCA(n_components=2).fit(X)
ref = SklearnPCA(n_components=2, svd_solver="full").fit(X)
print("ratios:", mine.explained_variance_ratio_.round(4)) # [0.7296 0.2285]
print("variances match:", np.allclose(mine.explained_variance_, ref.explained_variance_)) # True
signs = np.sign(np.sum(mine.components_ * ref.components_, axis=1))
print("components match up to sign:", np.allclose(mine.components_ * signs[:, None], ref.components_)) # True
print("projections match:", np.allclose(mine.transform(X) * signs, ref.transform(X))) # True
X_back = mine.inverse_transform(mine.transform(X))
print(f"reconstruction MSE with 2 components: {np.mean((X - X_back) ** 2):.4f}")
# equals the discarded share of variance: about (1 - 0.9581) of the total
The reconstruction error per entry is the discarded variance (about 4.2% of the total of 4 features of unit variance), so it prints about 0.042.
Design Notes
Choice
Reason
SVD instead of covariance eigen-decomposition
Better numerical stability, no d × d matrix
Store mean_
Same centring for new data
Sign convention
Reproducible output
ddof = n − 1
Matches scikit-learn’s explained variance
Common Pitfalls
Forgetting to centre, or centring test data with its own mean.
Using np.linalg.eig on the covariance and forgetting to sort.
Comparing component signs directly with another library.
Interview Tips
Walk through centre → SVD → components → projection.
Mention checking against scikit-learn and the sign ambiguity.
27
Apply PCA with scikit-learn to a high-dimensional dataset and interpret it
KnownRevise
To apply PCA to a high-dimensional dataset with scikit-learn, put StandardScaler and PCA in a Pipeline, fit it on training data, and interpret the result in three layers: the explained variance ratio (how many components carry most of the structure), the loadings in components_ (which original features define each component), and the scores (where samples sit, which reveals groups, gradients and outliers). For example, on the 30-feature breast cancer dataset the first component explains about 44% of the variance and loads on almost every size and shape feature with the same sign, so it measures “overall tumour size and irregularity”; together the first two components separate malignant and benign tumours well, and about 10 components keep 95% of the variance. The interpretation should be checked against the target and against stability (does a bootstrap give the same loadings?).
Read full answerHide full answer
Applies toscikit-learn 1.5+, pandas 2.x
Code Example
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X, y = data.data, data.target # 569 x 30
pipe = make_pipeline(StandardScaler(), PCA()).fit(X)
pca = pipe[-1]
ratio = pca.explained_variance_ratio_
print("first five ratios:", ratio[:5].round(3)) # about [0.443 0.19 0.094 0.066 0.055]
print("components for 95% variance:", int(np.searchsorted(np.cumsum(ratio), 0.95) + 1)) # 10
loadings = pd.DataFrame(pca.components_[:2].T, index=data.feature_names, columns=["PC1", "PC2"])
print(loadings["PC1"].abs().sort_values(ascending=False).head(5).round(3))
print(loadings["PC2"].sort_values().head(3).round(3), loadings["PC2"].sort_values().tail(3).round(3))
# PC1: concave points, concavity, perimeter, area (size and irregularity)
# PC2: contrasts radius/area with fractal dimension and smoothness (size vs texture detail)
scores = pipe.transform(X)[:, :2]
for label, name in enumerate(data.target_names):
print(f"{name:9s} mean PC1 score {scores[y == label, 0].mean():.2f}") # malignant high, benign low
for k in [2, 5, 10]:
model = make_pipeline(StandardScaler(), PCA(n_components=k), LogisticRegression(max_iter=1000))
print(f"k={k:>2}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
How to Interpret
Output
Question it answers
explained_variance_ratio_
How low-dimensional is the data?
components_ (loadings)
What does each component mean?
Scores (transform)
How do samples or classes spread out?
Downstream CV score
Does the reduction keep what matters?
Common Pitfalls
Interpreting signs as meaningful in isolation (the whole component can flip).
Naming components from one or two loadings without looking at the full pattern.
Fitting PCA on all data and then reporting test accuracy.
Interview Tips
Walk through variance, loadings and scores with a named dataset.
Mention checking with a downstream model and loading stability.
28
Write a script to visualize eigenfaces with PCA
KnownRevise
Eigenfaces are the principal components of a set of aligned face images: flatten each grayscale image into a vector, fit PCA (usually randomized PCA with whitening, because there are thousands of pixel features), and reshape each component back to the image size to see it as a face-like pattern; the first eigenfaces capture lighting and overall face shape, later ones finer features such as eyes, nose and mouth. Every face is then approximately the mean face plus a weighted sum of eigenfaces, so a script typically shows the mean face, a grid of the top eigenfaces, the explained variance curve, and reconstructions of a face with increasing numbers of components; the component scores can also feed a classifier for face recognition. The classic example uses scikit-learn’s Labeled Faces in the Wild (LFW) dataset, which downloads about 200 MB on first use.
Read full answerHide full answer
Applies toscikit-learn 1.5+, Matplotlib 3.8+
Code Example
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import fetch_lfw_people
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
faces = fetch_lfw_people(min_faces_per_person=70, resize=0.4) # downloads on first run
X, y = faces.data, faces.target
h, w = faces.images.shape[1:]
print(f"{X.shape[0]} images of {h}x{w} = {X.shape[1]} pixels, {len(faces.target_names)} people")
X_train, X_test = train_test_split(X, test_size=0.25, random_state=42, stratify=y)
n_components = 150
pca = PCA(n_components=n_components, svd_solver="randomized", whiten=True, random_state=42).fit(X_train)
print(f"{n_components} components keep {pca.explained_variance_ratio_.sum():.1%} of the pixel variance")
fig, axes = plt.subplots(3, 5, figsize=(10, 7))
axes[0, 0].imshow(pca.mean_.reshape(h, w), cmap="gray")
axes[0, 0].set_title("mean face")
for i, ax in enumerate(axes.flat[1:]):
ax.imshow(pca.components_[i].reshape(h, w), cmap="gray")
ax.set_title(f"eigenface {i + 1}")
for ax in axes.flat:
ax.axis("off")
fig.tight_layout()
fig.savefig("eigenfaces.png", dpi=120)
face = X_test[:1]
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
axes[0].imshow(face.reshape(h, w), cmap="gray")
axes[0].set_title("original")
for ax, k in zip(axes[1:], [10, 30, 75, 150]):
partial = PCA(n_components=k, svd_solver="randomized", random_state=42).fit(X_train)
recon = partial.inverse_transform(partial.transform(face))
ax.imshow(recon.reshape(h, w), cmap="gray")
ax.set_title(f"{k} components")
for ax in axes:
ax.axis("off")
fig.tight_layout()
fig.savefig("eigenface_reconstructions.png", dpi=120)
fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(np.cumsum(PCA(n_components=300, svd_solver="randomized", random_state=42)
.fit(X_train).explained_variance_ratio_))
ax.set(xlabel="number of eigenfaces", ylabel="cumulative explained variance")
fig.tight_layout()
fig.savefig("eigenface_variance.png", dpi=120)
The reconstructions use separate non-whitened PCA models because a whitened model’s scores are rescaled; inverse_transform undoes whitening in scikit-learn, but fitting plain models keeps the example easy to follow.
Common Pitfalls
Unaligned or differently sized faces; eigenfaces need cropped, aligned images.
Lighting dominating the first components; drop the first few or normalize lighting for recognition.
Plotting components without reshaping to (h, w) or with a colour map that hides the sign.
Interview Tips
Explain faces as mean plus weighted eigenfaces.
Mention randomized PCA, whitening and using scores for recognition.
29
Implement PCA for feature extraction before a machine learning model
KnownRevise
To use PCA for feature extraction, build a scikit-learn Pipeline of scaling, PCA and the model, so the scaler and PCA are fitted only on each training fold and the same transformation is applied to validation and test data; then tune the number of components together with the model’s hyperparameters using cross-validation, and compare against the same model without PCA to confirm it actually helps. PCA tends to help models that suffer from many correlated or noisy inputs (logistic regression, SVMs, kNN) by reducing variance and training time, and helps less for tree ensembles. The final report should give the chosen k, the variance it keeps, the cross-validated and test scores with and without PCA, and the training-time savings.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example
import time
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
pipe = Pipeline([("scale", StandardScaler()), ("pca", PCA(random_state=42)), ("svm", SVC())])
grid = {
"pca__n_components": [10, 20, 30, 40, 0.95],
"svm__C": [1, 10],
"svm__gamma": ["scale", 0.01],
}
search = GridSearchCV(pipe, grid, cv=5, n_jobs=-1).fit(X_train, y_train)
best = search.best_estimator_
print("best params:", search.best_params_)
print(f"CV accuracy {search.best_score_:.3f}, test accuracy {best.score(X_test, y_test):.3f}")
print(f"components kept: {best.named_steps['pca'].n_components_}, "
f"variance kept: {best.named_steps['pca'].explained_variance_ratio_.sum():.1%}")
baseline = GridSearchCV(Pipeline([("scale", StandardScaler()), ("svm", SVC())]),
{"svm__C": [1, 10], "svm__gamma": ["scale", 0.01]}, cv=5, n_jobs=-1).fit(X_train, y_train)
print(f"without PCA: CV {baseline.best_score_:.3f}, test {baseline.best_estimator_.score(X_test, y_test):.3f}")
for name, model in [("with PCA", best), ("without PCA", baseline.best_estimator_)]:
start = time.perf_counter()
model.fit(X_train, y_train)
print(f"{name}: refit time {time.perf_counter() - start:.3f}s")
On digits both versions reach about 98% accuracy; the PCA version uses far fewer inputs, which matters more as datasets grow.
Pipeline Rules
Rule
Why
Scaler and PCA inside the Pipeline
No leakage from validation folds
Tune k with the model
The best k depends on the model
Compare with no PCA
PCA is not always an improvement
Keep the test set untouched until the end
Honest final estimate
Common Pitfalls
PCA().fit(X_all) before splitting.
Fixing k at 95% variance without checking alternatives.
Adding PCA in front of random forests or boosting out of habit.
Interview Tips
Show the Pipeline and tuning k with cross-validation.
Report the comparison with and without PCA.
30
How would you use PCA for compression in a real-time streaming system?
KnownRevise
In a streaming system you can’t refit PCA on all the data, so learn the components incrementally and compress each record on arrival: fit an initial basis on a warm-up batch (or use IncrementalPCA.partial_fit on mini-batches, or an online algorithm such as Oja’s rule or frequent directions), then encode each new record as z = W_kᵀ(x − μ), sending only the k numbers plus a basis version number, and decode on the receiver with x̂ = μ + W_kz. Because the data distribution drifts, monitor the reconstruction error of incoming records; when it rises, update the basis with recent data, version it, and send the new basis to decoders before it is used. Latency is low because encoding is one d × k matrix-vector product. Decide k by the error budget (for example 95% of variance or a maximum per-record error), quantize the coefficients to save more space, and flag records whose error is unusually high, because they are often anomalies worth keeping in full.
Read full answerHide full answer
Applies tosystem design; example with scikit-learn 1.5+
Design
Component
Choice
Basis learning
IncrementalPCA.partial_fit on mini-batches, or a sliding window refit
Encoding
z = W_kᵀ(x − μ), O(dk) per record
Transmission
k coefficients (optionally quantized to 8–16 bits) + basis version
Decoding
x̂ = μ + W_kz
Drift monitoring
Rolling reconstruction error; refit trigger
Basis updates
Versioned, sent ahead of use; old versions kept for replay
Anomalies
High-error records stored raw
Code Example
import numpy as np
from sklearn.decomposition import IncrementalPCA
rng = np.random.default_rng(42)
d, k, batch = 100, 8, 200
basis_a = rng.normal(size=(k, d))
basis_b = rng.normal(size=(k, d)) # the process changes later
def stream(n_batches, basis):
for _ in range(n_batches):
yield rng.normal(size=(batch, k)) @ basis + rng.normal(0, 0.1, (batch, d))
ipca = IncrementalPCA(n_components=k)
for chunk in stream(10, basis_a): # warm-up
ipca.partial_fit(chunk)
version = 1
def encode(x):
return (ipca.transform(x).astype(np.float16), version) # 8 half-precision numbers instead of 100 floats
def recon_error(x):
z = ipca.transform(x)
return float(np.mean((x - ipca.inverse_transform(z)) ** 2))
baseline_error = recon_error(next(stream(1, basis_a)))
print(f"compression {d * 8 / (k * 2):.0f}x (float64 -> {k} float16 values), baseline error {baseline_error:.4f}")
for step, chunk in enumerate(stream(6, basis_b)):
err = recon_error(chunk)
drift = err > 3 * baseline_error
print(f"batch {step}: error {err:.4f}{' -> drift, updating basis' if drift else ''}")
if drift:
ipca = IncrementalPCA(n_components=k) # refit on recent data and bump the version
ipca.partial_fit(chunk)
version += 1
baseline_error = recon_error(chunk)
payload, v = encode(next(stream(1, basis_b))[:1])
print("payload shape", payload.shape, "basis version", v)
The first batch after the change shows a large error and triggers a refit; later batches compress well again with basis version 2.
Common Pitfalls
Updating the basis silently, so the decoder reconstructs with the wrong components.
No drift monitoring, so quality decays unnoticed.
Ignoring per-feature scaling that also drifts; update the scaler along with PCA.
Using PCA where a domain codec (audio, video) would do much better.
Interview Tips
Describe incremental fitting, encode/decode and basis versioning.
Mention drift detection with reconstruction error and quantization.
31
How would you apply PCA to stock market data?
KnownRevise
In stock market analysis, PCA is applied to the matrix of daily returns (days × stocks), usually standardized, so the components are the main independent patterns that move many stocks together: the first component is almost always the “market factor”, with every stock loading with the same sign and explaining a large share of the variance (often 30–50%), and later components often line up with sectors or styles such as tech versus utilities or growth versus value. This is used for risk models (a portfolio’s exposure to each component, and how concentrated its risk is), statistical arbitrage (trading the residual after removing the top components, which tends to mean-revert), building factor-neutral portfolios, reducing noise in the covariance matrix used for portfolio optimization, and yield-curve analysis, where three components (level, slope, curvature) explain almost all bond yield moves. Work on returns, not prices (prices are non-stationary), use rolling windows because the structure changes over time (correlations jump in crises), and test the results out of sample.
import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
rng = np.random.default_rng(42)
days = 750
sectors = {"tech": 6, "banks": 5, "energy": 4}
market = rng.normal(0, 0.010, days)
sector_moves = {s: rng.normal(0, 0.006, days) for s in sectors}
columns, data = [], []
for sector, count in sectors.items():
for i in range(count):
beta = rng.uniform(0.8, 1.2)
data.append(beta * market + sector_moves[sector] + rng.normal(0, 0.008, days))
columns.append(f"{sector}_{i}")
returns = pd.DataFrame(np.column_stack(data), columns=columns)
z = (returns - returns.mean()) / returns.std()
pca = PCA().fit(z)
print("explained variance ratio:", pca.explained_variance_ratio_[:5].round(3))
pc1 = pd.Series(pca.components_[0], index=columns)
print("PC1 loadings all one sign (market factor):", bool((pc1 > 0).all() or (pc1 < 0).all()))
for k in [1, 2]:
loadings = pd.Series(pca.components_[k], index=columns)
print(f"PC{k + 1} mean loading by sector:", loadings.groupby(lambda c: c.split("_")[0]).mean().round(2).to_dict())
window, k = 250, 3 # rolling PCA residuals for stat-arb style signals
train = z.iloc[:window]
basis = PCA(n_components=k).fit(train)
test = z.iloc[window:window + 20]
residual = test - basis.inverse_transform(basis.transform(test))
print("residual sd vs raw sd:", round(float(residual.to_numpy().std()), 3), round(float(test.to_numpy().std()), 3))
With one market factor and three sectors in the simulation, the first component explains the largest share and the next two separate the sectors; real data looks similar but noisier.
Uses
Use
How PCA helps
Risk decomposition
Exposure to each component; how much risk is “market”
Look-ahead bias: fitting on the full period and backtesting on it.
Assuming components are stable; they rotate across regimes.
Over-interpreting later components as economic factors; check loadings and stability.
Interview Tips
Name the market factor and sector components.
Mention returns not prices, rolling windows and out-of-sample testing.
32
When can PCA hurt a machine learning model's performance?
KnownRevise
PCA can hurt a model when the information that predicts the target lies in low-variance directions that PCA throws away: it chooses components by variance without looking at the label, so a small but decisive signal (for example a tiny voltage difference that marks faulty parts, or a rare word that marks fraud) can be discarded while large but irrelevant variation (lighting, scale, noise in one sensor) is kept. Other cases are tree-based models, which split on single features and can lose performance when PCA mixes features into rotated combinations; non-linear structure that a linear projection flattens; features with different units used without scaling, so one feature dominates; outliers that rotate the components; sparse data where centring destroys sparsity; and loss of interpretability when explanations per original feature are required (credit decisions, medicine). Always compare the model with and without PCA in cross-validation.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: A Small but Important Feature
import numpy as np
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
rng = np.random.default_rng(42)
n = 2000
y = rng.integers(0, 2, n)
nuisance = rng.normal(0, 5, (n, 10)) # big variance, no signal
signal = (y - 0.5) * 0.4 + rng.normal(0, 0.1, n) # small variance, decisive
X = np.column_stack([nuisance, signal])
for name, model in [
("logistic, all features", LogisticRegression(max_iter=2000)),
("logistic, PCA to 5", make_pipeline(PCA(n_components=5), LogisticRegression(max_iter=2000))),
("forest, raw features", RandomForestClassifier(n_estimators=200, random_state=42)),
("forest, PCA to 5", make_pipeline(PCA(n_components=5), RandomForestClassifier(n_estimators=200, random_state=42))),
]:
print(f"{name:24s}: accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
# Models on raw features are near-perfect; after PCA to 5 components they fall to about chance,
# because the signal direction has the smallest variance and is dropped
When to Avoid or Be Careful
Situation
Why PCA can hurt
Signal in low-variance directions
Dropped by PCA
Tree-based models
Rotations break axis-aligned splits
Non-linear class boundaries
Linear projection may merge classes
Interpretability required
Components mix all features
Sparse high-dimensional data
Centring densifies; use TruncatedSVD or none
Mixed units without scaling
Dominated by large-unit features
Common Pitfalls
Adding PCA by default as a “preprocessing step”.
Choosing k by variance alone.
Evaluating the PCA pipeline but not the baseline.
Interview Tips
Give the low-variance-signal scenario and the tree-model scenario.
Recommend a with/without comparison and supervised alternatives (LDA, PLS).
33
How do you make PCA results robust to variations in the data?
KnownRevise
PCA results are robust when the components you report don’t change much if the data is resampled, slightly perturbed or collected at a different time, so you should test that directly and design for it: bootstrap or cross-validate the PCA and measure how similar the components and their explained variance stay (using absolute cosine similarity because signs flip, or subspace angles for components with similar eigenvalues), and only interpret components that are stable. Practical safeguards are consistent preprocessing (the same scaling fitted on training data, handled missing values), robust methods when outliers exist (RobustScaler, MCD-based or robust PCA), enough samples relative to features, and checking the eigenvalue gaps: components whose eigenvalues are nearly equal can rotate freely into each other, so treat them as a shared subspace rather than as individual axes. For data that changes over time, refit on rolling windows and track drift in the components and in reconstruction error.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Code Example: Bootstrap Stability of Components
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X = StandardScaler().fit_transform(load_wine().data)
k = 4
reference = PCA(n_components=k).fit(X)
rng = np.random.default_rng(42)
similarities, ratios = [], []
for _ in range(500):
sample = X[rng.integers(0, len(X), len(X))]
boot = PCA(n_components=k).fit(sample)
sims = np.abs(np.sum(boot.components_ * reference.components_, axis=1)) # |cosine|, sign-free
similarities.append(sims)
ratios.append(boot.explained_variance_ratio_)
similarities, ratios = np.array(similarities), np.array(ratios)
for i in range(k):
lo, hi = np.percentile(ratios[:, i], [2.5, 97.5])
print(f"PC{i + 1}: median |cos| with reference {np.median(similarities[:, i]):.3f}, "
f"5th percentile {np.percentile(similarities[:, i], 5):.3f}, variance ratio 95% CI [{lo:.3f}, {hi:.3f}]")
# PC1 and PC2 are very stable; later components with close eigenvalues vary more
def subspace_similarity(A, B):
"""Mean squared cosine of principal angles between two k-dim subspaces (1 = identical)."""
s = np.linalg.svd(A @ B.T, compute_uv=False)
return float(np.mean(s**2))
sample = X[rng.integers(0, len(X), len(X))]
print("subspace similarity of PCs 3-4:", round(subspace_similarity(PCA(4).fit(sample).components_[2:],
reference.components_[2:]), 3))
Robustness Checklist
Check or safeguard
Purpose
Bootstrap absolute cosine of components
Are loadings stable?
Eigenvalue gaps
Are components individually identifiable?
Subspace angles
Stability of a group of components
Same preprocessing pipeline
Avoids spurious differences
Robust scaling / robust PCA
Outlier resistance
Rolling refits and drift monitoring
Changing data over time
Sample size
n well above d for stable covariance estimates
Common Pitfalls
Comparing components by raw dot products and calling sign flips “instability”.
Interpreting individual components with nearly equal eigenvalues.
Reporting loadings from one fit without any uncertainty.
Interview Tips
Describe bootstrapping with sign-free similarity and eigenvalue gaps.
Mention robust methods and monitoring over time.
34
How can PCA be parallelized for very large datasets?
KnownRevise
PCA parallelizes well because its core computations are sums over rows: each worker can compute partial statistics on its partition (row count, column sums, and the d × d Gram matrix XᵢᵀXᵢ), the driver adds them up to get the global mean and covariance, and a single small eigen-decomposition gives the components, which works whenever the number of features d is moderate (up to a few thousand) no matter how many rows there are. When d is also large, use distributed or randomized SVD: randomized PCA multiplies the data by a random matrix and needs only a few passes of parallel matrix products (as in Dask or Spark MLlib); tall-and-skinny QR (TSQR) combines per-partition QR factorizations; and incremental or streaming methods (IncrementalPCA, frequent directions) process chunks so memory stays bounded. On a single machine, the BLAS libraries behind NumPy already use many cores, and GPUs (cuML, PyTorch torch.pca_lowrank) speed up the matrix products dramatically.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+; Spark MLlib, Dask-ML and cuML follow the same patterns
Strategies
Strategy
Pattern
Best when
Distributed covariance
Map: partial XᵀX and sums; reduce: add; eigh on d × d
Huge n, moderate d
Randomized SVD
Few passes of parallel matrix products
Large n and d, top k only
TSQR
QR per block, then QR of the stacked R factors
Tall, skinny matrices
Incremental PCA
Chunk-by-chunk updates
Out-of-core, single machine
GPU
Dense matrix multiplications on GPU
Medium-large dense data
Code Example: Map-Reduce PCA with a Process Pool
from concurrent.futures import ProcessPoolExecutor
import numpy as np
from sklearn.decomposition import PCA
def partial_stats(chunk):
"""Map step: sufficient statistics for one partition."""
return len(chunk), chunk.sum(axis=0), chunk.T @ chunk
def distributed_pca(chunks, k, workers=4):
with ProcessPoolExecutor(max_workers=workers) as pool:
results = list(pool.map(partial_stats, chunks))
n = sum(r[0] for r in results) # reduce step
total = sum(r[1] for r in results)
gram = sum(r[2] for r in results)
mean = total / n
cov = (gram - n * np.outer(mean, mean)) / (n - 1)
vals, vecs = np.linalg.eigh(cov)
order = np.argsort(vals)[::-1][:k]
return mean, vecs[:, order].T, vals[order]
if __name__ == "__main__":
rng = np.random.default_rng(42)
X = rng.normal(size=(200_000, 30)) @ rng.normal(size=(30, 30))
chunks = np.array_split(X, 16) # stand-ins for partitions on different machines
mean, components, variances = distributed_pca(chunks, k=5)
ref = PCA(n_components=5).fit(X)
signs = np.sign(np.sum(components * ref.components_, axis=1))
print("components match:", np.allclose(components * signs[:, None], ref.components_, atol=1e-6))
print("variances match:", np.allclose(variances, ref.explained_variance_))
rand = PCA(n_components=5, svd_solver="randomized", random_state=42).fit(X)
print("randomized variances close:", np.allclose(rand.explained_variance_, ref.explained_variance_, rtol=1e-3))
The covariance-from-Gram formula can lose precision when the mean is large relative to the spread; shifting each partition by a rough global mean first, or using pairwise (Chan’s) updates, avoids that.
Common Pitfalls
Collecting all data on the driver instead of only the d × d statistics.
Building a d × d matrix when d is in the hundreds of thousands; use randomized or sparse methods.
Numerical cancellation in the one-pass covariance formula.
Shuffling data across the network more than needed.
Interview Tips
Explain the map-reduce of sufficient statistics and when it works (moderate d).
Mention randomized SVD, TSQR, incremental PCA and GPUs.
35
What are best practices for visualizing PCA components?
KnownRevise
Good PCA visualizations make three things clear: how much variance each component explains (a scree plot with per-component and cumulative bars), where samples fall on the main components (a scores scatter plot coloured by a meaningful variable, with axes labelled “PC1 (44.3%)” and equal scaling), and what each component means in terms of the original features (a loadings bar chart or heatmap sorted by magnitude, or a biplot that overlays the strongest loading arrows on the scores). Use diverging colour maps centred on zero for loadings, show only the top features or components so plots stay readable, keep the sign convention consistent across figures, and for image or signal data display components in their natural shape (eigenfaces as images, spectral components as curves). Adding uncertainty, for example bootstrap intervals on loadings, prevents over-reading small differences.
Read full answerHide full answer
Applies toscikit-learn 1.5+, Matplotlib 3.8+
Plot Types
Plot
Shows
Tip
Scree plot
Variance per component, cumulative
Mark the chosen k
Scores scatter
Samples on PC1 vs PC2
Label axes with %; equal aspect
Loadings bar chart
Features driving one component
Sort by absolute value
Loadings heatmap
Features × components
Diverging map centred at 0
Biplot
Scores plus loading arrows
Show only the top features
Component as image/curve
Pixel or spectral components
Natural shape of the data
Code Example
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)
pca = PCA(n_components=6).fit(X)
ratio = pca.explained_variance_ratio_
scores = pca.transform(X)
fig, axes = plt.subplots(1, 3, figsize=(16, 5))
axes[0].bar(range(1, 7), ratio, label="per component")
axes[0].plot(range(1, 7), np.cumsum(ratio), marker="o", color="black", label="cumulative")
axes[0].set(xlabel="component", ylabel="explained variance ratio", title="Scree plot")
axes[0].legend()
for label, marker in [(0, "o"), (1, "s")]:
pts = scores[data.target == label]
axes[1].scatter(pts[:, 0], pts[:, 1], s=12, alpha=0.6, marker=marker, label=data.target_names[label])
axes[1].set(xlabel=f"PC1 ({ratio[0]:.1%})", ylabel=f"PC2 ({ratio[1]:.1%})", title="Scores")
axes[1].set_aspect("equal", adjustable="datalim")
axes[1].legend()
top = np.argsort(-np.abs(pca.components_[:3]).max(axis=0))[:12] # 12 strongest features
limit = np.abs(pca.components_[:3, top]).max()
im = axes[2].imshow(pca.components_[:3, top].T, cmap="RdBu_r", vmin=-limit, vmax=limit, aspect="auto")
axes[2].set_yticks(range(len(top)), [data.feature_names[i] for i in top], fontsize=8)
axes[2].set_xticks(range(3), ["PC1", "PC2", "PC3"])
axes[2].set_title("Loadings (top features)")
fig.colorbar(im, ax=axes[2], shrink=0.8)
fig.tight_layout()
fig.savefig("pca_components.png", dpi=120)
Common Pitfalls
Unlabelled axes or missing explained variance percentages.
Non-equal axis scales that exaggerate spread along PC2.
Sequential colour maps for signed loadings.
Biplots with all 30 arrows, which become unreadable.
Interview Tips
Name scree, scores, loadings and biplots and what each answers.
Mention diverging colour maps, equal aspect and uncertainty.
36
How does PCA compare with other feature selection methods?
KnownRevise
PCA is feature extraction, not feature selection: it builds new features as linear combinations of all the original ones, so it reduces dimensionality and removes redundancy but every original feature is still needed at prediction time and the new features are hard to explain; feature selection keeps a subset of the original columns, which preserves their meaning and can cut data collection costs. Selection methods come in three families: filter methods score each feature independently of a model (variance threshold, correlation, mutual information, chi-squared), wrapper methods search subsets with a model (recursive feature elimination, sequential selection), and embedded methods select during training (L1/Lasso, tree feature importances). PCA is unsupervised, so it may keep irrelevant high-variance directions, while most selection methods use the target. Choose PCA for compression and highly correlated inputs, and selection when interpretability, cost or using the target matters; they can be combined.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Comparison
Method
Type
Uses y
Keeps original features
Cost
PCA
Extraction, unsupervised
No
No (combinations)
Low
Variance threshold
Filter
No
Yes
Very low
Mutual information / chi² / ANOVA F
Filter
Yes
Yes
Low
RFE / sequential selection
Wrapper
Yes
Yes
High
L1 (Lasso, L1 logistic)
Embedded
Yes
Yes
Medium
Tree importances / permutation
Embedded
Yes
Yes
Medium
Code Example
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.feature_selection import RFE, SelectFromModel, SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
clf = LogisticRegression(max_iter=5000)
k = 5
candidates = {
"all 30 features": make_pipeline(StandardScaler(), clf),
"PCA (5 components)": make_pipeline(StandardScaler(), PCA(n_components=k), clf),
"mutual information (5)": make_pipeline(StandardScaler(), SelectKBest(mutual_info_classif, k=k), clf),
"RFE (5)": make_pipeline(StandardScaler(), RFE(LogisticRegression(max_iter=5000), n_features_to_select=k), clf),
"L1 selection": make_pipeline(
StandardScaler(),
SelectFromModel(LogisticRegression(penalty="l1", solver="liblinear", C=0.1)),
clf),
}
for name, model in candidates.items():
print(f"{name:24s}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
# All reach roughly 0.94-0.98; selection methods do it with 5 real, nameable features
rfe = make_pipeline(StandardScaler(), RFE(LogisticRegression(max_iter=5000), n_features_to_select=k)).fit(X, y)
names = load_breast_cancer().feature_names
print("RFE keeps:", list(names[rfe[-1].support_]))
When to Choose Which
Goal
Better choice
Compress correlated features
PCA
Explain the model with original features
Selection
Reduce the cost of collecting features
Selection
Remove noise in a block of sensors
PCA
Very high-dimensional sparse text
Selection (chi²) or TruncatedSVD
Common Pitfalls
Calling PCA “feature selection” and claiming it identifies important features.
Running selection on the full data before cross-validation (leakage).
Comparing methods at different numbers of features without saying so.
Interview Tips
Distinguish extraction from selection and supervised from unsupervised.
Name filter, wrapper and embedded methods with examples.
37
How does Incremental PCA differ from standard PCA?
KnownRevise
Standard PCA computes an SVD of the whole centred dataset at once, so the full matrix must fit in memory; Incremental PCA (sklearn.decomposition.IncrementalPCA) processes the data in mini-batches with partial_fit, updating the mean, the components and the singular values after each batch, so memory depends on the batch size and the number of components rather than on the number of rows. The final components closely match standard PCA (exactly for the explained variance if all data is seen, up to small numerical differences), at the cost of more total computation and the need to choose a batch size at least as large as the number of components. Use it for datasets too large for memory (reading chunks from disk, memory-mapped arrays or a database), for data that arrives over time, or to update a model without refitting from scratch; for data that fits in memory, standard or randomized PCA is simpler and faster.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Comparison
PCA
IncrementalPCA
Data access
All at once
Mini-batches (partial_fit)
Memory
O(n·d)
O(batch·d + k·d)
Exactness
Exact (or randomized)
Very close to exact
New data
Refit
Continue partial_fit
Batch size
—
≥ n_components
Typical use
In-memory data
Out-of-core, streaming
Code Example
import numpy as np
from sklearn.decomposition import PCA, IncrementalPCA
rng = np.random.default_rng(42)
n, d, k = 20_000, 50, 5
X = rng.normal(size=(n, 10)) @ rng.normal(size=(10, d)) + rng.normal(0, 0.1, (n, d))
full = PCA(n_components=k).fit(X)
ipca = IncrementalPCA(n_components=k)
for chunk in np.array_split(X, 40): # 40 batches of 500 rows, e.g. read from disk
ipca.partial_fit(chunk)
print("variance ratios, PCA: ", full.explained_variance_ratio_.round(4))
print("variance ratios, IPCA: ", ipca.explained_variance_ratio_.round(4))
cos = np.abs(np.sum(full.components_ * ipca.components_, axis=1))
print("|cosine| between components:", cos.round(4)) # close to 1 for well-separated components
path = "big_matrix.npy" # out-of-core pattern with a memory-mapped file
np.save(path, X)
X_mm = np.load(path, mmap_mode="r")
ipca2 = IncrementalPCA(n_components=k, batch_size=1000).fit(X_mm)
print("memory-mapped fit matches:", np.allclose(ipca2.explained_variance_ratio_, ipca.explained_variance_ratio_, atol=1e-3))
Common Pitfalls
Batches smaller than n_components (scikit-learn raises an error).
Feeding batches that are each from one class or time period, which can bias intermediate results; shuffle when possible.
Using IncrementalPCA when data fits in memory (slower, no benefit).
Forgetting that scaling must also be done incrementally (StandardScaler.partial_fit).
Interview Tips
Contrast whole-data SVD with mini-batch updates and memory use.
Name out-of-core and streaming use cases.
38
How is randomized PCA used and what are its benefits?
KnownRevise
Randomized PCA approximates the top k principal components without a full SVD: it multiplies the centred data matrix by a random Gaussian matrix with k plus a few extra columns, so the result captures the range of the dominant directions; a few power iterations sharpen it when the spectrum decays slowly, an orthonormal basis Q of that sketch is computed, and an exact SVD of the small matrix QᵀX gives the components. Its cost is roughly O(n·d·k) instead of O(n·d·min(n, d)), it needs only a few passes over the data, it parallelizes as matrix multiplications, and its error is close to the best rank-k approximation with high probability, which is why scikit-learn’s PCA(svd_solver="randomized") (chosen automatically for large inputs when k is small) and TruncatedSVD use it. Use it when you want a few components from a large matrix; when you need many components or exact values, use the full solver.
Read full answerHide full answer
Applies toNumPy 2.x, scikit-learn 1.5+
Algorithm (Halko, Martinsson and Tropp)
Step
Operation
1
Draw a random matrix Ω (d × (k + p)), p ≈ 5–10 oversampling
2
Y = XcΩ; optionally power iterations Y = (XcXcᵀ)^q XcΩ
3
Q = orthonormal basis of Y (QR decomposition)
4
B = QᵀXc (small (k + p) × d matrix)
5
SVD of B = ŨΣVᵀ; components are the top k rows of Vᵀ
Code Example
import time
import numpy as np
from sklearn.decomposition import PCA
def randomized_pca(X, k, oversample=10, power_iters=4, seed=42):
rng = np.random.default_rng(seed)
Xc = X - X.mean(axis=0)
omega = rng.normal(size=(X.shape[1], k + oversample))
Y = Xc @ omega
for _ in range(power_iters):
Y, _ = np.linalg.qr(Y) # re-orthonormalize for stability
Y = Xc @ (Xc.T @ Y)
Q, _ = np.linalg.qr(Y)
B = Q.T @ Xc
_, s, Vt = np.linalg.svd(B, full_matrices=False)
return Vt[:k], s[:k] ** 2 / (len(X) - 1)
rng = np.random.default_rng(42)
n, d = 5000, 2000
spectrum = 1 / np.arange(1, d + 1) ** 0.7 # slowly decaying singular values
U, _ = np.linalg.qr(rng.normal(size=(n, 200)))
V, _ = np.linalg.qr(rng.normal(size=(d, 200)))
X = (U * spectrum[:200] * 100) @ V.T + rng.normal(0, 0.01, (n, d))
start = time.perf_counter()
exact = PCA(n_components=10, svd_solver="full").fit(X)
t_full = time.perf_counter() - start
start = time.perf_counter()
comps, variances = randomized_pca(X, 10)
t_rand = time.perf_counter() - start
print(f"full SVD {t_full:.2f}s, randomized {t_rand:.2f}s")
print("relative error of variances:", np.abs(variances / exact.explained_variance_ - 1).max().round(5))
print("min |cosine| with exact components:", np.abs(np.sum(comps * exact.components_, axis=1)).min().round(4))
The randomized version is typically several times faster here and matches the top variances to well under 1%.
Benefits and Trade-offs
Benefit
Trade-off
Much faster for small k
Approximate (tiny, controllable error)
Few passes over the data
Slow spectral decay needs power iterations
Parallel, GPU-friendly
Random seed affects tiny details
Works with sparse matrices (TruncatedSVD)
Not for computing all components
Common Pitfalls
Skipping oversampling or power iterations on data with a flat spectrum, giving poor accuracy.
Forgetting to fix random_state for reproducibility.
Using it for k close to min(n, d), where it has no advantage.
Interview Tips
Describe the random projection → QR → small SVD steps.
Mention O(ndk) cost, power iterations and scikit-learn’s solver choice.
39
What recent advances have been made in PCA for big data?
KnownRevise
PCA for big data has moved from exact SVDs on one machine to approximate, streaming, distributed and hardware-accelerated methods: randomized SVD and sketching (random projections, CountSketch, frequent directions) compute top components in a few passes with provable error bounds; streaming algorithms such as Oja’s rule, incremental SVD and frequent directions update components one record or batch at a time with bounded memory; distributed implementations in Spark MLlib, Dask and Ray split the work as map-reduce of sufficient statistics or parallel randomized SVD; and GPU libraries (RAPIDS cuML, PyTorch torch.pca_lowrank) make the dense matrix products very fast. Research has also strengthened the method itself: robust PCA (low-rank plus sparse decomposition) handles corrupted data, sparse PCA gives interpretable components with few non-zero loadings, federated and differentially private PCA learn components without centralizing raw data, and random matrix theory gives principled thresholds for separating signal from noise components in high dimensions.
Read full answerHide full answer
Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+
Advances at a Glance
Area
Methods
Benefit
Approximation
Randomized SVD, sketching
O(ndk) cost, few passes
Streaming
Oja’s rule, frequent directions, IncrementalPCA
Bounded memory, online updates
Distributed
Spark MLlib, Dask-ML, TSQR
Scale across clusters
Hardware
GPU PCA (cuML, PyTorch)
Large speedups for dense data
Robustness
Principal Component Pursuit, ROBPCA
Outliers and corruption
Interpretability
Sparse PCA, structured sparsity
Few features per component
Privacy
Federated PCA, differentially private PCA
No raw data sharing
High-dimensional theory
Marchenko-Pastur thresholds, spiked models
Choose k reliably
Code Example: Frequent Directions Sketch
import numpy as np
from sklearn.decomposition import PCA
def frequent_directions(rows, ell):
"""Streaming sketch B (ell x d) with ||X^T X - B^T B|| bounded by ||X||_F^2 / ell (Liberty, 2013)."""
d = rows.shape[1]
B = np.zeros((2 * ell, d))
filled = 0
for row in rows:
if filled == 2 * ell: # sketch full: shrink
_, s, Vt = np.linalg.svd(B, full_matrices=False)
s = np.sqrt(np.maximum(s**2 - s[ell] ** 2, 0))
B = np.zeros((2 * ell, d))
B[:ell] = s[:ell, None] * Vt[:ell]
filled = ell
B[filled] = row
filled += 1
return B
rng = np.random.default_rng(42)
X = rng.normal(size=(20_000, 8)) @ rng.normal(size=(8, 200)) + rng.normal(0, 0.1, (20_000, 200))
Xc = X - X.mean(axis=0)
sketch = frequent_directions(Xc, ell=20) # 40 x 200 sketch from one pass
_, _, Vt_sketch = np.linalg.svd(sketch, full_matrices=False)
exact = PCA(n_components=8).fit(X)
cos = np.abs(np.sum(Vt_sketch[:8] * exact.components_, axis=1))
print("|cosine| of sketch components with exact PCA:", cos.round(3)) # near 1 for the 8 signal directions
print(f"sketch rows {sketch.shape[0]} vs data rows {X.shape[0]}")
Centring here uses the full-data mean for simplicity; a real stream keeps a running mean and corrects for it.
Common Pitfalls
Choosing an approximate method without checking its error on a sample.
Ignoring the communication cost in distributed settings.
Assuming privacy-preserving or federated versions give identical results to centralized PCA.
Interview Tips
Group advances into approximation, streaming, distribution, hardware, robustness and privacy.
Explain one method (randomized SVD or frequent directions) in detail.
40
How does robust PCA handle outliers, and what are its uses?
KnownRevise
Robust PCA in its best-known form, Principal Component Pursuit (Candès, Li, Ma and Wright, 2011), assumes the data matrix is the sum of a low-rank matrix L (the clean structure) and a sparse matrix S (a few large errors or outliers), M = L + S, and recovers both by solving the convex problem minimize ‖L‖* + λ‖S‖₁ subject to L + S = M, where the nuclear norm ‖L‖* encourages low rank and the ℓ₁ norm encourages sparsity, typically with λ = 1/√max(n, d). Unlike standard PCA, where one corrupted value can rotate the components, it isolates arbitrarily large but sparse corruptions in S and keeps them out of L. Other robust approaches downweight whole outlying rows instead: PCA on a robust covariance (Minimum Covariance Determinant), ROBPCA, or M-estimators. Practical uses include separating the static background (L) from moving objects (S) in surveillance video, removing shadows and specular highlights from face images, cleaning sensor and network-traffic data, and anomaly detection, where S itself is the list of anomalies. The cost is iterative SVDs, which are slower than one PCA, and the model needs the corruption to be sparse and the clean part to be genuinely low-rank.
Read full answerHide full answer
Applies toNumPy 2.x
Code Example: Principal Component Pursuit with ADMM
import numpy as np
def shrink(X, tau):
return np.sign(X) * np.maximum(np.abs(X) - tau, 0)
def svd_shrink(X, tau):
U, s, Vt = np.linalg.svd(X, full_matrices=False)
return (U * shrink(s, tau)) @ Vt
def robust_pca(M, max_iter=500, tol=1e-7):
n, d = M.shape
lam = 1 / np.sqrt(max(n, d))
mu = n * d / (4 * np.abs(M).sum())
L, S, Y = np.zeros_like(M), np.zeros_like(M), np.zeros_like(M)
for _ in range(max_iter):
L = svd_shrink(M - S + Y / mu, 1 / mu) # low-rank update
S = shrink(M - L + Y / mu, lam / mu) # sparse update
residual = M - L - S
Y += mu * residual
if np.linalg.norm(residual) / np.linalg.norm(M) < tol:
break
return L, S
rng = np.random.default_rng(42)
n, d, rank = 200, 100, 3
L_true = rng.normal(size=(n, rank)) @ rng.normal(size=(rank, d))
S_true = np.zeros((n, d))
corrupt = rng.random((n, d)) < 0.05 # 5% of entries grossly corrupted
S_true[corrupt] = rng.uniform(-20, 20, corrupt.sum())
M = L_true + S_true
L_hat, S_hat = robust_pca(M)
print(f"relative error of L: {np.linalg.norm(L_hat - L_true) / np.linalg.norm(L_true):.2e}") # tiny
print("rank of recovered L:", np.linalg.matrix_rank(L_hat, tol=1e-3)) # 3
print(f"corrupted entries found: {np.mean((np.abs(S_hat) > 1e-3) == corrupt):.4f} agreement")
U, s, Vt = np.linalg.svd(M - M.mean(axis=0), full_matrices=False) # standard PCA for comparison
L_pca = (U[:, :rank] * s[:rank]) @ Vt[:rank] + M.mean(axis=0)
print(f"standard rank-3 PCA error: {np.linalg.norm(L_pca - L_true) / np.linalg.norm(L_true):.2f}") # much larger
Approaches
Method
Outlier model
Notes
Principal Component Pursuit
Sparse corrupted entries
Convex; exact recovery under conditions
MCD-based PCA
Whole outlying rows
Robust covariance, moderate d
ROBPCA
Whole rows, high d
Projection pursuit + MCD
Outlier pursuit
Corrupted columns
Column-sparse S
Common Pitfalls
Applying PCP when corruption is dense rather than sparse.
Using the default λ blindly; tune it when the outlier share is high.
Expecting robust PCA to be as fast as standard PCA.
Interview Tips
Write M = L + S and the nuclear norm plus ℓ₁ objective.
Give video background subtraction and anomaly detection as uses.
41
How does Generalized PCA differ from standard PCA?
KnownRevise
“Generalized PCA” names two related extensions of PCA, and a good answer distinguishes them: Generalized PCA for exponential-family data (Collins, Dasgupta and Schapire, 2001; also called exponential family PCA) replaces PCA’s squared-error loss, which implicitly assumes Gaussian noise, with the likelihood of a distribution that fits the data (Bernoulli for binary, Poisson for counts, multinomial for categories), fitting a low-rank matrix of natural parameters instead of the data itself; and Generalized PCA in the subspace-clustering sense (Vidal, Ma and Sastry, 2005) fits data lying on a union of several different subspaces, rather than one, by representing them as the roots of polynomials. Standard PCA is the special case of the first with Gaussian noise and of the second with one subspace. Applications include binary survey or click data (logistic PCA), word or gene counts (Poisson PCA), and motion segmentation and face clustering under varying lighting (subspace clustering); the cost is iterative, non-convex fitting instead of a single SVD.
Read full answerHide full answer
Applies togeneral concepts; example with NumPy 2.x
Comparison
Standard PCA
Exponential-family GPCA
Subspace-clustering GPCA
Data model
One subspace + Gaussian noise
Low-rank natural parameters, any exponential family
Union of several subspaces
Loss
Squared error
Negative log-likelihood (e.g. logistic, Poisson)
Algebraic fit of polynomials
Solution
SVD (closed form)
Iterative optimization
Polynomial fitting + clustering
Typical data
Continuous
Binary, counts, categories
Mixed motions, faces under lighting
Code Example: Logistic PCA vs Standard PCA on Binary Data
import numpy as np
rng = np.random.default_rng(42)
n, d, k = 500, 40, 2
U_true = rng.normal(size=(n, k)) * 2
V_true = rng.normal(size=(d, k))
theta_true = U_true @ V_true.T # natural parameters (log-odds)
X = (rng.random((n, d)) < 1 / (1 + np.exp(-theta_true))).astype(float)
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def logistic_pca(X, k, lr=0.05, iters=1500, seed=42):
r = np.random.default_rng(seed)
U = r.normal(0, 0.1, (X.shape[0], k))
V = r.normal(0, 0.1, (X.shape[1], k))
b = np.zeros(X.shape[1])
for _ in range(iters): # gradient descent on the Bernoulli log-likelihood
P = sigmoid(U @ V.T + b)
G = P - X
U -= lr * (G @ V) / X.shape[1]
V -= lr * (G.T @ U) / X.shape[0]
b -= lr * G.mean(axis=0)
return sigmoid(U @ V.T + b)
P_logistic = logistic_pca(X, k)
Xc = X - X.mean(axis=0)
Us, s, Vt = np.linalg.svd(Xc, full_matrices=False)
raw_pca = (Us[:, :k] * s[:k]) @ Vt[:k] + X.mean(axis=0)
print(f"standard PCA reconstructions outside [0, 1]: {np.mean((raw_pca < 0) | (raw_pca > 1)):.1%}")
P_pca = np.clip(raw_pca, 1e-6, 1 - 1e-6)
P_true = sigmoid(theta_true)
for name, P in [("standard PCA", P_pca), ("logistic PCA", P_logistic)]:
print(f"{name:13s}: mean abs error of probabilities {np.mean(np.abs(P - P_true)):.3f}")
# Logistic PCA's fitted values are proper probabilities; standard PCA's reconstructions are not
# limited to [0, 1] and usually fit the true probabilities less well
Common Pitfalls
Using standard PCA on binary or count data and reading reconstructions as probabilities or counts.
Confusing the two meanings of “Generalized PCA” in a discussion.
Expecting closed-form, unique solutions; these methods need good initialization.
Interview Tips
Explain PCA as Gaussian-likelihood fitting and the exponential-family generalization.
Mention subspace clustering as the other meaning and give one application of each.
42
Implement PCA in PyTorch and compare it with scikit-learn
KnownRevise
PCA in PyTorch follows the same steps as in NumPy: centre the data tensor, compute torch.linalg.svd(Xc, full_matrices=False) (or the faster approximate torch.pca_lowrank for a few components), take the top k rows of Vᴴ as components and the squared singular values divided by n − 1 as explained variances, then project with Xc @ Vₖᵀ; running it on a GPU makes large dense problems much faster. To compare with scikit-learn fairly, use float64 (or loosen tolerances for float32), compare explained variances directly, and compare components and projections only up to sign, because each singular vector is defined up to multiplication by −1. Small differences around 1e-6 in float32 are expected and are not bugs.
Read full answerHide full answer
Applies toPyTorch 2.x, scikit-learn 1.5+
Code Example
import numpy as np
import torch
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
class TorchPCA:
def __init__(self, n_components, device="cpu", dtype=torch.float64):
self.k, self.device, self.dtype = n_components, device, dtype
def fit(self, X):
X = torch.as_tensor(X, dtype=self.dtype, device=self.device)
self.mean_ = X.mean(dim=0)
U, S, Vh = torch.linalg.svd(X - self.mean_, full_matrices=False)
self.components_ = Vh[: self.k]
self.explained_variance_ = S[: self.k] ** 2 / (X.shape[0] - 1)
self.explained_variance_ratio_ = S[: self.k] ** 2 / (S ** 2).sum()
return self
def transform(self, X):
X = torch.as_tensor(X, dtype=self.dtype, device=self.device)
return (X - self.mean_) @ self.components_.T
device = "cuda" if torch.cuda.is_available() else "cpu"
X = load_digits().data
k = 10
tp = TorchPCA(k, device=device).fit(X)
sk = PCA(n_components=k, svd_solver="full").fit(X)
comps = tp.components_.cpu().numpy()
signs = np.sign(np.sum(comps * sk.components_, axis=1))
print("variances match:", np.allclose(tp.explained_variance_.cpu().numpy(), sk.explained_variance_)) # True
print("components match up to sign:", np.allclose(comps * signs[:, None], sk.components_, atol=1e-8)) # True
print("projections match:", np.allclose(tp.transform(X).cpu().numpy() * signs, sk.transform(X), atol=1e-8)) # True
torch.manual_seed(42)
Xt = torch.as_tensor(X, dtype=torch.float32, device=device)
U, S, V = torch.pca_lowrank(Xt, q=k, center=True, niter=4) # randomized, float32
approx_var = (S ** 2 / (Xt.shape[0] - 1)).cpu().numpy()
print("pca_lowrank relative variance error:", np.abs(approx_var / sk.explained_variance_ - 1).max().round(5))
Notes
Topic
Detail
Sign ambiguity
Align signs before comparing
Precision
float64 for exact checks; float32 with atol around 1e-4
torch.pca_lowrank
Randomized; returns V (not Vᴴ), center=True by default
GPU
Move tensors with device="cuda"; big speedups for large dense X
Autograd
SVD is differentiable, but gradients are unstable for repeated singular values
Common Pitfalls
Forgetting to centre (torch.svd_lowrank doesn’t centre; pca_lowrank does by default).
Comparing float32 GPU results with float64 scikit-learn results using tight tolerances.
Using the deprecated torch.svd instead of torch.linalg.svd.
Interview Tips
Show centre → SVD → components → projection with tensors.
Mention sign alignment, precision and pca_lowrank.
43
How do you choose the number of PCA dimensions with the elbow method?
KnownRevise
The elbow method plots the explained variance (or eigenvalue) of each principal component in order, a scree plot, and picks the number of components at the “elbow”, where the curve bends from a steep drop to a flat tail, because components after that point add little information each. Doing it in Python means fitting PCA with all components on standardized data, plotting explained_variance_ratio_ with its cumulative sum, and either reading the elbow by eye or detecting it automatically, for example by finding the point farthest from the straight line joining the first and last points of the curve (the “knee” or kneedle idea). Because elbows are often vague, confirm the choice with a second criterion such as cumulative variance (for example 90%) or downstream cross-validation.
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n, d, true_rank = 1000, 30, 5
latent = rng.normal(size=(n, true_rank)) * np.array([5, 4, 3, 2.5, 2])
X = latent @ rng.normal(size=(true_rank, d)) + rng.normal(0, 1.0, (n, d))
Xs = StandardScaler().fit_transform(X)
ratio = PCA().fit(Xs).explained_variance_ratio_
components = np.arange(1, d + 1)
def elbow_index(y):
"""Point farthest from the line joining the first and last points (both axes scaled to 0..1)."""
x = np.linspace(0, 1, len(y))
y_scaled = (y - y.min()) / (y.max() - y.min())
start, end = np.array([x[0], y_scaled[0]]), np.array([x[-1], y_scaled[-1]])
direction = (end - start) / np.linalg.norm(end - start)
points = np.column_stack([x, y_scaled]) - start
distances = np.abs(points[:, 0] * direction[1] - points[:, 1] * direction[0])
return int(np.argmax(distances))
k_elbow = elbow_index(ratio) + 1
k_90 = int(np.searchsorted(np.cumsum(ratio), 0.90) + 1)
print(f"elbow at k = {k_elbow}, 90% variance at k = {k_90}") # compare with the true rank 5; the rules can differ by a component or two
fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(components, ratio, marker="o", label="explained variance ratio")
ax.plot(components, np.cumsum(ratio), marker=".", label="cumulative")
ax.axvline(k_elbow, color="black", linestyle="--", label=f"elbow (k={k_elbow})")
ax.set(xlabel="number of components", ylabel="variance ratio", title="Scree plot and elbow")
ax.legend()
fig.tight_layout()
fig.savefig("pca_elbow.png", dpi=120)
The distance-to-chord rule can land one component before or after where the eye would put the elbow, so treat it as a suggestion and check it against the plot.
Common Pitfalls
Relying on the elbow when the curve decays smoothly with no clear bend.
Detecting the elbow on the cumulative curve with a method meant for the per-component curve (or the reverse) without care.
Choosing k by the elbow for a predictive model without validating it.
Interview Tips
Explain the scree plot and what the bend means.
Show an automatic detection rule and a second criterion to confirm it.
44
Write a script to remove outliers automatically before PCA
KnownRevise
A robust automatic approach flags outliers with a method that the outliers themselves can’t distort, removes (or sets aside) them, and only then fits PCA: standardize with the median and IQR (RobustScaler), estimate a robust covariance with the Minimum Covariance Determinant (MinCovDet), and flag points whose robust Mahalanobis distance exceeds a chi-squared cut-off such as the 97.5% quantile with d degrees of freedom; alternatives are IsolationForest or LocalOutlierFactor for non-Gaussian data, or a two-pass check of PCA reconstruction error. The script should report how many rows were removed and why, keep them for review instead of silently deleting them (they may be real rare events), cap the removed share, and apply the same fitted detector to new data inside the pipeline, so preprocessing is identical in training and production.
import numpy as np
from scipy import stats
from sklearn.covariance import MinCovDet
from sklearn.decomposition import PCA
from sklearn.preprocessing import RobustScaler
def remove_outliers_robust(X, quantile=0.975, max_fraction=0.1, random_state=42):
"""Return (X_clean, mask_kept, distances) using robust Mahalanobis distances."""
Xs = RobustScaler().fit_transform(X)
mcd = MinCovDet(random_state=random_state).fit(Xs)
d2 = mcd.mahalanobis(Xs) # squared robust distances
cutoff = stats.chi2.ppf(quantile, df=X.shape[1])
keep = d2 <= cutoff
if (~keep).mean() > max_fraction: # never drop more than max_fraction
keep = d2 <= np.quantile(d2, 1 - max_fraction)
return X[keep], keep, d2
rng = np.random.default_rng(42)
n, d = 500, 6
X = rng.normal(size=(n, 2)) @ rng.normal(size=(2, d)) + rng.normal(0, 0.3, (n, d))
outlier_rows = rng.choice(n, 15, replace=False)
X[outlier_rows] += rng.normal(0, 8, (15, d)) # 3% gross outliers
X_clean, keep, d2 = remove_outliers_robust(X)
removed = np.where(~keep)[0]
print(f"removed {len(removed)} rows; true outliers caught: {np.isin(outlier_rows, removed).sum()} of 15")
pca_dirty = PCA(n_components=2).fit(X)
pca_clean = PCA(n_components=2).fit(X_clean)
print("variance in 2 PCs: with outliers", pca_dirty.explained_variance_ratio_.sum().round(3),
"after removal", pca_clean.explained_variance_ratio_.sum().round(3))
# After removal the two real components explain most of the variance again
review = np.column_stack([removed, d2[removed].round(1)]) # keep for human review, don't discard silently
print("rows set aside (index, squared distance):\n", review[:5])
The chi-squared cut-off assumes the clean data are roughly Gaussian; with a 97.5% cut-off about 2.5% of clean points are flagged as well, so a few false positives are expected.
Method Options
Method
Works when
Notes
MCD + chi-squared cut-off
Roughly elliptical data, moderate d
Classic, interpretable
IsolationForest
Any shape, high d
Needs a contamination setting
LocalOutlierFactor
Local density differences
Needs novelty=True for new data
PCA reconstruction error
Low-rank data
Use a robust first fit
Per-feature IQR rule
Simple univariate checks
Misses multivariate outliers
Common Pitfalls
Detecting outliers with a standard (non-robust) PCA or mean/covariance, which the outliers distort (masking).
Removing too much data with a strict cut-off.
Deleting rows silently that may be valid rare events.
Fitting the detector on the full dataset including the test set.
Interview Tips
Use robust distances with a chi-squared cut-off and explain masking.
Mention review instead of silent deletion and caps on removal.
45
Show the effect of PCA on classification accuracy with a synthetic dataset
KnownRevise
A clear experiment creates a synthetic classification dataset with a known structure, a few informative features plus many redundant and noisy ones, then compares a classifier’s cross-validated accuracy and training time with no PCA against PCA with an increasing number of components, using a Pipeline so PCA is fitted inside each fold. The usual result is a curve: too few components lose class information and accuracy drops, a moderate number matches or beats the full feature set because PCA removes noise and redundancy (especially for kNN and SVMs, less for linear models with regularization), and many components bring back the noise. The experiment is honest only if the comparison includes the baseline and uses the same folds; a second synthetic case, where the class signal hides in a low-variance direction, shows PCA hurting.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example
import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=2000, n_features=100, n_informative=8, n_redundant=12,
n_repeated=0, n_classes=3, n_clusters_per_class=1,
class_sep=1.5, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
knn = KNeighborsClassifier(n_neighbors=15)
def evaluate(model):
start = time.perf_counter()
score = cross_val_score(model, X, y, cv=cv).mean()
return score, time.perf_counter() - start
base, base_t = evaluate(make_pipeline(StandardScaler(), knn))
print(f"no PCA (100 features): accuracy {base:.3f}, time {base_t:.2f}s")
for k in [2, 5, 10, 20, 40, 80]:
score, t = evaluate(make_pipeline(StandardScaler(), PCA(n_components=k), knn))
print(f"PCA k={k:>2}: accuracy {score:.3f}, time {t:.2f}s")
# Accuracy usually peaks at a moderate k above the no-PCA baseline: the 80 noise features hurt kNN distances
rng = np.random.default_rng(42)
y2 = rng.integers(0, 2, 2000)
X2 = np.column_stack([rng.normal(0, 5, (2000, 20)), (y2 - 0.5) + rng.normal(0, 0.2, 2000)])
logreg = LogisticRegression(max_iter=1000)
for name, model in [("no PCA", make_pipeline(logreg)), ("PCA k=5", make_pipeline(PCA(n_components=5), logreg))]:
print(f"low-variance signal, {name}: {cross_val_score(model, X2, y2, cv=cv).mean():.3f}")
# No PCA: close to 1.0. PCA keeps 5 of the 20 high-variance noise directions, drops the signal and falls to about chance
What to Report
Item
Why
Baseline without PCA
The reference point
Accuracy vs k curve
Shows the trade-off
Variance kept at the best k
Links to explained variance
Training/prediction time
Practical benefit
A failure case
PCA is not always helpful
Common Pitfalls
Fitting PCA on all data before cross-validation.
Showing only the best k and not the curve or the baseline.
Generalizing from one synthetic setup to all data.
Interview Tips
Describe the experimental design (pipeline, same folds, baseline).
Explain why accuracy rises then falls, and show a case where PCA hurts.
46
How do you decide between PCA and a classification algorithm?
KnownRevise
The question compares two things that do different jobs: PCA is an unsupervised dimensionality reduction step that never predicts labels, while a classification algorithm (logistic regression, random forest, gradient boosting, SVM, a neural network) learns to predict a label, so if the goal is to assign classes you always need a classifier, and the real decision is whether to put PCA in front of it. Use PCA before the classifier when there are many correlated or noisy features, when distance-based or linear models struggle with dimensionality, when training speed or memory is a constraint, or when you want a quick 2-D view of class structure; skip it when features are few or already informative, when you use tree ensembles, when predictions must be explained in terms of original features, or when the signal is in low-variance directions. Decide with evidence: cross-validate the classifier with and without PCA (tuning k), and if dimensionality reduction should use the labels, try LDA or supervised feature selection instead.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Decision Guide
Situation
Recommendation
Goal is prediction of a label
A classifier (always)
Many correlated features, kNN/SVM/linear model
Try PCA + classifier
Tree ensembles
Usually no PCA
Need explanations per original feature
Avoid PCA or use selection
Exploring structure, no labels
PCA alone (plus clustering)
Labels available for reduction
LDA, PLS, supervised selection
Very wide sparse data (text)
TruncatedSVD or none
Code Example: Let Cross-Validation Decide
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_digits(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()), ("reduce", "passthrough"), ("clf", SVC())])
grid = [
{"reduce": ["passthrough"], "clf": [SVC(), RandomForestClassifier(n_estimators=200, random_state=42)]},
{"reduce": [PCA(n_components=20), PCA(n_components=40)],
"clf": [SVC(), RandomForestClassifier(n_estimators=200, random_state=42)]},
{"reduce": [LinearDiscriminantAnalysis(n_components=9)], "clf": [SVC()]},
]
search = GridSearchCV(pipe, grid, cv=5, n_jobs=-1).fit(X, y)
results = sorted(zip(search.cv_results_["mean_test_score"], search.cv_results_["params"]),
key=lambda r: r[0], reverse=True)
for score, params in results:
print(f"{score:.3f} reduce={params['reduce']}, clf={type(params['clf']).__name__}")
The ranking shows whether PCA helps for each classifier on this data; on digits the SVM does about as well with and without PCA, and the forest gains little from it.
Common Pitfalls
Treating PCA as a replacement for a classifier.
Adding PCA by habit without comparing.
Evaluating with leakage (PCA fitted outside cross-validation).
Interview Tips
Clarify that PCA and classifiers do different jobs.
Give the cases for and against PCA and decide by cross-validation.
47
Describe a case where PCA reduced overfitting
KnownRevise
A typical case is a model with many correlated, noisy features and few training samples, such as predicting a disease outcome from 2,000 gene-expression measurements on 150 patients, a quality score from hundreds of near-infrared spectral wavelengths, or a failure from 300 correlated sensor readings: an unregularized or lightly regularized model fits noise in hundreds of directions, so training accuracy is near perfect while validation accuracy is poor. Projecting onto the top 10–30 principal components, chosen by cross-validation, keeps the shared structure (biological pathways, chemical composition, machine state) and discards most noise dimensions, so the model has far fewer effective parameters, the gap between training and validation scores shrinks, and validation accuracy improves; this is principal component regression, a close cousin of ridge regression, which shrinks low-variance directions instead of cutting them off. The proof is in the numbers: report training and validation scores before and after, chosen inside cross-validation.
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: Spectral-Style Data with More Features than Samples
import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n, d, latent_dim = 120, 400, 4
wavelengths = np.linspace(0, 1, d)
peaks = np.vstack([np.exp(-((wavelengths - c) ** 2) / 0.005) for c in [0.2, 0.4, 0.6, 0.8]])
concentrations = rng.uniform(0, 1, (n, latent_dim))
X = concentrations @ peaks + rng.normal(0, 0.05, (n, d)) # smooth, highly correlated spectra
y = concentrations @ np.array([3.0, -2.0, 1.5, 0.5]) + rng.normal(0, 0.1, n)
models = {
"linear regression, 400 features": make_pipeline(StandardScaler(), LinearRegression()),
"PCA (8) + linear regression": make_pipeline(StandardScaler(), PCA(n_components=8), LinearRegression()),
"ridge (alpha=10)": make_pipeline(StandardScaler(), Ridge(alpha=10)),
}
for name, model in models.items():
res = cross_validate(model, X, y, cv=5, scoring="r2", return_train_score=True)
print(f"{name:32s}: train R2 {res['train_score'].mean():.3f}, validation R2 {res['test_score'].mean():.3f}")
# Plain regression fits the training data perfectly (more features than rows), so its train R2 says nothing;
# the PCA and ridge versions keep train and validation scores close, which is the sign of less overfitting
Why It Works
Effect
Explanation
Fewer parameters
8 coefficients instead of 400
Noise removal
Discarded components are mostly noise
Collinearity removed
Components are uncorrelated
Relation to ridge
Ridge shrinks small-variance directions smoothly; PCR drops them
Common Pitfalls
Picking k with the test set.
Assuming PCA always reduces overfitting; it fails when the signal lives in low-variance directions (partial least squares helps there).
Not comparing with simpler regularization like ridge or Lasso.
Interview Tips
Describe the high-dimensional, correlated, small-sample setting.
Show the train/validation gap closing and mention ridge as an alternative.
48
Give an example of PCA applied incorrectly and how to fix it
KnownRevise
A common mistake is to fit the scaler and PCA on the entire dataset, then split it into training and test sets and report the test accuracy: the components were learned using the test rows, so information leaks into training and the score is optimistic, which matters most with small datasets, high dimensions, or when the test data comes from a later time. The fix is to put StandardScaler and PCA inside a scikit-learn Pipeline and fit it only on training data (or within each cross-validation fold), then transform the test set with the already-fitted pipeline. Other frequent misuses, each with a fix: running PCA on unscaled features with different units (standardize first), applying it to categorical codes (use MCA or encoding), choosing components by variance when prediction is the goal (tune k by cross-validation), and interpreting the sign of a component as meaningful (only relative signs within a component matter).
Read full answerHide full answer
Applies toscikit-learn 1.5+
Code Example: The Leak and the Fix
import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
n, d = 60, 2000 # few samples, many features, pure noise
X = rng.normal(size=(n, d))
y = rng.integers(0, 2, n) # labels are random: true accuracy is 50%
# Wrong: unsupervised steps fitted on all rows, then feature selection by correlation with y on all rows
Xs_all = StandardScaler().fit_transform(X)
corr = np.abs((Xs_all * (y - y.mean())[:, None]).mean(axis=0))
top = np.argsort(corr)[-20:] # uses the labels of the test rows too
Z_all = PCA(n_components=5).fit_transform(Xs_all[:, top])
Z_tr, Z_te, y_tr, y_te = train_test_split(Z_all, y, test_size=0.3, random_state=42, stratify=y)
print(f"leaky test accuracy: {LogisticRegression().fit(Z_tr, y_tr).score(Z_te, y_te):.2f}") # far above 0.5
# Right: every fitted step lives inside the pipeline, fitted per fold
pipe = make_pipeline(StandardScaler(), PCA(n_components=5), LogisticRegression())
print(f"honest CV accuracy: {cross_val_score(pipe, X, y, cv=5).mean():.2f}") # about 0.5, as it should be
The example adds a label-based selection step to make the leak dramatic; PCA alone leaks less, because it doesn’t see labels, but fitting it on test rows still makes evaluation optimistic and is never correct practice.
Misuses and Fixes
Misuse
Fix
Fit scaler/PCA before splitting
Pipeline fitted on training folds only
No scaling with mixed units
StandardScaler (or RobustScaler) first
PCA on categorical codes
MCA/FAMD or proper encoding
k chosen by variance for a prediction task
Tune k with cross-validation
Interpreting the sign of a component
Interpret relative signs and magnitudes
PCA before tree ensembles by habit
Compare with and without
Outliers driving components
Robust scaling or robust PCA
Common Pitfalls
Re-fitting PCA on the test set separately, which gives different axes from training.
Applying a time-series PCA fitted on future data in a backtest.
Interview Tips
Lead with the leakage example and the Pipeline fix.
List two or three other misuses with their corrections.