Skip to content
Machine Learning & Data Science

PCA interview questions and answers

48 questions, 48 with code. About 2 h 36 min to read them all.

By TestLab · Updated

Plan for an interview
48 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Principal Component Analysis (PCA)?

Principal Component Analysis (PCA) is an unsupervised, linear technique that finds new axes, called principal components, along which the data varies the most: the first component is the direction of maximum variance, the second is the direction of maximum remaining variance at a right angle to the first, and so on, so that projecting the data onto the first few components keeps most of its information in far fewer dimensions. Mathematically, the components are the eigenvectors of the data’s covariance matrix (equivalently, the right singular vectors of the centred data matrix), and each eigenvalue is the variance captured by its component. The steps are: centre the data (and usually standardize it), compute the covariance matrix or the SVD, sort components by eigenvalue, keep the top k, and project. PCA is used for dimensionality reduction, visualization, noise removal, compression, and decorrelating features before modelling.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Steps

Step What happens
1. Standardize Subtract the mean of each feature; divide by its standard deviation if units differ
2. Covariance or SVD C = XᵀX / (n − 1), or the SVD X = UΣVᵀ
3. Eigen-decomposition Eigenvectors = components; eigenvalues = variance along them
4. Sort and choose k Keep the components with the largest eigenvalues
5. Project Z = X·W_k (n × k scores)

Code Example: From Scratch vs scikit-learn on Iris

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(load_iris().data)        # 150 x 4

Xc = X - X.mean(axis=0)
cov = Xc.T @ Xc / (len(Xc) - 1)
eigvals, eigvecs = np.linalg.eigh(cov)                       # ascending order for symmetric matrices
order = np.argsort(eigvals)[::-1]
eigvals, eigvecs = eigvals[order], eigvecs[:, order]
print("explained variance ratio:", (eigvals / eigvals.sum()).round(4))   # [0.7296 0.2285 0.0367 0.0052]

Z_mine = Xc @ eigvecs[:, :2]
pca = PCA(n_components=2).fit(X)
Z_sk = pca.transform(X)
signs = np.sign(np.sum(Z_mine * Z_sk, axis=0))               # components are defined up to sign
print("same projection up to sign:", np.allclose(Z_mine * signs, Z_sk))  # True
print("first two components keep", round(float(pca.explained_variance_ratio_.sum()), 4))   # 0.9581

Two components keep about 96% of the variance of the four standardized Iris features.

Properties

  • Components are orthogonal, and the projected scores are uncorrelated.

  • Components are ordered by the variance they explain.

  • Signs are arbitrary: a component and its negative are equally valid.

  • PCA is linear and sensitive to feature scale and outliers.

Common Pitfalls

  • Skipping standardization when features have different units.

  • Fitting PCA on all data before the train/test split (leakage); put it in a Pipeline.

  • Assuming the highest-variance directions are the most predictive for a target.

Interview Tips

  • Describe the steps and the eigenvector/eigenvalue meaning.

  • Mention scaling, SVD and the explained variance ratio.

2

How is PCA used for dimensionality reduction?

PCA reduces dimensionality by replacing the original d features with the first k principal components, the k orthogonal directions that capture the most variance, and representing each sample by its k projections (scores); because later components usually carry little variance, often mostly noise, dropping them loses little information while making the data smaller, less redundant and easier to model or plot. k is chosen from the cumulative explained variance ratio (for example the smallest k reaching 95%), a scree plot’s elbow, or downstream validation performance. In practice you fit PCA on the training data only, inside a Pipeline with scaling, then transform both training and test data; inverse_transform maps scores back to the original space and its reconstruction error shows how much was lost.

Read full answer

Applies toscikit-learn 1.5+

Code Example: Digits from 64 to Fewer Dimensions

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)                          # 1797 images, 64 pixel features
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

scaler = StandardScaler().fit(X_train)
pca = PCA(n_components=0.95).fit(scaler.transform(X_train))  # keep 95% of the variance
print(f"components kept: {pca.n_components_} of 64")         # about 40

Z_test = pca.transform(scaler.transform(X_test))
X_back = pca.inverse_transform(Z_test)
mse = np.mean((scaler.transform(X_test) - X_back) ** 2)
print(f"test reconstruction MSE (standardized units): {mse:.3f}")

for k in [10, 20, 40, None]:
    steps = [StandardScaler()] + ([PCA(n_components=k)] if k else []) + [LogisticRegression(max_iter=2000)]
    score = cross_val_score(make_pipeline(*steps), X_train, y_train, cv=5).mean()
    print(f"k={k or 'all 64':>6}: CV accuracy {score:.3f}")

Accuracy typically stays close to the full-feature model from around 20 to 40 components while the model trains on far fewer inputs.

Benefits and Costs

Benefit Cost
Fewer features, faster training Components are harder to interpret
Removes correlated redundancy Linear only
Filters some noise Variance is not the same as predictive value
2-D/3-D plots Needs scaling and is sensitive to outliers

Common Pitfalls

  • Fitting the scaler and PCA on the full dataset before splitting.

  • Choosing k by variance alone when downstream accuracy is what matters.

  • Applying PCA to sparse, high-dimensional text without care (centring makes it dense; use TruncatedSVD).

Interview Tips

  • Explain projection onto the top-k components and choosing k.

  • Mention pipelines, inverse_transform and TruncatedSVD for sparse data.

3

What are eigenvalues and eigenvectors in PCA?

An eigenvector of a square matrix A is a non-zero vector v whose direction doesn’t change when A is applied, only its length: Av = λv, where the scalar λ is the eigenvalue; in PCA the matrix is the covariance matrix of the centred data, its eigenvectors are the principal components (directions in feature space), and each eigenvalue equals the variance of the data along its eigenvector. Because the covariance matrix is symmetric and positive semi-definite, its eigenvalues are real and non-negative and its eigenvectors can be chosen orthonormal, so they form a new coordinate system. Sorting by eigenvalue ranks the directions by how much variance they explain, the explained variance ratio of component i is λᵢ / Σλ, and the sum of all eigenvalues equals the total variance (the trace of the covariance matrix).

Read full answer

Applies toNumPy 2.x

Worked Example (2 × 2)

Take the covariance matrix C = [[3, 1], [1, 3]].

  • Solve det(C − λI) = (3 − λ)² − 1 = 0, so λ = 4 or λ = 2.

  • For λ = 4: (C − 4I)v = 0 gives v = [1, 1]/√2.

  • For λ = 2: v = [1, −1]/√2.

  • The first component, along [1, 1], explains 4 / (4 + 2) = 66.7% of the variance, and the total variance 6 equals the trace 3 + 3.

Code Example

import numpy as np

C = np.array([[3.0, 1.0], [1.0, 3.0]])
vals, vecs = np.linalg.eigh(C)                     # symmetric: eigh, ascending eigenvalues
print(vals)                                        # [2. 4.]
v = vecs[:, 1]
print(np.allclose(C @ v, 4 * v), np.abs(v).round(4))   # True [0.7071 0.7071]

rng = np.random.default_rng(42)
X = rng.multivariate_normal([0, 0], C, size=50_000)
Xc = X - X.mean(axis=0)
sample_vals, sample_vecs = np.linalg.eigh(np.cov(Xc, rowvar=False))
print("sample eigenvalues:", sample_vals.round(2))   # close to [2, 4]

proj = Xc @ sample_vecs[:, ::-1]                       # project onto components, largest first
print("variance along each component:", proj.var(axis=0, ddof=1).round(2))   # matches the eigenvalues
print("correlation of the scores:", round(float(np.corrcoef(proj.T)[0, 1]), 4))   # about 0: uncorrelated
print("trace equals total variance:", np.isclose(np.trace(np.cov(Xc, rowvar=False)), sample_vals.sum()))   # True

Summary

Term Meaning in PCA
Eigenvector Principal component direction (a loading vector)
Eigenvalue Variance along that direction
λᵢ / Σλ Explained variance ratio
Σλ Total variance = trace of the covariance matrix
Orthogonality Components are perpendicular; scores are uncorrelated

Common Pitfalls

  • Using np.linalg.eig on a symmetric matrix and getting tiny complex parts or unsorted output; eigh is designed for it.

  • Forgetting that eigenvectors are defined up to sign.

  • Computing eigenvectors on uncentred data.

Interview Tips

  • Write Av = λv and work a 2 × 2 example.

  • Connect eigenvalues to variance and the trace.

4

What role does the covariance matrix play in PCA?

The covariance matrix summarizes how every pair of features varies together, C = XcᵀXc / (n − 1) for centred data Xc: its diagonal holds each feature’s variance and its off-diagonal entries the covariances, and PCA is exactly the eigen-decomposition of this matrix. The variance of the data projected onto any unit direction w is wᵀCw, so finding the direction of maximum variance means maximizing wᵀCw subject to ‖w‖ = 1, whose solution is the top eigenvector of C; the remaining eigenvectors give the next orthogonal directions, and rotating the data into this eigenbasis produces uncorrelated components (the covariance matrix of the scores is diagonal, holding the eigenvalues). Because C depends on units, standardized data is usually used, which makes C the correlation matrix.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

What the Matrix Tells You

Part Meaning
Cᵢᵢ Variance of feature i
Cᵢⱼ Covariance of features i and j (positive: rise together)
Large off-diagonal values Redundant, correlated features that PCA can combine
wᵀCw Variance of the data along direction w
Trace© Total variance

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(load_iris().data)
Xc = X - X.mean(axis=0)
C = Xc.T @ Xc / (len(X) - 1)
print(np.allclose(C, np.cov(X, rowvar=False)))              # True
print("correlation of petal length and petal width:", round(float(C[2, 3]), 3))   # about 0.97: very redundant

vals, vecs = np.linalg.eigh(C)
w = vecs[:, -1]                                              # top eigenvector
rng = np.random.default_rng(42)
random_dirs = rng.normal(size=(1000, 4))
random_dirs /= np.linalg.norm(random_dirs, axis=1, keepdims=True)
random_var = np.einsum("ij,jk,ik->i", random_dirs, C, random_dirs)
print(f"variance along top eigenvector {w @ C @ w:.3f}, best random direction {random_var.max():.3f}")
# No random direction beats the top eigenvector (its eigenvalue, about 2.94)

scores = PCA().fit_transform(X)
print(np.cov(scores, rowvar=False).round(3))                 # diagonal: components are uncorrelated

Standardized Iris has total variance 4 (one per feature, slightly more with ddof = 1), and the top eigenvalue of about 2.94 is about 73% of it.

Covariance vs Correlation

Use When
Covariance matrix (raw data) Features share units and their scale is meaningful
Correlation matrix (standardized data) Features have different units or ranges

Common Pitfalls

  • Using raw covariance with mixed units, so the feature with the largest numbers dominates.

  • Forgetting to centre before computing XᵀX.

  • Building a d × d covariance matrix when d is huge; use SVD or randomized methods instead.

Interview Tips

  • Show that projected variance is wᵀCw and why the top eigenvector maximizes it.

  • Mention diagonal covariance of the scores and covariance vs correlation.

5

What is the variance explained by a principal component?

The variance explained by a principal component is the variance of the data along that component’s direction, which equals its eigenvalue λᵢ of the covariance matrix; the explained variance ratio λᵢ / Σⱼ λⱼ is the share of the total variance that component captures, and the cumulative ratio over the first k components tells you how much of the data’s spread a k-dimensional projection keeps. In scikit-learn these are explained_variance_ and explained_variance_ratio_; from an SVD of the centred data they are sᵢ² / (n − 1) and sᵢ² / Σ s². They are used to choose k (a threshold such as 90–95%, or the elbow of a scree plot) and to judge how much structure is low-dimensional, keeping in mind that “variance explained” describes the inputs, not how well a target can be predicted.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Example (Standardized Iris)

Component Explained variance ratio Cumulative
PC1 0.7296 0.7296
PC2 0.2285 0.9581
PC3 0.0367 0.9948
PC4 0.0052 1.0000

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(load_iris().data)
pca = PCA().fit(X)
print("explained variance:", pca.explained_variance_.round(4))
print("ratio:", pca.explained_variance_ratio_.round(4))           # [0.7296 0.2285 0.0367 0.0052]
print("cumulative:", np.cumsum(pca.explained_variance_ratio_).round(4))

Xc = X - X.mean(axis=0)
s = np.linalg.svd(Xc, compute_uv=False)
print(np.allclose(s**2 / (len(X) - 1), pca.explained_variance_))   # True
print(np.allclose(s**2 / np.sum(s**2), pca.explained_variance_ratio_))   # True

scores = pca.transform(X)
print(np.allclose(scores.var(axis=0, ddof=1), pca.explained_variance_))   # True: variance of each score

k = int(np.searchsorted(np.cumsum(pca.explained_variance_ratio_), 0.95) + 1)
print("components needed for 95%:", k)                            # 2

Choosing k from Explained Variance

Method Rule
Threshold Smallest k with cumulative ratio ≥ 0.90 or 0.95
Scree plot Keep components before the curve flattens
Kaiser rule (standardized data) Eigenvalue > 1
Downstream k with the best validation score

Common Pitfalls

  • Comparing explained variance across datasets with different scaling.

  • Treating a low-variance component as useless for prediction; it can still separate classes.

  • Forgetting that unscaled data makes the ratios reflect units (raw Iris PC1 is about 92%).

Interview Tips

  • Equate explained variance with eigenvalues and the ratio with λᵢ / Σλ.

  • Show how the cumulative curve is used to pick k.

6

How does feature scaling affect PCA?

PCA looks for directions of maximum variance, and variance depends on units, so without scaling the features with the largest numeric ranges dominate the first components regardless of how informative they are: a salary in rupees (variance in the billions) would swamp age in years, and the “first component” would simply be salary. Standardizing each feature to mean 0 and standard deviation 1 (equivalent to running PCA on the correlation matrix) gives every feature an equal starting weight so components reflect shared structure rather than units; scaling is the right default when features have different units, while leaving data unscaled can be correct when all features share a meaningful unit (pixel intensities, spectra) and their relative variances matter. Scaling also changes the explained variance ratios and the loadings, so always say which version you used, and fit the scaler on training data only, inside a Pipeline.

Read full answer

Applies toscikit-learn 1.5+

Code Example

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 1000
age = rng.normal(40, 10, n)                                   # years
income = 20_000 + 900 * age + rng.normal(0, 15_000, n)        # rupees, depends on age
score = rng.normal(600, 50, n)                                # credit score, unrelated
X = np.column_stack([age, income, score])
names = ["age", "income", "score"]

raw = PCA().fit(X)
print("unscaled ratios:", raw.explained_variance_ratio_.round(4))
print("unscaled PC1 loadings:", dict(zip(names, raw.components_[0].round(4))))
# PC1 is almost purely income, because its variance is millions of times larger

Xs = StandardScaler().fit_transform(X)
scaled = PCA().fit(Xs)
print("scaled ratios:", scaled.explained_variance_ratio_.round(3))
print("scaled PC1 loadings:", dict(zip(names, scaled.components_[0].round(3))))
# PC1 now combines age and income (they are correlated) and score gets its own component

When to Scale

Situation Scale?
Mixed units (age, income, counts) Yes, standardize
Same unit, variance meaningful (pixels, spectra) Often no, just centre
Heavy outliers Consider RobustScaler or robust PCA
Sparse text features Use TruncatedSVD without centring

Common Pitfalls

  • Running PCA on raw data with mixed units and interpreting the first component as “most important”.

  • Fitting the scaler on the full dataset before splitting.

  • Comparing loadings between scaled and unscaled analyses.

Interview Tips

  • Give a concrete example of one feature dominating.

  • Mention correlation-matrix PCA and when not to scale.

7

What is the difference between PCA and factor analysis?

PCA is a data-summarizing technique that finds orthogonal linear combinations of the observed variables explaining the most total variance, while factor analysis (FA) is a statistical model that assumes the observed variables are generated by a smaller number of hidden factors plus feature-specific noise, x = μ + Lf + ε, and tries to explain the shared variance (covariance) between variables. PCA’s components are functions of the data; FA’s factors are latent causes behind the data. FA estimates a separate noise variance (uniqueness) for each variable, so it doesn’t spend components on variance that only one variable has, and its loadings can be rotated (varimax, promax) to make factors easier to interpret, which is why FA is used in psychology and survey analysis to discover constructs such as “satisfaction”. PCA is preferred for compression, preprocessing and visualization; when uniquenesses are small and equal, the two give similar results (probabilistic PCA is FA with equal noise variance for every feature).

Read full answer

Applies toscikit-learn 1.5+

Comparison

PCA Factor analysis
Goal Summarize total variance Model shared variance with latent factors
Model No generative model (except probabilistic PCA) x = μ + Lf + ε, ε with per-feature variance
Noise Treated as part of the variance Explicit uniqueness per variable
Solution Eigenvectors, unique up to sign Loadings unique up to rotation
Rotation Not used Varimax/promax for interpretability
Typical use Compression, preprocessing, plots Surveys, psychometrics, latent constructs

Code Example

import numpy as np
from sklearn.decomposition import PCA, FactorAnalysis

rng = np.random.default_rng(42)
n = 2000
factors = rng.normal(size=(n, 2))                      # two latent factors
L = np.array([[0.9, 0.0], [0.8, 0.0], [0.7, 0.0],     # items 0-2 load on factor 1
              [0.0, 0.9], [0.0, 0.8], [0.0, 0.7]])    # items 3-5 load on factor 2
noise_sd = np.array([0.3, 0.5, 0.7, 0.3, 0.5, 2.0])   # item 5 is very noisy
X = factors @ L.T + rng.normal(size=(n, 6)) * noise_sd

fa = FactorAnalysis(n_components=2, rotation="varimax", random_state=42).fit(X)
print("FA loadings:\n", fa.components_.T.round(2))
print("FA noise variances:", fa.noise_variance_.round(2))   # close to noise_sd**2, item 5 about 4.0

pca = PCA(n_components=2).fit(X)
print("PCA loadings:\n", pca.components_.T.round(2))
# PCA gives the noisy item 5 a large loading because its total variance is high;
# FA attributes that variance to the item's own noise

Common Pitfalls

  • Calling PCA “factor analysis” in reports; they answer different questions.

  • Interpreting rotated factors as proven causes.

  • Running FA with too few observations per variable.

Interview Tips

  • Contrast total variance vs shared variance and the generative model.

  • Mention uniquenesses, rotation and probabilistic PCA.

8

Why is PCA considered an unsupervised technique?

PCA is unsupervised because it uses only the input features X and never a target label y: it finds directions of maximum variance in X, and those directions are the same whether the data is labelled spam/not spam, cat/dog, or not labelled at all. That is useful for exploring data, compressing it and removing redundancy, but it also means PCA has no guarantee of keeping the information that matters for a particular prediction task: a low-variance direction can be exactly the one that separates classes, and PCA might drop it. When labels are available and the goal is prediction, supervised alternatives such as linear discriminant analysis (maximizes class separation), partial least squares (maximizes covariance with the target) or simply selecting k by validation performance inside a Pipeline make sure the reduction serves the task.

Read full answer

Applies toscikit-learn 1.5+

Code Example: Variance Is Not the Same as Class Separation

import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
n = 1000
y = rng.integers(0, 2, n)
big_noise = rng.normal(0, 10, n)                         # high variance, unrelated to y
small_signal = (2 * y - 1) * 0.5 + rng.normal(0, 0.2, n)  # low variance, separates the classes
X = np.column_stack([big_noise, small_signal])

pca = PCA(n_components=1).fit(X)                          # never sees y
print("PCA direction:", pca.components_[0].round(3))     # about [1, 0]: the noise axis
acc_pca = cross_val_score(make_pipeline(PCA(n_components=1), LogisticRegression()), X, y, cv=5).mean()
acc_lda = cross_val_score(make_pipeline(LinearDiscriminantAnalysis(n_components=1), LogisticRegression()), X, y, cv=5).mean()
print(f"accuracy with 1 PCA component {acc_pca:.3f}, with 1 LDA component {acc_lda:.3f}")
# about 0.5 (chance) vs nearly 1.0

Unsupervised vs Supervised Reduction

Method Uses y? Objective
PCA No Maximize variance
t-SNE / UMAP No Preserve neighbourhoods
LDA Yes Maximize between-class vs within-class variance
PLS Yes Maximize covariance with the target
Feature selection by model Yes Predictive importance

Common Pitfalls

  • Assuming the top components are the most predictive.

  • Choosing k without checking downstream performance.

  • Thinking “unsupervised” means PCA can be fitted on test data; it still leaks information.

Interview Tips

  • Say PCA ignores labels and give the low-variance-signal counterexample.

  • Mention LDA and PLS as supervised alternatives.

9

Derive PCA as minimizing reconstruction error

PCA can be derived as finding the k-dimensional linear subspace that minimizes the average squared distance between each centred data point and its projection onto that subspace; because the squared length of each point splits into the part inside the subspace plus the part outside it (Pythagoras), minimizing the reconstruction error is the same as maximizing the variance of the projections, and the solution is the subspace spanned by the top k eigenvectors of the covariance matrix. The minimum error equals the sum of the discarded eigenvalues, (n − 1)·Σᵢ₌ₖ₊₁ λᵢ in total, which is the Eckart-Young theorem: the truncated SVD is the best rank-k approximation of the centred data in the Frobenius norm.

Read full answer

Applies tolinear algebra; check with NumPy 2.x

Derivation

  1. Setup. Centred data x₁, …, xₙ ∈ ℝᵈ. Choose an orthonormal basis W ∈ ℝᵈˣᵏ (WᵀW = I). The projection of x is WWᵀx, and the reconstruction error is J(W) = Σᵢ ‖xᵢ − WWᵀxᵢ‖².

  2. Expand. Since WWᵀ is a projection (symmetric and idempotent), ‖x − WWᵀx‖² = xᵀx − xᵀWWᵀx. So J(W) = Σᵢ ‖xᵢ‖² − Σᵢ ‖Wᵀxᵢ‖².

  3. Convert to variance. The first term doesn’t depend on W, so minimizing J means maximizing Σᵢ ‖Wᵀxᵢ‖² = tr(WᵀXᵀXW) = (n − 1)·tr(WᵀCW), where C is the covariance matrix.

  4. One direction (k = 1). Maximize wᵀCw subject to wᵀw = 1. The Lagrangian wᵀCw − λ(wᵀw − 1) has gradient 2Cw − 2λw = 0, so Cw = λw: w is an eigenvector, and the objective equals λ, so the best choice is the top eigenvector.

  5. k directions. Maximizing tr(WᵀCW) with orthonormal columns gives the top k eigenvectors (by the Ky Fan theorem, or by induction adding one orthogonal direction at a time), and the maximum is λ₁ + … + λₖ.

  6. Minimum error. J_min = Σᵢ ‖xᵢ‖² − (n − 1)(λ₁ + … + λₖ) = (n − 1)(λₖ₊₁ + … + λ_d), because Σᵢ ‖xᵢ‖² = (n − 1)·tr©.

Code Check

import numpy as np

rng = np.random.default_rng(42)
A = rng.normal(size=(5, 5))
X = rng.normal(size=(500, 5)) @ A                        # correlated data
Xc = X - X.mean(axis=0)
n, k = len(Xc), 2

C = Xc.T @ Xc / (n - 1)
vals, vecs = np.linalg.eigh(C)
vals, vecs = vals[::-1], vecs[:, ::-1]

def recon_error(W):
    return float(np.sum((Xc - Xc @ W @ W.T) ** 2))

W_pca = vecs[:, :k]
print(f"PCA error {recon_error(W_pca):.2f}, predicted (n-1)*sum(discarded) {(n - 1) * vals[k:].sum():.2f}")   # equal

best_random = min(recon_error(np.linalg.qr(rng.normal(size=(5, k)))[0]) for _ in range(2000))
print(f"best of 2000 random 2-D subspaces: {best_random:.2f}")   # never below the PCA error

U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
rank_k = U[:, :k] * s[:k] @ Vt[:k]
print(np.isclose(np.sum((Xc - rank_k) ** 2), recon_error(W_pca)))   # True: Eckart-Young

Common Pitfalls

  • Forgetting to centre, which makes the “best subspace” pass through the origin instead of the mean.

  • Mixing up the totals: error sums over points, so the factor n − 1 (or n) matters when comparing to eigenvalues.

  • Treating minimum reconstruction error as optimal for a downstream prediction task.

Interview Tips

  • Use the Pythagoras split to show the equivalence with maximum variance.

  • Solve the one-direction case with a Lagrange multiplier and state the discarded-eigenvalue error.

10

What is SVD and how does it relate to PCA?

Singular Value Decomposition factors any n × d matrix as X = UΣVᵀ, where U (n × r) and V (d × r) have orthonormal columns and Σ is diagonal with non-negative singular values s₁ ≥ s₂ ≥ … ; for centred data, the columns of V are exactly the principal component directions, the variance along component i is sᵢ² / (n − 1), and the scores (projections) are XV = UΣ. That follows from XᵀX = VΣ²Vᵀ, so the eigenvectors of the covariance matrix XᵀX/(n − 1) are V and its eigenvalues are sᵢ²/(n − 1). Computing PCA through the SVD of X is what scikit-learn does because it never forms XᵀX, which squares the condition number and loses precision, and truncated or randomized SVD makes PCA practical for large or sparse data (TruncatedSVD skips centring, which is the basis of latent semantic analysis for text).

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Correspondence

SVD of centred X PCA
Columns of V (rows of Vᵀ) Principal components (components_)
sᵢ² / (n − 1) Explained variance (explained_variance_)
sᵢ² / Σ s² Explained variance ratio
UΣ Scores (transform output)
U_kΣ_kV_kᵀ Best rank-k reconstruction

Code Example

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

X = load_digits().data                                   # 1797 x 64
Xc = X - X.mean(axis=0)
U, s, Vt = np.linalg.svd(Xc, full_matrices=False)

pca = PCA(n_components=10).fit(X)
print(np.allclose(s[:10] ** 2 / (len(X) - 1), pca.explained_variance_))     # True
signs = np.sign(np.sum(Vt[:10] * pca.components_, axis=1))
print(np.allclose(Vt[:10] * signs[:, None], pca.components_))               # True (up to sign)
print(np.allclose((U[:, :10] * s[:10]) * signs, pca.transform(X)))          # True: scores = U * S

C = Xc.T @ Xc / (len(X) - 1)
eigvals = np.linalg.eigvalsh(C)[::-1]
print(np.allclose(eigvals[:10], s[:10] ** 2 / (len(X) - 1)))                # True: same eigenvalues

k = 10
approx = U[:, :k] * s[:k] @ Vt[:k] + X.mean(axis=0)
print(f"relative error of rank-{k} reconstruction: {np.linalg.norm(X - approx) / np.linalg.norm(Xc):.3f}")

SVD Variants

Variant Use
Full SVD Small or medium dense data
Truncated SVD (ARPACK) Top k components only
Randomized SVD Large data, approximate top k fast
TruncatedSVD without centring Sparse matrices such as TF-IDF (LSA)

Common Pitfalls

  • Forgetting to centre before the SVD (then it is not PCA, unless intended as with TruncatedSVD).

  • Comparing signs of components between libraries.

  • Building the covariance matrix for very wide data instead of using the SVD directly.

Interview Tips

  • Derive XᵀX = VΣ²Vᵀ to connect SVD and PCA.

  • Mention numerical stability and truncated/randomized SVD.

11

How do you choose the number of principal components?

Choose the number of components k from what the reduction is for: for compression or noise removal, take the smallest k whose cumulative explained variance ratio reaches a target such as 90–95%; for exploration, look for the elbow in a scree plot where extra components add little; for prediction, treat k as a hyperparameter and pick it by cross-validated performance of the whole pipeline; and for a statistically principled cut-off, use parallel analysis (keep components whose eigenvalues beat those from random data of the same shape) or probabilistic PCA’s likelihood, which scikit-learn’s n_components="mle" uses. Rules of thumb like the Kaiser criterion (eigenvalue > 1 on standardized data) are quick but unreliable. For visualization, k is simply 2 or 3, and you should report how much variance those keep.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Methods

Method Rule Best for
Cumulative variance Smallest k with ≥ 90–95% Compression, general use
Scree / elbow Bend in the eigenvalue curve Exploration
Cross-validation Best downstream score Predictive pipelines
Parallel analysis Eigenvalue above the random-data 95th percentile Finding real structure
Kaiser rule λ > 1 (standardized data) Quick look only
PPCA likelihood PCA(n_components="mle") Automatic choice
Reconstruction error Error below a tolerance Compression with quality targets

Code Example

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
Xs = StandardScaler().fit_transform(X)
pca = PCA().fit(Xs)
cum = np.cumsum(pca.explained_variance_ratio_)
for target in [0.80, 0.90, 0.95, 0.99]:
    print(f"{target:.0%} variance needs k = {int(np.searchsorted(cum, target) + 1)}")

rng = np.random.default_rng(42)                     # parallel analysis
random_eigs = np.array([
    PCA().fit(StandardScaler().fit_transform(rng.normal(size=Xs.shape))).explained_variance_
    for _ in range(20)
])
threshold = np.percentile(random_eigs, 95, axis=0)
print("parallel analysis keeps", int(np.sum(pca.explained_variance_ > threshold)), "components")
print("Kaiser rule keeps", int(np.sum(pca.explained_variance_ > 1)), "components")

pipe = Pipeline([("scale", StandardScaler()), ("pca", PCA()), ("clf", LogisticRegression(max_iter=3000))])
search = GridSearchCV(pipe, {"pca__n_components": [10, 20, 30, 40, 50]}, cv=5).fit(X, y)
print("best k by cross-validation:", search.best_params_["pca__n_components"],
      f"accuracy {search.best_score_:.3f}")

Different rules give different answers; that is normal, and the right one depends on the goal.

Common Pitfalls

  • Using a fixed 95% rule for a predictive model without validating it.

  • Choosing k on the full dataset before splitting, then reporting test scores.

  • Comparing thresholds between scaled and unscaled analyses.

Interview Tips

  • Name several methods and match each to a goal.

  • Recommend cross-validation in a Pipeline when there is a target.

12

What does "loading" mean in PCA?

A loading describes how strongly an original feature contributes to a principal component: in the most common usage it is the feature’s coefficient (weight) in the component’s eigenvector, so component 1 = 0.52·sepal length − 0.27·sepal width + 0.58·petal length + 0.56·petal width for standardized Iris, and in the stricter statistical usage it is that weight scaled by the square root of the eigenvalue, which for standardized data equals the correlation between the feature and the component’s scores. Large absolute loadings show which features define a component, the sign shows the direction (features with the same sign move together along it), and squared loadings across components show how well each feature is represented. In scikit-learn the unit-length weights are pca.components_ (one row per component), and the correlation-style loadings are components_.T * sqrt(explained_variance_).

Read full answer

Applies toscikit-learn 1.5+

Two Conventions

Name Formula Meaning
Weights / coefficients Eigenvector entries vⱼₖ (components_) Recipe for building the score from features
Loadings (correlation-style) vⱼₖ·√λₖ Correlation of feature j with component k (standardized data)

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X = StandardScaler().fit_transform(iris.data)
names = ["sepal length", "sepal width", "petal length", "petal width"]
pca = PCA(n_components=2).fit(X)

for k, row in enumerate(pca.components_, start=1):
    print(f"PC{k} weights:", dict(zip(names, row.round(3))))
# PC1 about [0.52, -0.27, 0.58, 0.56] up to sign: petal size and sepal length together
# PC2 is dominated by sepal width

loadings = pca.components_.T * np.sqrt(pca.explained_variance_)
scores = pca.transform(X)
corr = np.array([[np.corrcoef(X[:, j], scores[:, k])[0, 1] for k in range(2)] for j in range(4)])
print("correlation-style loadings:\n", loadings.round(3))
print("match correlations (up to the ddof factor):", np.allclose(loadings, corr, atol=0.01))   # True

print("share of each feature's variance kept by 2 PCs:", np.sum(corr**2, axis=1).round(3))

The small mismatch between loadings and correlations comes from StandardScaler using ddof = 0 while PCA’s explained variance uses ddof = 1; with 150 rows the difference is about 0.3%.

Interpreting Loadings

  • Look at the largest absolute loadings per component and name the theme (PC1 here: “overall flower size”).

  • Compare signs to see which features trade off.

  • Use a biplot to show scores and loading vectors together.

Common Pitfalls

  • Interpreting loadings from unscaled data with mixed units.

  • Reading small loadings as meaningful without checking stability (bootstrap the PCA).

  • Forgetting that the overall sign of a component is arbitrary.

Interview Tips

  • Distinguish eigenvector weights from correlation-style loadings.

  • Show how to read a component from its largest loadings.

13

Explain eigenvalue decomposition in PCA

Eigenvalue decomposition writes a symmetric matrix as C = VΛVᵀ, where V’s columns are orthonormal eigenvectors and Λ is a diagonal matrix of eigenvalues; in PCA you compute the covariance (or correlation) matrix of the centred data, decompose it, sort the eigenpairs from largest to smallest eigenvalue, and the sorted eigenvectors are the principal components while the eigenvalues are the variances they explain. Projecting the data with Z = XcV rotates it into a coordinate system where the covariance is the diagonal Λ, so the new features are uncorrelated. Because covariance matrices are symmetric positive semi-definite, the spectral theorem guarantees real, non-negative eigenvalues and orthogonal eigenvectors; numerically you use a symmetric solver (np.linalg.eigh), or skip forming C and use the SVD of Xc, which gives the same result more stably.

Read full answer

Applies toNumPy 2.x

Process

Step Operation Cost
1 Centre (and standardize) X (n × d) O(nd)
2 C = XcᵀXc / (n − 1) O(nd²)
3 C = VΛVᵀ with a symmetric solver O(d³)
4 Sort eigenvalues descending, reorder V O(d log d)
5 Keep the top k columns, Z = XcV_k O(ndk)

Code Example

import numpy as np

rng = np.random.default_rng(42)
X = rng.normal(size=(300, 4)) @ np.array([[2, 0, 0, 0],
                                          [1.5, 1, 0, 0],
                                          [0, 0, 0.5, 0],
                                          [0.5, 0.3, 0, 0.2]])
Xc = X - X.mean(axis=0)
C = np.cov(Xc, rowvar=False)

eigvals, V = np.linalg.eigh(C)              # ascending
order = np.argsort(eigvals)[::-1]
eigvals, V = eigvals[order], V[:, order]
print("eigenvalues:", eigvals.round(3))
print("C rebuilt from V Lambda V^T:", np.allclose(V @ np.diag(eigvals) @ V.T, C))   # True
print("V orthonormal:", np.allclose(V.T @ V, np.eye(4)))                            # True

Z = Xc @ V
print("covariance of the rotated data:\n", np.cov(Z, rowvar=False).round(3))   # diagonal = eigenvalues
print("explained variance ratio:", (eigvals / eigvals.sum()).round(3))

k = 2
X_approx = Z[:, :k] @ V[:, :k].T + X.mean(axis=0)
print(f"rank-{k} reconstruction keeps {eigvals[:k].sum() / eigvals.sum():.1%} of the variance")

Eigen-decomposition vs SVD

Eigen-decomposition of C SVD of Xc
Forms d × d matrix Yes No
Precision Squares the condition number Better
Good when d small, n huge General default
Library np.linalg.eigh np.linalg.svd, sklearn’s PCA

Common Pitfalls

  • Using np.linalg.eig and forgetting to sort or to take the real part.

  • Decomposing the uncentred XᵀX.

  • Building C when d is very large (memory d²).

Interview Tips

  • Write C = VΛVᵀ and explain sorting, projection and diagonal covariance.

  • Mention the spectral theorem and why SVD is often preferred.

14

Why does the trace of a matrix matter in PCA?

The trace of a square matrix is the sum of its diagonal entries, and it equals the sum of its eigenvalues; in PCA the trace of the covariance matrix is the total variance of the data (the sum of every feature’s variance), so it is the denominator of the explained variance ratio, λᵢ / tr©, and it is preserved by the rotation into principal components because tr(VᵀCV) = tr© for an orthogonal V. The trace also appears in the derivation: PCA maximizes tr(WᵀCW), the variance kept by k orthonormal directions, whose maximum is λ₁ + … + λₖ, and the reconstruction error is (n − 1)·(tr© − Σᵢ≤ₖ λᵢ). For standardized data the trace is simply the number of features d, which is why the Kaiser rule compares each eigenvalue with 1 (the average).

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Where the Trace Appears

Role Formula
Total variance tr© = Σⱼ Var(xⱼ) = Σᵢ λᵢ
Explained variance ratio λᵢ / tr©
Objective maximize tr(WᵀCW) subject to WᵀW = I
Reconstruction error (n − 1)·(tr© − Σᵢ≤ₖ λᵢ)
Invariance tr(VᵀCV) = tr© for orthogonal V
Standardized data tr© = d (with matching ddof)

Code Example

import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA

X = load_wine().data
Xs = (X - X.mean(axis=0)) / X.std(axis=0, ddof=1)        # standardize with ddof=1 to match np.cov
C = np.cov(Xs, rowvar=False)
eigvals = np.linalg.eigvalsh(C)[::-1]

print(f"trace {np.trace(C):.4f}, sum of eigenvalues {eigvals.sum():.4f}, features {X.shape[1]}")   # 13.0000 all three
pca = PCA().fit(Xs)
print(np.allclose(pca.explained_variance_ratio_, eigvals / np.trace(C)))   # True

V = pca.components_.T
print("trace after rotation:", round(float(np.trace(V.T @ C @ V)), 4))     # 13.0: preserved

k = 3
n = len(Xs)
W = V[:, :k]
error = np.sum((Xs - Xs @ W @ W.T) ** 2)
print(np.isclose(error, (n - 1) * (np.trace(C) - eigvals[:k].sum())))     # True
print("Kaiser rule keeps", int(np.sum(eigvals > np.trace(C) / len(C))), "components")

Common Pitfalls

  • Using a trace computed with a different ddof from the eigenvalues, which makes ratios slightly off.

  • Comparing traces of scaled and unscaled data.

  • Forgetting that the trace is invariant only under orthogonal (rotation) changes of basis.

Interview Tips

  • State tr© = Σλ = total variance.

  • Show its role in the explained variance ratio and the PCA objective.

15

What are PCA's limitations with non-linear relationships?

PCA can only find flat, linear subspaces: each component is a weighted sum of the original features, so when the data lies on a curved structure (a spiral, a circle, a “Swiss roll”, clusters arranged in rings) the directions of maximum variance cut straight across it, mixing points that are far apart along the curve and needing many components to describe something that is really low-dimensional. PCA also captures only second-order statistics (variances and covariances), so it misses non-linear dependence such as y = x², where the correlation is zero but the relationship is perfect. When non-linear structure matters, use kernel PCA (PCA in a feature space defined by a kernel), manifold learning (Isomap, locally linear embedding), t-SNE or UMAP for visualization, or autoencoders, which are non-linear generalizations of PCA, and still try PCA first because it is fast, stable and invertible.

Read full answer

Applies toscikit-learn 1.5+

Code Example: Concentric Circles

import numpy as np
from sklearn.datasets import make_circles
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = make_circles(n_samples=1000, factor=0.3, noise=0.05, random_state=42)

z_pca = PCA(n_components=1).fit_transform(X)
z_kpca = KernelPCA(n_components=1, kernel="rbf", gamma=10).fit_transform(X)

for name, z in [("linear PCA", z_pca), ("kernel PCA (RBF)", z_kpca)]:
    acc = cross_val_score(LogisticRegression(), z, y, cv=5).mean()
    print(f"{name:17s}: accuracy from 1 component {acc:.3f}")
# about 0.5 for linear PCA (the rings overlap on any straight line) vs close to 1.0 for kernel PCA

rng = np.random.default_rng(42)
x = rng.uniform(-1, 1, 5000)
print(f"corr(x, x^2) = {np.corrcoef(x, x**2)[0, 1]:.3f}")   # about 0: PCA sees no relationship

Alternatives

Method Non-linear? Invertible Out-of-sample Best for
PCA No Yes Yes Baseline, compression
Kernel PCA Yes Approximate Yes Curved structure, moderate n
Isomap / LLE Yes No Limited Smooth manifolds
t-SNE Yes No No 2-D visualization of clusters
UMAP Yes Approximate Yes Visualization and features, large n
Autoencoder Yes Yes (decoder) Yes Large data, images

Common Pitfalls

  • Concluding there’s no structure because PCA plots look like a blob.

  • Using t-SNE output as model features (no transform for new data, unstable distances).

  • Kernel PCA on very large n (an n × n kernel matrix).

Interview Tips

  • Give a concrete failure (circles, Swiss roll, x vs x²).

  • List kernel PCA, manifold methods and autoencoders with their trade-offs.

16

How is PCA used in image processing?

In image processing PCA treats each image (or each small patch) as a long vector of pixel values and finds the few directions that capture most of the variation across images; this gives compression (store k scores per image instead of every pixel), denoising (reconstruct from the top components, which keep structure, and drop the rest, which are mostly noise), face recognition with eigenfaces (compare faces by their component scores), and fast low-dimensional features for classifiers. Other uses are colour-space decorrelation (PCA on RGB values, used in AlexNet-style “PCA colour augmentation”), aligning or detecting objects with PCA on shapes, and analysing hyperspectral images with hundreds of bands. Modern deep networks outperform PCA for recognition, but it remains a fast, interpretable baseline and a common preprocessing step.

Read full answer

Applies toscikit-learn 1.5+

Applications

Application How PCA is used
Compression Keep k component scores, reconstruct with inverse_transform
Denoising Reconstruct from the top k components
Eigenfaces Faces as combinations of “eigenface” images
Feature extraction Scores as classifier inputs
Colour augmentation Perturb images along RGB principal axes
Hyperspectral imaging Reduce hundreds of bands to a few
Shape models Active shape models of landmark points

Code Example: Compression and Denoising of Digits

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split

X = load_digits().data / 16.0                         # 8x8 images, pixels scaled to 0..1
X_train, X_test = train_test_split(X, test_size=0.3, random_state=42)

for k in [5, 10, 20, 40]:
    pca = PCA(n_components=k).fit(X_train)
    recon = pca.inverse_transform(pca.transform(X_test))
    mse = np.mean((X_test - recon) ** 2)
    print(f"k={k:>2}: {k} numbers instead of 64 ({64 / k:.1f}x smaller), MSE {mse:.4f}")

rng = np.random.default_rng(42)
noisy = np.clip(X_test + rng.normal(0, 0.25, X_test.shape), 0, 1)
pca = PCA(n_components=12).fit(X_train)              # fit on clean training images
denoised = pca.inverse_transform(pca.transform(noisy))
print(f"error vs clean: noisy {np.mean((noisy - X_test) ** 2):.4f}, denoised {np.mean((denoised - X_test) ** 2):.4f}")
# The denoised images are much closer to the clean ones than the noisy input

Storing the mean image and the k components is a fixed cost shared by every image, so the savings grow with the number of images compressed.

Common Pitfalls

  • Applying PCA to unaligned images; faces must be cropped and aligned for eigenfaces to make sense.

  • Using global PCA on large images (millions of pixels); use patches or randomized PCA.

  • Expecting PCA to match modern codecs (JPEG, learned compression) for photo quality.

Interview Tips

  • Explain images as vectors and reconstruction from k components.

  • Give compression, denoising and eigenfaces as examples.

17

What is the curse of dimensionality and how does PCA help?

The curse of dimensionality is the set of problems that appear as the number of features grows: the volume of the space grows exponentially, so data becomes sparse and you need exponentially more samples to cover it; distances between points become nearly equal, which weakens nearest-neighbour methods and clustering; and models have more parameters to fit, so they overfit more easily and cost more to train. PCA helps when the data actually lies close to a lower-dimensional linear subspace, which is common because real features are correlated: projecting onto the top k components keeps most of the structure, removes redundant and noisy directions, makes distances more meaningful, and reduces model size and training time. It cannot help if the information is spread evenly across many dimensions or lies on a non-linear manifold, and it doesn’t use the target, so check that downstream performance actually improves.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Symptoms

Symptom Example
Data sparsity Covering [0, 1]ᵈ with 10 bins per axis needs 10ᵈ cells
Distance concentration Nearest and farthest neighbours almost equally far
Overfitting More features than samples
Computation Cost grows with d

Code Example

import numpy as np
from sklearn.decomposition import PCA
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
for d in [2, 10, 100, 1000]:                       # distance concentration
    pts = rng.random((500, d))
    dist = np.linalg.norm(pts[1:] - pts[0], axis=1)
    print(f"d={d:>4}: farthest/nearest distance ratio {dist.max() / dist.min():.2f}")
# The ratio falls toward 1 as d grows, so "nearest" means less and less

n, latent_dim, noise_dims = 400, 5, 195
y = rng.integers(0, 2, n)
latent = rng.normal(size=(n, latent_dim)) + y[:, None] * 0.8     # class signal in 5 dimensions
mix = rng.normal(size=(latent_dim, 50))
X = np.hstack([latent @ mix, rng.normal(size=(n, noise_dims)) * 0.5])   # 245 features

knn = KNeighborsClassifier(n_neighbors=15)
raw = cross_val_score(make_pipeline(StandardScaler(), knn), X, y, cv=5).mean()
reduced = cross_val_score(make_pipeline(StandardScaler(), PCA(n_components=5), knn), X, y, cv=5).mean()
print(f"kNN on 245 features: {raw:.3f}; after PCA to 5: {reduced:.3f}")
# PCA keeps the correlated signal directions and drops most of the noise dimensions

Common Pitfalls

  • Expecting PCA to help when every feature carries independent information.

  • Fitting PCA before cross-validation splits.

  • Ignoring non-linear structure that PCA can’t capture.

Interview Tips

  • Name sparsity, distance concentration and overfitting.

  • Explain that PCA works by exploiting correlation between features.

18

How does PCA handle missing values?

Standard PCA does not handle missing values at all: scikit-learn’s PCA raises an error on NaN, and the covariance and SVD computations need complete data, so you must deal with gaps first. Simple options are dropping rows (fine when few rows are affected and values are missing at random) or imputing with the mean or median inside a Pipeline (fast, but it shrinks variances and correlations, which shrinks and distorts the components). Better options treat PCA itself as the imputation model: iterative PCA (fill gaps, fit a low-rank PCA, replace the gaps with the reconstruction, repeat until it stabilizes), probabilistic PCA fitted with the EM algorithm, which handles missing entries in the likelihood, or a multivariate iterative imputer; matrix completion methods (soft-impute, nuclear-norm minimization) do the same at scale. Whichever you choose, fit the imputer on training data only, and add missing-value indicators if missingness itself is informative.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Options

Approach Pros Cons
Drop rows Simple Loses data; biased if not MCAR
Mean/median imputation Fast, pipeline-friendly Shrinks variance and correlations
KNN imputation Uses similar rows Slow for large data
Iterative PCA (low-rank imputation) Uses the structure PCA finds Needs k chosen; iterative
Probabilistic PCA with EM Principled likelihood More complex
Iterative imputer Flexible models per feature Slower

Code Example: Mean Imputation vs Iterative PCA

import numpy as np
from sklearn.decomposition import PCA

rng = np.random.default_rng(42)
n, d, k = 500, 8, 2
X_true = rng.normal(size=(n, k)) @ rng.normal(size=(k, d)) + rng.normal(0, 0.1, (n, d))   # rank-2 data
mask = rng.random((n, d)) < 0.2                                  # 20% missing at random
X_obs = np.where(mask, np.nan, X_true)

def iterative_pca_impute(X, k, iters=100, tol=1e-6):
    missing = np.isnan(X)
    filled = np.where(missing, np.nanmean(X, axis=0), X)        # start from column means
    for _ in range(iters):
        pca = PCA(n_components=k).fit(filled)
        recon = pca.inverse_transform(pca.transform(filled))
        change = np.sqrt(np.mean((filled[missing] - recon[missing]) ** 2))
        filled[missing] = recon[missing]                         # only replace the gaps
        if change < tol:
            break
    return filled

mean_filled = np.where(mask, np.nanmean(X_obs, axis=0), X_obs)
pca_filled = iterative_pca_impute(X_obs, k)
for name, Xf in [("mean imputation", mean_filled), ("iterative PCA", pca_filled)]:
    rmse = np.sqrt(np.mean((Xf[mask] - X_true[mask]) ** 2))
    ratio = PCA(n_components=2).fit(Xf).explained_variance_ratio_.sum()
    print(f"{name:16s}: RMSE on missing cells {rmse:.3f}, variance in 2 PCs {ratio:.3f}")
# Iterative PCA recovers the gaps far better and restores the true low-rank structure (about 0.99)

Common Pitfalls

  • Fitting imputers on the full dataset before splitting.

  • Mean imputation with a lot of missing data, which flattens components.

  • Ignoring why values are missing (MNAR data biases every method).

Interview Tips

  • Say PCA needs complete data and list the options.

  • Describe iterative PCA or PPCA with EM as model-based imputation.

19

How is PCA used in feature engineering?

In feature engineering PCA turns many correlated raw features into a few uncorrelated summary features: it can compress groups of related columns (dozens of sensor readings, survey items or embedding dimensions) into a handful of components, decorrelate inputs for models that dislike multicollinearity (linear and logistic regression), remove noise directions, speed up distance-based models such as kNN and SVMs, and create new informative features such as the reconstruction error, which is a good anomaly score. A practical pattern is to apply PCA to a block of similar features rather than to everything, keep the original important features alongside, and choose the number of components by cross-validation inside a Pipeline so the transformation is learned from training data only. Tree-based models usually gain little from PCA, because rotating features can make their axis-aligned splits harder, while linear and distance-based models gain the most.

Read full answer

Applies toscikit-learn 1.5+

Patterns

Pattern Example
Block-wise PCA 40 sensor columns → 3 components, other features unchanged
Decorrelation Fix multicollinearity before linear regression
Embedding compression 768-d text embeddings → 64 components
Reconstruction error feature Anomaly score for fraud or defects
Whitening Unit-variance, uncorrelated inputs for some algorithms

Code Example: PCA on One Block of Features

import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 1500
machine_state = rng.normal(size=(n, 2))                              # hidden drivers
sensors = machine_state @ rng.normal(size=(2, 40)) + rng.normal(0, 1.0, (n, 40))   # 40 noisy sensors
age, load = rng.uniform(0, 10, n), rng.uniform(0, 1, n)
logit = 1.2 * machine_state[:, 0] - 0.8 * machine_state[:, 1] + 0.3 * age - 3
y = rng.random(n) < 1 / (1 + np.exp(-logit))
X = np.column_stack([sensors, age, load])
sensor_cols, other_cols = list(range(40)), [40, 41]

def model(n_components):
    if n_components is None:
        prep = StandardScaler()
    else:
        prep = ColumnTransformer([
            ("sensors", Pipeline([("scale", StandardScaler()), ("pca", PCA(n_components=n_components))]), sensor_cols),
            ("other", StandardScaler(), other_cols),
        ])
    return Pipeline([("prep", prep), ("clf", LogisticRegression(C=0.1, max_iter=2000))])

for k in [None, 1, 2, 5, 10]:
    score = cross_val_score(model(k), X, y, cv=5, scoring="roc_auc").mean()
    label = "raw 42 features" if k is None else f"{k} sensor PCs + 2 others"
    print(f"{label:24s}: AUC {score:.3f}")
# Two components summarize the 40 sensors about as well as the raw columns, with far fewer inputs

Common Pitfalls

  • Applying PCA to all features, mixing unrelated groups and losing interpretability.

  • Fitting PCA outside the cross-validation loop.

  • Using PCA in front of gradient-boosted trees by default.

  • Forgetting to scale each block first.

Interview Tips

  • Mention block-wise PCA, decorrelation and reconstruction error as a feature.

  • Say which models benefit and how k is chosen.

20

What is the difference between PCA and t-SNE?

PCA is a linear, deterministic projection that keeps the directions of maximum global variance, so distances and large-scale structure are preserved as well as a linear map can, and it can transform new data and be inverted; t-SNE is a non-linear, stochastic method for visualization that converts distances into neighbour probabilities and arranges points in 2-D or 3-D so that each point’s close neighbours stay close, which reveals clusters and local structure that PCA can miss. The price is that t-SNE distorts global geometry: the distances between clusters, the sizes of clusters and the empty space around them mean little, results change with the perplexity setting and random seed, it is slow on large data, and it has no transform for new points, so it is used for exploratory plots rather than as model features. A common workflow runs PCA first (to about 50 dimensions) to remove noise and speed up t-SNE, and UMAP is a faster alternative that keeps somewhat more global structure.

Read full answer

Applies toscikit-learn 1.5+

Comparison

PCA t-SNE
Type Linear Non-linear
Preserves Global variance and structure Local neighbourhoods
Deterministic Yes (up to sign) No; depends on seed and perplexity
New data transform Not supported
Invertible Yes No
Speed Fast Slow for large n
Output use Features, compression, plots Plots only
Interpretable axes Yes (loadings) No

Code Example

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE, trustworthiness
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
Xs = StandardScaler().fit_transform(X)

pca_2d = PCA(n_components=2).fit_transform(Xs)
tsne_2d = TSNE(n_components=2, perplexity=30, init="pca", random_state=42).fit_transform(
    PCA(n_components=30).fit_transform(Xs))

for name, emb in [("PCA", pca_2d), ("t-SNE", tsne_2d)]:
    trust = trustworthiness(Xs, emb, n_neighbors=10)
    knn = cross_val_score(KNeighborsClassifier(5), emb, y, cv=5).mean()
    print(f"{name:5s}: trustworthiness {trust:.3f}, kNN accuracy in 2-D {knn:.3f}")
# t-SNE keeps neighbourhoods much better: digit classes form clear separate groups in 2-D

Cross-validated kNN on a t-SNE embedding is only a way to measure how well classes separate in the plot; the embedding was computed on all points, so it is not a valid modelling pipeline.

Common Pitfalls

  • Reading cluster sizes or distances between clusters from a t-SNE plot.

  • Drawing conclusions from one t-SNE run; try several perplexities and seeds.

  • Using t-SNE coordinates as model features.

Interview Tips

  • Contrast global vs local structure, linear vs non-linear, transform vs none.

  • Mention PCA-then-t-SNE and UMAP.

21

How can PCA be used to reduce noise?

PCA reduces noise by assuming the real signal lives in a few high-variance directions while random noise spreads its variance thinly over all directions: project the data onto the top k components and map it back with the inverse transform, x̂ = μ + W_kW_kᵀ(x − μ), and the reconstruction keeps the structured signal while discarding the low-variance components that are mostly noise. It works best when the signal is approximately low-rank and the noise is roughly independent and similar in size across features (white noise); the number of components is chosen where the eigenvalue spectrum flattens into a “noise floor”, by cross-validation on a denoising target, or by random matrix theory thresholds. Typical uses are cleaning sensor and spectral data, image and patch denoising, removing noise from gene expression or financial returns, and stabilizing features before modelling.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Code Example

import numpy as np
from sklearn.decomposition import PCA

rng = np.random.default_rng(42)
n, d, rank = 1000, 50, 3
t = np.linspace(0, 1, d)
basis = np.vstack([np.sin(2 * np.pi * t), np.cos(2 * np.pi * t), t])     # 3 smooth patterns
clean = rng.normal(size=(n, rank)) @ basis                                 # low-rank signal
noisy = clean + rng.normal(0, 0.5, (n, d))

pca_full = PCA().fit(noisy)
print("top eigenvalues:", pca_full.explained_variance_[:6].round(3))
# Three large values, then a flat floor near the noise variance 0.25

for k in [1, 3, 10, 50]:
    pca = PCA(n_components=k).fit(noisy)
    denoised = pca.inverse_transform(pca.transform(noisy))
    print(f"k={k:>2}: MSE vs clean {np.mean((denoised - clean) ** 2):.4f}")
print(f"noisy input MSE vs clean: {np.mean((noisy - clean) ** 2):.4f}")   # about 0.25
# k=3 gives the lowest error: about 3/50 of the noise is left (roughly 0.015)

With k = 3, the reconstruction keeps only the noise that falls inside the 3-dimensional signal subspace, about 3/50 of it, so the error falls from about 0.25 to roughly 0.015.

Choosing k for Denoising

Method Idea
Spectrum inspection Keep components above the flat noise floor
Known noise level Keep eigenvalues clearly above σ²
Marchenko-Pastur bound Random-noise eigenvalues fall below σ²(1 + √(d/n))²
Validation Minimize error on held-out clean or repeated measurements

Common Pitfalls

  • Too many components, which keep noise; too few, which remove real signal.

  • Non-uniform noise across features; standardize or whiten the noise first.

  • Fitting on the data you evaluate on when the goal is a reusable denoiser.

  • Assuming small components are always noise; rare but important signals can hide there.

Interview Tips

  • Explain signal in a low-rank subspace and noise spread across all directions.

  • Show how to read the noise floor from the spectrum.

22

How would you use PCA for visualization?

To visualize high-dimensional data with PCA, standardize the features, fit PCA, and plot each sample’s scores on the first two (or three) components, coloured by a label or a variable of interest; because PCA is a linear projection that preserves as much variance as possible, the plot shows the main global structure, such as clusters, gradients, outliers and groups that overlap, along axes you can explain from the loadings. Always report how much variance the plotted components explain (a 2-D plot that keeps 30% of the variance hides a lot), add a scree plot, and use a biplot (scores plus loading arrows) when the audience needs to see which features drive the axes. If the structure is non-linear, follow up with t-SNE or UMAP, often run on the top PCA components.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+

Code Example: Scores Plot, Biplot and Scree Plot

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

wine = load_wine()
X = StandardScaler().fit_transform(wine.data)
pca = PCA().fit(X)
scores = pca.transform(X)[:, :2]
ratio = pca.explained_variance_ratio_

fig, axes = plt.subplots(1, 2, figsize=(12, 5))
ax = axes[0]
for cls, marker in zip(range(3), ["o", "s", "^"]):
    pts = scores[wine.target == cls]
    ax.scatter(pts[:, 0], pts[:, 1], marker=marker, alpha=0.7, label=wine.target_names[cls])
scale = np.abs(scores).max() * 0.8
for j in np.argsort(-np.linalg.norm(pca.components_[:2].T, axis=1))[:5]:   # 5 strongest features
    vx, vy = pca.components_[0, j] * scale, pca.components_[1, j] * scale
    ax.arrow(0, 0, vx, vy, color="black", width=0.01, head_width=0.1)
    ax.annotate(wine.feature_names[j], (vx * 1.1, vy * 1.1), fontsize=8)
ax.set(xlabel=f"PC1 ({ratio[0]:.1%})", ylabel=f"PC2 ({ratio[1]:.1%})", title="Wine: PCA biplot")
ax.legend()

ax = axes[1]
ax.bar(range(1, len(ratio) + 1), ratio, label="per component")
ax.plot(range(1, len(ratio) + 1), np.cumsum(ratio), marker="o", color="black", label="cumulative")
ax.set(xlabel="component", ylabel="explained variance ratio", title="Scree plot")
ax.legend()
fig.tight_layout()
fig.savefig("pca_visualization.png", dpi=120)
print(f"the 2-D plot shows {ratio[:2].sum():.1%} of the variance")   # about 55% for standardized wine

Checklist

  • Standardize unless features share a meaningful unit.

  • Label axes with the explained variance of each component.

  • Use colour, marker shape and an accessible palette for groups.

  • Show the scree plot so readers know what is missing.

  • Use equal axis scales so distances aren’t distorted.

Common Pitfalls

  • Concluding that groups overlap when they may separate in later components.

  • Unscaled data making the plot about one feature’s units.

  • Over-reading small differences in a 2-D view that keeps little variance.

Interview Tips

  • Describe scores plots, biplots and scree plots.

  • Say you would report explained variance and try non-linear methods if needed.

23

How are PCA results affected by outliers, and how can you fix it?

PCA is sensitive to outliers because it maximizes variance, a sum of squared deviations, so a few extreme points can contribute most of the variance and pull the first components toward themselves, rotating the axes away from the structure of the bulk of the data and inflating the explained variance of a misleading direction. To address it, first detect outliers (high Mahalanobis distance, large scores on some components, or large reconstruction error), then decide whether they are errors to remove or real rare events to keep; use robust scaling (median and IQR) instead of mean and standard deviation; and use robust PCA variants: PCA on a robust covariance estimate such as the Minimum Covariance Determinant (MinCovDet), projection-pursuit methods like ROBPCA, or Principal Component Pursuit, which splits the data into a low-rank part plus a sparse outlier part.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Code Example: Outliers Rotate the First Component

import numpy as np
from sklearn.covariance import MinCovDet
from sklearn.decomposition import PCA

rng = np.random.default_rng(42)
n = 300
x = rng.normal(0, 3, n)
y = 0.5 * x + rng.normal(0, 0.5, n)                      # main direction about 26.6 degrees
X = np.column_stack([x, y])
outliers = np.array([[2, 30], [3, 28], [1, 32], [2.5, 29], [1.5, 31]])   # 5 points far up the y-axis
X_dirty = np.vstack([X, outliers])

def angle(v):
    v = v if v[0] >= 0 else -v
    return float(np.degrees(np.arctan2(v[1], v[0])))

print(f"clean PC1 angle {angle(PCA(1).fit(X).components_[0]):.1f} degrees")       # about 26.6
print(f"with 5 outliers {angle(PCA(1).fit(X_dirty).components_[0]):.1f} degrees")  # pulled up to roughly 70

mcd = MinCovDet(random_state=42).fit(X_dirty)            # robust covariance ignores the outliers
vals, vecs = np.linalg.eigh(mcd.covariance_)
print(f"robust (MCD) PC1 angle {angle(vecs[:, -1]):.1f} degrees")                 # back near 26.6

d2 = mcd.mahalanobis(X_dirty)                            # flag outliers
print("flagged rows:", np.where(d2 > np.percentile(d2, 98.5))[0])                # includes 300-304

Remedies

Remedy How
Inspect and clean Fix data-entry errors; justify any removal
Robust scaling RobustScaler (median, IQR)
Robust covariance PCA Eigenvectors of the MCD covariance
ROBPCA Projection pursuit plus MCD, for high dimensions
Principal Component Pursuit X = L (low rank) + S (sparse outliers)
Transformations Log or rank transforms for heavy-tailed features

Common Pitfalls

  • Deleting outliers automatically when they may be the interesting cases (fraud, faults).

  • Detecting outliers with the same non-robust PCA they distort (masking).

  • Standardizing with mean and standard deviation that the outliers have inflated.

Interview Tips

  • Explain why squared deviations make PCA sensitive.

  • Mention MCD-based PCA, robust PCA (L + S) and careful outlier handling.

24

Can PCA be applied to categorical data?

Standard PCA is designed for continuous numeric variables, because it relies on means, variances and Euclidean distances, which have no natural meaning for categories such as colour or city; you can technically run PCA on one-hot encoded columns and it will produce something, but the result is distorted: dummy columns of rare categories have small variances and get little weight, the columns of one variable are dependent (they sum to 1), and distances reflect the coding rather than real similarity. The proper tools are Multiple Correspondence Analysis (MCA) for purely categorical data, which is PCA on a suitably weighted indicator matrix using chi-squared distances; Factor Analysis of Mixed Data (FAMD) for a mix of numeric and categorical variables; and categorical PCA (CATPCA) for ordinal data. For prediction tasks, embeddings or target encoding are often better than any linear reduction, and ordinal codes like 1, 2, 3 should only be used when the order and spacing are meaningful.

Read full answer

Applies toNumPy 2.x, pandas 2.x, scikit-learn 1.5+ (the prince library implements MCA and FAMD)

Options

Data Method Idea
Nominal categories only MCA Correspondence analysis of the indicator matrix
Two categorical variables Correspondence analysis SVD of standardized contingency residuals
Mixed numeric and categorical FAMD Scale numeric columns, weight dummies, then PCA
Ordinal CATPCA, or careful integer codes Optimal scaling of levels
High-cardinality categories Embeddings, target encoding Learned or supervised representations

Code Example: Correspondence Analysis from Scratch

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n = 2000
region = rng.choice(["north", "south", "east"], n, p=[0.5, 0.3, 0.2])
pref = np.where(region == "north", rng.choice(["tea", "coffee", "juice"], n, p=[0.7, 0.2, 0.1]),
                np.where(region == "south", rng.choice(["tea", "coffee", "juice"], n, p=[0.2, 0.7, 0.1]),
                         rng.choice(["tea", "coffee", "juice"], n, p=[0.3, 0.3, 0.4])))
table = pd.crosstab(region, pref)
print(table)

P = table.to_numpy() / table.to_numpy().sum()
r, c = P.sum(axis=1), P.sum(axis=0)
S = (P - np.outer(r, c)) / np.sqrt(np.outer(r, c))         # standardized residuals
U, s, Vt = np.linalg.svd(S, full_matrices=False)
inertia = s**2 / np.sum(s**2)
print("share of inertia per dimension:", inertia.round(3))  # a 3x3 table has at most 2 dimensions

row_coords = (U[:, :2] * s[:2]) / np.sqrt(r)[:, None]
col_coords = (Vt[:2].T * s[:2]) / np.sqrt(c)[:, None]
print(pd.DataFrame(row_coords, index=table.index, columns=["dim1", "dim2"]).round(3))
print(pd.DataFrame(col_coords, index=table.columns, columns=["dim1", "dim2"]).round(3))
# North sits near tea and south near coffee on dimension 1; east sits nearer juice
chi2 = n * np.sum(s**2)                                   # total inertia times n is the chi-squared statistic
print(f"chi-squared = {chi2:.1f}")

Common Pitfalls

  • Treating arbitrary category codes (1 = red, 2 = blue) as numbers.

  • Running plain PCA on one-hot data and over-interpreting it.

  • Forgetting that rare categories can dominate MCA dimensions; merge very rare levels.

Interview Tips

  • Explain why variance and distance don’t fit nominal data.

  • Name MCA, FAMD and CATPCA, and embeddings for modelling.

25

What are the pros and cons of kernel PCA versus linear PCA?

Kernel PCA performs PCA in a high-dimensional feature space φ(x) without ever computing φ explicitly: it builds the n × n kernel matrix Kᵢⱼ = k(xᵢ, xⱼ) (for example an RBF kernel exp(−γ‖xᵢ − xⱼ‖²)), centres it, and takes its top eigenvectors, which lets it capture non-linear structure such as concentric circles or curved manifolds that linear PCA cannot. The drawbacks follow from the same construction: memory and time grow with n² and n³ rather than with the number of features, results depend heavily on the kernel and its parameters (γ, degree) which must be tuned, components are no longer combinations of the original features so there are no interpretable loadings, mapping back to the input space only gives an approximate “pre-image”, and explained variance in feature space is not comparable with linear PCA’s. Linear PCA stays the default for speed, interpretability and exact reconstruction; kernel PCA is for moderate-sized data with clear non-linear structure, with Nyström approximations for larger n.

Read full answer

Applies toscikit-learn 1.5+

Comparison

Linear PCA Kernel PCA
Structure captured Linear Non-linear (via kernel)
Cost O(nd² ) or O(n d k) with SVD O(n²) memory, O(n³) time
Hyperparameters k k, kernel, γ/degree
Interpretability Loadings on original features None directly
Inverse transform Exact (for kept components) Approximate pre-image (learned)
New data Fast projection Kernel against all training points
Scale Millions of rows Thousands (Nyström for more)

Code Example

import time

import numpy as np
from sklearn.datasets import make_moons
from sklearn.decomposition import PCA, KernelPCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score
from sklearn.pipeline import Pipeline

X, y = make_moons(n_samples=1000, noise=0.08, random_state=42)

linear = Pipeline([("pca", PCA(n_components=2)), ("clf", LogisticRegression())])
kernel = Pipeline([("kpca", KernelPCA(n_components=2, kernel="rbf")), ("clf", LogisticRegression())])
search = GridSearchCV(kernel, {"kpca__gamma": [0.5, 2, 5, 15]}, cv=5).fit(X, y)
print(f"linear PCA + logistic: {cross_val_score(linear, X, y, cv=5).mean():.3f}")   # about 0.88
print(f"kernel PCA + logistic: {search.best_score_:.3f} with gamma={search.best_params_['kpca__gamma']}")

rng = np.random.default_rng(42)
for n in [1000, 4000]:                                    # cost grows quickly with n
    Xn = rng.normal(size=(n, 10))
    start = time.perf_counter()
    KernelPCA(n_components=2, kernel="rbf", eigen_solver="dense").fit(Xn)
    print(f"n={n}: kernel PCA fit {time.perf_counter() - start:.2f}s, kernel matrix {n * n * 8 / 1e6:.0f} MB")

kpca = KernelPCA(n_components=2, kernel="rbf", gamma=15, fit_inverse_transform=True).fit(X)
X_back = kpca.inverse_transform(kpca.transform(X))
print(f"approximate pre-image error: {np.mean((X - X_back) ** 2):.4f}")

The 4000-point kernel matrix takes about 128 MB, sixteen times the 1000-point one, while linear PCA’s cost barely changes.

Common Pitfalls

  • Using kernel PCA on large n without approximation (memory errors).

  • Not tuning γ; too small behaves like linear PCA, too large overfits.

  • Expecting loadings or exact reconstructions.

Interview Tips

  • Explain the kernel trick and the n × n kernel matrix.

  • Weigh non-linearity against cost, tuning and interpretability.

26

Write a Python function to perform PCA from scratch with NumPy

A from-scratch PCA needs four steps: centre the data by subtracting each feature’s mean (and optionally divide by its standard deviation), compute the singular value decomposition of the centred matrix (or the eigen-decomposition of its covariance), take the top k right singular vectors as components, and project the centred data onto them; the explained variances are the squared singular values divided by n − 1. A good implementation stores the mean so new data can be transformed the same way, fixes the arbitrary signs of the components so results are reproducible, offers an inverse transform for reconstruction, and is checked against sklearn.decomposition.PCA, which should agree up to the sign of each component.

Read full answer

Applies toPython 3.12, NumPy 2.x, scikit-learn 1.5+

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA as SklearnPCA
from sklearn.preprocessing import StandardScaler

class PCA:
    """Minimal PCA via SVD of the centred data."""

    def __init__(self, n_components=None):
        self.n_components = n_components

    def fit(self, X):
        X = np.asarray(X, dtype=float)
        n, d = X.shape
        self.mean_ = X.mean(axis=0)
        Xc = X - self.mean_
        U, s, Vt = np.linalg.svd(Xc, full_matrices=False)
        max_abs_cols = np.argmax(np.abs(U), axis=0)          # deterministic signs (like sklearn's svd_flip)
        signs = np.sign(U[max_abs_cols, range(U.shape[1])])
        Vt = Vt * signs[:, None]
        k = self.n_components or min(n, d)
        self.components_ = Vt[:k]
        self.explained_variance_ = (s[:k] ** 2) / (n - 1)
        self.explained_variance_ratio_ = (s[:k] ** 2) / np.sum(s ** 2)
        self.singular_values_ = s[:k]
        return self

    def transform(self, X):
        return (np.asarray(X, dtype=float) - self.mean_) @ self.components_.T

    def fit_transform(self, X):
        return self.fit(X).transform(X)

    def inverse_transform(self, Z):
        return Z @ self.components_ + self.mean_

X = StandardScaler().fit_transform(load_iris().data)
mine = PCA(n_components=2).fit(X)
ref = SklearnPCA(n_components=2, svd_solver="full").fit(X)

print("ratios:", mine.explained_variance_ratio_.round(4))                            # [0.7296 0.2285]
print("variances match:", np.allclose(mine.explained_variance_, ref.explained_variance_))   # True
signs = np.sign(np.sum(mine.components_ * ref.components_, axis=1))
print("components match up to sign:", np.allclose(mine.components_ * signs[:, None], ref.components_))   # True
print("projections match:", np.allclose(mine.transform(X) * signs, ref.transform(X)))      # True

X_back = mine.inverse_transform(mine.transform(X))
print(f"reconstruction MSE with 2 components: {np.mean((X - X_back) ** 2):.4f}")
# equals the discarded share of variance: about (1 - 0.9581) of the total

The reconstruction error per entry is the discarded variance (about 4.2% of the total of 4 features of unit variance), so it prints about 0.042.

Design Notes

Choice Reason
SVD instead of covariance eigen-decomposition Better numerical stability, no d × d matrix
Store mean_ Same centring for new data
Sign convention Reproducible output
ddof = n − 1 Matches scikit-learn’s explained variance

Common Pitfalls

  • Forgetting to centre, or centring test data with its own mean.

  • Using np.linalg.eig on the covariance and forgetting to sort.

  • Comparing component signs directly with another library.

Interview Tips

  • Walk through centre → SVD → components → projection.

  • Mention checking against scikit-learn and the sign ambiguity.

27

Apply PCA with scikit-learn to a high-dimensional dataset and interpret it

To apply PCA to a high-dimensional dataset with scikit-learn, put StandardScaler and PCA in a Pipeline, fit it on training data, and interpret the result in three layers: the explained variance ratio (how many components carry most of the structure), the loadings in components_ (which original features define each component), and the scores (where samples sit, which reveals groups, gradients and outliers). For example, on the 30-feature breast cancer dataset the first component explains about 44% of the variance and loads on almost every size and shape feature with the same sign, so it measures “overall tumour size and irregularity”; together the first two components separate malignant and benign tumours well, and about 10 components keep 95% of the variance. The interpretation should be checked against the target and against stability (does a bootstrap give the same loadings?).

Read full answer

Applies toscikit-learn 1.5+, pandas 2.x

Code Example

import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X, y = data.data, data.target                                 # 569 x 30
pipe = make_pipeline(StandardScaler(), PCA()).fit(X)
pca = pipe[-1]

ratio = pca.explained_variance_ratio_
print("first five ratios:", ratio[:5].round(3))              # about [0.443 0.19 0.094 0.066 0.055]
print("components for 95% variance:", int(np.searchsorted(np.cumsum(ratio), 0.95) + 1))   # 10

loadings = pd.DataFrame(pca.components_[:2].T, index=data.feature_names, columns=["PC1", "PC2"])
print(loadings["PC1"].abs().sort_values(ascending=False).head(5).round(3))
print(loadings["PC2"].sort_values().head(3).round(3), loadings["PC2"].sort_values().tail(3).round(3))
# PC1: concave points, concavity, perimeter, area (size and irregularity)
# PC2: contrasts radius/area with fractal dimension and smoothness (size vs texture detail)

scores = pipe.transform(X)[:, :2]
for label, name in enumerate(data.target_names):
    print(f"{name:9s} mean PC1 score {scores[y == label, 0].mean():.2f}")   # malignant high, benign low

for k in [2, 5, 10]:
    model = make_pipeline(StandardScaler(), PCA(n_components=k), LogisticRegression(max_iter=1000))
    print(f"k={k:>2}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")

How to Interpret

Output Question it answers
explained_variance_ratio_ How low-dimensional is the data?
components_ (loadings) What does each component mean?
Scores (transform) How do samples or classes spread out?
Downstream CV score Does the reduction keep what matters?

Common Pitfalls

  • Interpreting signs as meaningful in isolation (the whole component can flip).

  • Naming components from one or two loadings without looking at the full pattern.

  • Fitting PCA on all data and then reporting test accuracy.

Interview Tips

  • Walk through variance, loadings and scores with a named dataset.

  • Mention checking with a downstream model and loading stability.

28

Write a script to visualize eigenfaces with PCA

Eigenfaces are the principal components of a set of aligned face images: flatten each grayscale image into a vector, fit PCA (usually randomized PCA with whitening, because there are thousands of pixel features), and reshape each component back to the image size to see it as a face-like pattern; the first eigenfaces capture lighting and overall face shape, later ones finer features such as eyes, nose and mouth. Every face is then approximately the mean face plus a weighted sum of eigenfaces, so a script typically shows the mean face, a grid of the top eigenfaces, the explained variance curve, and reconstructions of a face with increasing numbers of components; the component scores can also feed a classifier for face recognition. The classic example uses scikit-learn’s Labeled Faces in the Wild (LFW) dataset, which downloads about 200 MB on first use.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import fetch_lfw_people
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split

faces = fetch_lfw_people(min_faces_per_person=70, resize=0.4)   # downloads on first run
X, y = faces.data, faces.target
h, w = faces.images.shape[1:]
print(f"{X.shape[0]} images of {h}x{w} = {X.shape[1]} pixels, {len(faces.target_names)} people")

X_train, X_test = train_test_split(X, test_size=0.25, random_state=42, stratify=y)
n_components = 150
pca = PCA(n_components=n_components, svd_solver="randomized", whiten=True, random_state=42).fit(X_train)
print(f"{n_components} components keep {pca.explained_variance_ratio_.sum():.1%} of the pixel variance")

fig, axes = plt.subplots(3, 5, figsize=(10, 7))
axes[0, 0].imshow(pca.mean_.reshape(h, w), cmap="gray")
axes[0, 0].set_title("mean face")
for i, ax in enumerate(axes.flat[1:]):
    ax.imshow(pca.components_[i].reshape(h, w), cmap="gray")
    ax.set_title(f"eigenface {i + 1}")
for ax in axes.flat:
    ax.axis("off")
fig.tight_layout()
fig.savefig("eigenfaces.png", dpi=120)

face = X_test[:1]
fig, axes = plt.subplots(1, 5, figsize=(10, 3))
axes[0].imshow(face.reshape(h, w), cmap="gray")
axes[0].set_title("original")
for ax, k in zip(axes[1:], [10, 30, 75, 150]):
    partial = PCA(n_components=k, svd_solver="randomized", random_state=42).fit(X_train)
    recon = partial.inverse_transform(partial.transform(face))
    ax.imshow(recon.reshape(h, w), cmap="gray")
    ax.set_title(f"{k} components")
for ax in axes:
    ax.axis("off")
fig.tight_layout()
fig.savefig("eigenface_reconstructions.png", dpi=120)

fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(np.cumsum(PCA(n_components=300, svd_solver="randomized", random_state=42)
                  .fit(X_train).explained_variance_ratio_))
ax.set(xlabel="number of eigenfaces", ylabel="cumulative explained variance")
fig.tight_layout()
fig.savefig("eigenface_variance.png", dpi=120)

The reconstructions use separate non-whitened PCA models because a whitened model’s scores are rescaled; inverse_transform undoes whitening in scikit-learn, but fitting plain models keeps the example easy to follow.

Common Pitfalls

  • Unaligned or differently sized faces; eigenfaces need cropped, aligned images.

  • Lighting dominating the first components; drop the first few or normalize lighting for recognition.

  • Plotting components without reshaping to (h, w) or with a colour map that hides the sign.

Interview Tips

  • Explain faces as mean plus weighted eigenfaces.

  • Mention randomized PCA, whitening and using scores for recognition.

29

Implement PCA for feature extraction before a machine learning model

To use PCA for feature extraction, build a scikit-learn Pipeline of scaling, PCA and the model, so the scaler and PCA are fitted only on each training fold and the same transformation is applied to validation and test data; then tune the number of components together with the model’s hyperparameters using cross-validation, and compare against the same model without PCA to confirm it actually helps. PCA tends to help models that suffer from many correlated or noisy inputs (logistic regression, SVMs, kNN) by reducing variance and training time, and helps less for tree ensembles. The final report should give the chosen k, the variance it keeps, the cross-validated and test scores with and without PCA, and the training-time savings.

Read full answer

Applies toscikit-learn 1.5+

Code Example

import time

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

pipe = Pipeline([("scale", StandardScaler()), ("pca", PCA(random_state=42)), ("svm", SVC())])
grid = {
    "pca__n_components": [10, 20, 30, 40, 0.95],
    "svm__C": [1, 10],
    "svm__gamma": ["scale", 0.01],
}
search = GridSearchCV(pipe, grid, cv=5, n_jobs=-1).fit(X_train, y_train)
best = search.best_estimator_
print("best params:", search.best_params_)
print(f"CV accuracy {search.best_score_:.3f}, test accuracy {best.score(X_test, y_test):.3f}")
print(f"components kept: {best.named_steps['pca'].n_components_}, "
      f"variance kept: {best.named_steps['pca'].explained_variance_ratio_.sum():.1%}")

baseline = GridSearchCV(Pipeline([("scale", StandardScaler()), ("svm", SVC())]),
                        {"svm__C": [1, 10], "svm__gamma": ["scale", 0.01]}, cv=5, n_jobs=-1).fit(X_train, y_train)
print(f"without PCA: CV {baseline.best_score_:.3f}, test {baseline.best_estimator_.score(X_test, y_test):.3f}")

for name, model in [("with PCA", best), ("without PCA", baseline.best_estimator_)]:
    start = time.perf_counter()
    model.fit(X_train, y_train)
    print(f"{name}: refit time {time.perf_counter() - start:.3f}s")

On digits both versions reach about 98% accuracy; the PCA version uses far fewer inputs, which matters more as datasets grow.

Pipeline Rules

Rule Why
Scaler and PCA inside the Pipeline No leakage from validation folds
Tune k with the model The best k depends on the model
Compare with no PCA PCA is not always an improvement
Keep the test set untouched until the end Honest final estimate

Common Pitfalls

  • PCA().fit(X_all) before splitting.

  • Fixing k at 95% variance without checking alternatives.

  • Adding PCA in front of random forests or boosting out of habit.

Interview Tips

  • Show the Pipeline and tuning k with cross-validation.

  • Report the comparison with and without PCA.

30

How would you use PCA for compression in a real-time streaming system?

In a streaming system you can’t refit PCA on all the data, so learn the components incrementally and compress each record on arrival: fit an initial basis on a warm-up batch (or use IncrementalPCA.partial_fit on mini-batches, or an online algorithm such as Oja’s rule or frequent directions), then encode each new record as z = W_kᵀ(x − μ), sending only the k numbers plus a basis version number, and decode on the receiver with x̂ = μ + W_kz. Because the data distribution drifts, monitor the reconstruction error of incoming records; when it rises, update the basis with recent data, version it, and send the new basis to decoders before it is used. Latency is low because encoding is one d × k matrix-vector product. Decide k by the error budget (for example 95% of variance or a maximum per-record error), quantize the coefficients to save more space, and flag records whose error is unusually high, because they are often anomalies worth keeping in full.

Read full answer

Applies tosystem design; example with scikit-learn 1.5+

Design

Component Choice
Basis learning IncrementalPCA.partial_fit on mini-batches, or a sliding window refit
Encoding z = W_kᵀ(x − μ), O(dk) per record
Transmission k coefficients (optionally quantized to 8–16 bits) + basis version
Decoding x̂ = μ + W_kz
Drift monitoring Rolling reconstruction error; refit trigger
Basis updates Versioned, sent ahead of use; old versions kept for replay
Anomalies High-error records stored raw

Code Example

import numpy as np
from sklearn.decomposition import IncrementalPCA

rng = np.random.default_rng(42)
d, k, batch = 100, 8, 200
basis_a = rng.normal(size=(k, d))
basis_b = rng.normal(size=(k, d))                          # the process changes later

def stream(n_batches, basis):
    for _ in range(n_batches):
        yield rng.normal(size=(batch, k)) @ basis + rng.normal(0, 0.1, (batch, d))

ipca = IncrementalPCA(n_components=k)
for chunk in stream(10, basis_a):                          # warm-up
    ipca.partial_fit(chunk)
version = 1

def encode(x):
    return (ipca.transform(x).astype(np.float16), version)  # 8 half-precision numbers instead of 100 floats

def recon_error(x):
    z = ipca.transform(x)
    return float(np.mean((x - ipca.inverse_transform(z)) ** 2))

baseline_error = recon_error(next(stream(1, basis_a)))
print(f"compression {d * 8 / (k * 2):.0f}x (float64 -> {k} float16 values), baseline error {baseline_error:.4f}")

for step, chunk in enumerate(stream(6, basis_b)):
    err = recon_error(chunk)
    drift = err > 3 * baseline_error
    print(f"batch {step}: error {err:.4f}{'  -> drift, updating basis' if drift else ''}")
    if drift:
        ipca = IncrementalPCA(n_components=k)              # refit on recent data and bump the version
        ipca.partial_fit(chunk)
        version += 1
        baseline_error = recon_error(chunk)
payload, v = encode(next(stream(1, basis_b))[:1])
print("payload shape", payload.shape, "basis version", v)

The first batch after the change shows a large error and triggers a refit; later batches compress well again with basis version 2.

Common Pitfalls

  • Updating the basis silently, so the decoder reconstructs with the wrong components.

  • No drift monitoring, so quality decays unnoticed.

  • Ignoring per-feature scaling that also drifts; update the scaler along with PCA.

  • Using PCA where a domain codec (audio, video) would do much better.

Interview Tips

  • Describe incremental fitting, encode/decode and basis versioning.

  • Mention drift detection with reconstruction error and quantization.

31

How would you apply PCA to stock market data?

In stock market analysis, PCA is applied to the matrix of daily returns (days × stocks), usually standardized, so the components are the main independent patterns that move many stocks together: the first component is almost always the “market factor”, with every stock loading with the same sign and explaining a large share of the variance (often 30–50%), and later components often line up with sectors or styles such as tech versus utilities or growth versus value. This is used for risk models (a portfolio’s exposure to each component, and how concentrated its risk is), statistical arbitrage (trading the residual after removing the top components, which tends to mean-revert), building factor-neutral portfolios, reducing noise in the covariance matrix used for portfolio optimization, and yield-curve analysis, where three components (level, slope, curvature) explain almost all bond yield moves. Work on returns, not prices (prices are non-stationary), use rolling windows because the structure changes over time (correlations jump in crises), and test the results out of sample.

Read full answer

Applies toNumPy 2.x, pandas 2.x, scikit-learn 1.5+

Code Example (Simulated Returns)

import numpy as np
import pandas as pd
from sklearn.decomposition import PCA

rng = np.random.default_rng(42)
days = 750
sectors = {"tech": 6, "banks": 5, "energy": 4}
market = rng.normal(0, 0.010, days)
sector_moves = {s: rng.normal(0, 0.006, days) for s in sectors}
columns, data = [], []
for sector, count in sectors.items():
    for i in range(count):
        beta = rng.uniform(0.8, 1.2)
        data.append(beta * market + sector_moves[sector] + rng.normal(0, 0.008, days))
        columns.append(f"{sector}_{i}")
returns = pd.DataFrame(np.column_stack(data), columns=columns)

z = (returns - returns.mean()) / returns.std()
pca = PCA().fit(z)
print("explained variance ratio:", pca.explained_variance_ratio_[:5].round(3))
pc1 = pd.Series(pca.components_[0], index=columns)
print("PC1 loadings all one sign (market factor):", bool((pc1 > 0).all() or (pc1 < 0).all()))
for k in [1, 2]:
    loadings = pd.Series(pca.components_[k], index=columns)
    print(f"PC{k + 1} mean loading by sector:", loadings.groupby(lambda c: c.split("_")[0]).mean().round(2).to_dict())

window, k = 250, 3                                       # rolling PCA residuals for stat-arb style signals
train = z.iloc[:window]
basis = PCA(n_components=k).fit(train)
test = z.iloc[window:window + 20]
residual = test - basis.inverse_transform(basis.transform(test))
print("residual sd vs raw sd:", round(float(residual.to_numpy().std()), 3), round(float(test.to_numpy().std()), 3))

With one market factor and three sectors in the simulation, the first component explains the largest share and the next two separate the sectors; real data looks similar but noisier.

Uses

Use How PCA helps
Risk decomposition Exposure to each component; how much risk is “market”
Statistical arbitrage Trade residuals after removing top components
Covariance cleaning Keep top components, shrink the rest
Yield curves Level, slope and curvature factors
Market regime monitoring Rising PC1 share signals correlated, stressed markets

Common Pitfalls

  • PCA on prices instead of returns.

  • Look-ahead bias: fitting on the full period and backtesting on it.

  • Assuming components are stable; they rotate across regimes.

  • Over-interpreting later components as economic factors; check loadings and stability.

Interview Tips

  • Name the market factor and sector components.

  • Mention returns not prices, rolling windows and out-of-sample testing.

32

When can PCA hurt a machine learning model's performance?

PCA can hurt a model when the information that predicts the target lies in low-variance directions that PCA throws away: it chooses components by variance without looking at the label, so a small but decisive signal (for example a tiny voltage difference that marks faulty parts, or a rare word that marks fraud) can be discarded while large but irrelevant variation (lighting, scale, noise in one sensor) is kept. Other cases are tree-based models, which split on single features and can lose performance when PCA mixes features into rotated combinations; non-linear structure that a linear projection flattens; features with different units used without scaling, so one feature dominates; outliers that rotate the components; sparse data where centring destroys sparsity; and loss of interpretability when explanations per original feature are required (credit decisions, medicine). Always compare the model with and without PCA in cross-validation.

Read full answer

Applies toscikit-learn 1.5+

Code Example: A Small but Important Feature

import numpy as np
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)
n = 2000
y = rng.integers(0, 2, n)
nuisance = rng.normal(0, 5, (n, 10))                                 # big variance, no signal
signal = (y - 0.5) * 0.4 + rng.normal(0, 0.1, n)                     # small variance, decisive
X = np.column_stack([nuisance, signal])

for name, model in [
    ("logistic, all features", LogisticRegression(max_iter=2000)),
    ("logistic, PCA to 5", make_pipeline(PCA(n_components=5), LogisticRegression(max_iter=2000))),
    ("forest, raw features", RandomForestClassifier(n_estimators=200, random_state=42)),
    ("forest, PCA to 5", make_pipeline(PCA(n_components=5), RandomForestClassifier(n_estimators=200, random_state=42))),
]:
    print(f"{name:24s}: accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
# Models on raw features are near-perfect; after PCA to 5 components they fall to about chance,
# because the signal direction has the smallest variance and is dropped

When to Avoid or Be Careful

Situation Why PCA can hurt
Signal in low-variance directions Dropped by PCA
Tree-based models Rotations break axis-aligned splits
Non-linear class boundaries Linear projection may merge classes
Interpretability required Components mix all features
Sparse high-dimensional data Centring densifies; use TruncatedSVD or none
Mixed units without scaling Dominated by large-unit features

Common Pitfalls

  • Adding PCA by default as a “preprocessing step”.

  • Choosing k by variance alone.

  • Evaluating the PCA pipeline but not the baseline.

Interview Tips

  • Give the low-variance-signal scenario and the tree-model scenario.

  • Recommend a with/without comparison and supervised alternatives (LDA, PLS).

33

How do you make PCA results robust to variations in the data?

PCA results are robust when the components you report don’t change much if the data is resampled, slightly perturbed or collected at a different time, so you should test that directly and design for it: bootstrap or cross-validate the PCA and measure how similar the components and their explained variance stay (using absolute cosine similarity because signs flip, or subspace angles for components with similar eigenvalues), and only interpret components that are stable. Practical safeguards are consistent preprocessing (the same scaling fitted on training data, handled missing values), robust methods when outliers exist (RobustScaler, MCD-based or robust PCA), enough samples relative to features, and checking the eigenvalue gaps: components whose eigenvalues are nearly equal can rotate freely into each other, so treat them as a shared subspace rather than as individual axes. For data that changes over time, refit on rolling windows and track drift in the components and in reconstruction error.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Code Example: Bootstrap Stability of Components

import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X = StandardScaler().fit_transform(load_wine().data)
k = 4
reference = PCA(n_components=k).fit(X)
rng = np.random.default_rng(42)

similarities, ratios = [], []
for _ in range(500):
    sample = X[rng.integers(0, len(X), len(X))]
    boot = PCA(n_components=k).fit(sample)
    sims = np.abs(np.sum(boot.components_ * reference.components_, axis=1))   # |cosine|, sign-free
    similarities.append(sims)
    ratios.append(boot.explained_variance_ratio_)

similarities, ratios = np.array(similarities), np.array(ratios)
for i in range(k):
    lo, hi = np.percentile(ratios[:, i], [2.5, 97.5])
    print(f"PC{i + 1}: median |cos| with reference {np.median(similarities[:, i]):.3f}, "
          f"5th percentile {np.percentile(similarities[:, i], 5):.3f}, variance ratio 95% CI [{lo:.3f}, {hi:.3f}]")
# PC1 and PC2 are very stable; later components with close eigenvalues vary more

def subspace_similarity(A, B):
    """Mean squared cosine of principal angles between two k-dim subspaces (1 = identical)."""
    s = np.linalg.svd(A @ B.T, compute_uv=False)
    return float(np.mean(s**2))

sample = X[rng.integers(0, len(X), len(X))]
print("subspace similarity of PCs 3-4:", round(subspace_similarity(PCA(4).fit(sample).components_[2:],
                                                               reference.components_[2:]), 3))

Robustness Checklist

Check or safeguard Purpose
Bootstrap absolute cosine of components Are loadings stable?
Eigenvalue gaps Are components individually identifiable?
Subspace angles Stability of a group of components
Same preprocessing pipeline Avoids spurious differences
Robust scaling / robust PCA Outlier resistance
Rolling refits and drift monitoring Changing data over time
Sample size n well above d for stable covariance estimates

Common Pitfalls

  • Comparing components by raw dot products and calling sign flips “instability”.

  • Interpreting individual components with nearly equal eigenvalues.

  • Reporting loadings from one fit without any uncertainty.

Interview Tips

  • Describe bootstrapping with sign-free similarity and eigenvalue gaps.

  • Mention robust methods and monitoring over time.

34

How can PCA be parallelized for very large datasets?

PCA parallelizes well because its core computations are sums over rows: each worker can compute partial statistics on its partition (row count, column sums, and the d × d Gram matrix XᵢᵀXᵢ), the driver adds them up to get the global mean and covariance, and a single small eigen-decomposition gives the components, which works whenever the number of features d is moderate (up to a few thousand) no matter how many rows there are. When d is also large, use distributed or randomized SVD: randomized PCA multiplies the data by a random matrix and needs only a few passes of parallel matrix products (as in Dask or Spark MLlib); tall-and-skinny QR (TSQR) combines per-partition QR factorizations; and incremental or streaming methods (IncrementalPCA, frequent directions) process chunks so memory stays bounded. On a single machine, the BLAS libraries behind NumPy already use many cores, and GPUs (cuML, PyTorch torch.pca_lowrank) speed up the matrix products dramatically.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+; Spark MLlib, Dask-ML and cuML follow the same patterns

Strategies

Strategy Pattern Best when
Distributed covariance Map: partial XᵀX and sums; reduce: add; eigh on d × d Huge n, moderate d
Randomized SVD Few passes of parallel matrix products Large n and d, top k only
TSQR QR per block, then QR of the stacked R factors Tall, skinny matrices
Incremental PCA Chunk-by-chunk updates Out-of-core, single machine
GPU Dense matrix multiplications on GPU Medium-large dense data

Code Example: Map-Reduce PCA with a Process Pool

from concurrent.futures import ProcessPoolExecutor

import numpy as np
from sklearn.decomposition import PCA

def partial_stats(chunk):
    """Map step: sufficient statistics for one partition."""
    return len(chunk), chunk.sum(axis=0), chunk.T @ chunk

def distributed_pca(chunks, k, workers=4):
    with ProcessPoolExecutor(max_workers=workers) as pool:
        results = list(pool.map(partial_stats, chunks))
    n = sum(r[0] for r in results)                       # reduce step
    total = sum(r[1] for r in results)
    gram = sum(r[2] for r in results)
    mean = total / n
    cov = (gram - n * np.outer(mean, mean)) / (n - 1)
    vals, vecs = np.linalg.eigh(cov)
    order = np.argsort(vals)[::-1][:k]
    return mean, vecs[:, order].T, vals[order]

if __name__ == "__main__":
    rng = np.random.default_rng(42)
    X = rng.normal(size=(200_000, 30)) @ rng.normal(size=(30, 30))
    chunks = np.array_split(X, 16)                       # stand-ins for partitions on different machines
    mean, components, variances = distributed_pca(chunks, k=5)

    ref = PCA(n_components=5).fit(X)
    signs = np.sign(np.sum(components * ref.components_, axis=1))
    print("components match:", np.allclose(components * signs[:, None], ref.components_, atol=1e-6))
    print("variances match:", np.allclose(variances, ref.explained_variance_))

    rand = PCA(n_components=5, svd_solver="randomized", random_state=42).fit(X)
    print("randomized variances close:", np.allclose(rand.explained_variance_, ref.explained_variance_, rtol=1e-3))

The covariance-from-Gram formula can lose precision when the mean is large relative to the spread; shifting each partition by a rough global mean first, or using pairwise (Chan’s) updates, avoids that.

Common Pitfalls

  • Collecting all data on the driver instead of only the d × d statistics.

  • Building a d × d matrix when d is in the hundreds of thousands; use randomized or sparse methods.

  • Numerical cancellation in the one-pass covariance formula.

  • Shuffling data across the network more than needed.

Interview Tips

  • Explain the map-reduce of sufficient statistics and when it works (moderate d).

  • Mention randomized SVD, TSQR, incremental PCA and GPUs.

35

What are best practices for visualizing PCA components?

Good PCA visualizations make three things clear: how much variance each component explains (a scree plot with per-component and cumulative bars), where samples fall on the main components (a scores scatter plot coloured by a meaningful variable, with axes labelled “PC1 (44.3%)” and equal scaling), and what each component means in terms of the original features (a loadings bar chart or heatmap sorted by magnitude, or a biplot that overlays the strongest loading arrows on the scores). Use diverging colour maps centred on zero for loadings, show only the top features or components so plots stay readable, keep the sign convention consistent across figures, and for image or signal data display components in their natural shape (eigenfaces as images, spectral components as curves). Adding uncertainty, for example bootstrap intervals on loadings, prevents over-reading small differences.

Read full answer

Applies toscikit-learn 1.5+, Matplotlib 3.8+

Plot Types

Plot Shows Tip
Scree plot Variance per component, cumulative Mark the chosen k
Scores scatter Samples on PC1 vs PC2 Label axes with %; equal aspect
Loadings bar chart Features driving one component Sort by absolute value
Loadings heatmap Features × components Diverging map centred at 0
Biplot Scores plus loading arrows Show only the top features
Component as image/curve Pixel or spectral components Natural shape of the data

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

data = load_breast_cancer()
X = StandardScaler().fit_transform(data.data)
pca = PCA(n_components=6).fit(X)
ratio = pca.explained_variance_ratio_
scores = pca.transform(X)

fig, axes = plt.subplots(1, 3, figsize=(16, 5))
axes[0].bar(range(1, 7), ratio, label="per component")
axes[0].plot(range(1, 7), np.cumsum(ratio), marker="o", color="black", label="cumulative")
axes[0].set(xlabel="component", ylabel="explained variance ratio", title="Scree plot")
axes[0].legend()

for label, marker in [(0, "o"), (1, "s")]:
    pts = scores[data.target == label]
    axes[1].scatter(pts[:, 0], pts[:, 1], s=12, alpha=0.6, marker=marker, label=data.target_names[label])
axes[1].set(xlabel=f"PC1 ({ratio[0]:.1%})", ylabel=f"PC2 ({ratio[1]:.1%})", title="Scores")
axes[1].set_aspect("equal", adjustable="datalim")
axes[1].legend()

top = np.argsort(-np.abs(pca.components_[:3]).max(axis=0))[:12]       # 12 strongest features
limit = np.abs(pca.components_[:3, top]).max()
im = axes[2].imshow(pca.components_[:3, top].T, cmap="RdBu_r", vmin=-limit, vmax=limit, aspect="auto")
axes[2].set_yticks(range(len(top)), [data.feature_names[i] for i in top], fontsize=8)
axes[2].set_xticks(range(3), ["PC1", "PC2", "PC3"])
axes[2].set_title("Loadings (top features)")
fig.colorbar(im, ax=axes[2], shrink=0.8)
fig.tight_layout()
fig.savefig("pca_components.png", dpi=120)

Common Pitfalls

  • Unlabelled axes or missing explained variance percentages.

  • Non-equal axis scales that exaggerate spread along PC2.

  • Sequential colour maps for signed loadings.

  • Biplots with all 30 arrows, which become unreadable.

Interview Tips

  • Name scree, scores, loadings and biplots and what each answers.

  • Mention diverging colour maps, equal aspect and uncertainty.

36

How does PCA compare with other feature selection methods?

PCA is feature extraction, not feature selection: it builds new features as linear combinations of all the original ones, so it reduces dimensionality and removes redundancy but every original feature is still needed at prediction time and the new features are hard to explain; feature selection keeps a subset of the original columns, which preserves their meaning and can cut data collection costs. Selection methods come in three families: filter methods score each feature independently of a model (variance threshold, correlation, mutual information, chi-squared), wrapper methods search subsets with a model (recursive feature elimination, sequential selection), and embedded methods select during training (L1/Lasso, tree feature importances). PCA is unsupervised, so it may keep irrelevant high-variance directions, while most selection methods use the target. Choose PCA for compression and highly correlated inputs, and selection when interpretability, cost or using the target matters; they can be combined.

Read full answer

Applies toscikit-learn 1.5+

Comparison

Method Type Uses y Keeps original features Cost
PCA Extraction, unsupervised No No (combinations) Low
Variance threshold Filter No Yes Very low
Mutual information / chi² / ANOVA F Filter Yes Yes Low
RFE / sequential selection Wrapper Yes Yes High
L1 (Lasso, L1 logistic) Embedded Yes Yes Medium
Tree importances / permutation Embedded Yes Yes Medium

Code Example

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.feature_selection import RFE, SelectFromModel, SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
clf = LogisticRegression(max_iter=5000)
k = 5

candidates = {
    "all 30 features": make_pipeline(StandardScaler(), clf),
    "PCA (5 components)": make_pipeline(StandardScaler(), PCA(n_components=k), clf),
    "mutual information (5)": make_pipeline(StandardScaler(), SelectKBest(mutual_info_classif, k=k), clf),
    "RFE (5)": make_pipeline(StandardScaler(), RFE(LogisticRegression(max_iter=5000), n_features_to_select=k), clf),
    "L1 selection": make_pipeline(
        StandardScaler(),
        SelectFromModel(LogisticRegression(penalty="l1", solver="liblinear", C=0.1)),
        clf),
}
for name, model in candidates.items():
    print(f"{name:24s}: CV accuracy {cross_val_score(model, X, y, cv=5).mean():.3f}")
# All reach roughly 0.94-0.98; selection methods do it with 5 real, nameable features

rfe = make_pipeline(StandardScaler(), RFE(LogisticRegression(max_iter=5000), n_features_to_select=k)).fit(X, y)
names = load_breast_cancer().feature_names
print("RFE keeps:", list(names[rfe[-1].support_]))

When to Choose Which

Goal Better choice
Compress correlated features PCA
Explain the model with original features Selection
Reduce the cost of collecting features Selection
Remove noise in a block of sensors PCA
Very high-dimensional sparse text Selection (chi²) or TruncatedSVD

Common Pitfalls

  • Calling PCA “feature selection” and claiming it identifies important features.

  • Running selection on the full data before cross-validation (leakage).

  • Comparing methods at different numbers of features without saying so.

Interview Tips

  • Distinguish extraction from selection and supervised from unsupervised.

  • Name filter, wrapper and embedded methods with examples.

37

How does Incremental PCA differ from standard PCA?

Standard PCA computes an SVD of the whole centred dataset at once, so the full matrix must fit in memory; Incremental PCA (sklearn.decomposition.IncrementalPCA) processes the data in mini-batches with partial_fit, updating the mean, the components and the singular values after each batch, so memory depends on the batch size and the number of components rather than on the number of rows. The final components closely match standard PCA (exactly for the explained variance if all data is seen, up to small numerical differences), at the cost of more total computation and the need to choose a batch size at least as large as the number of components. Use it for datasets too large for memory (reading chunks from disk, memory-mapped arrays or a database), for data that arrives over time, or to update a model without refitting from scratch; for data that fits in memory, standard or randomized PCA is simpler and faster.

Read full answer

Applies toscikit-learn 1.5+

Comparison

PCA IncrementalPCA
Data access All at once Mini-batches (partial_fit)
Memory O(n·d) O(batch·d + k·d)
Exactness Exact (or randomized) Very close to exact
New data Refit Continue partial_fit
Batch size — ≥ n_components
Typical use In-memory data Out-of-core, streaming

Code Example

import numpy as np
from sklearn.decomposition import PCA, IncrementalPCA

rng = np.random.default_rng(42)
n, d, k = 20_000, 50, 5
X = rng.normal(size=(n, 10)) @ rng.normal(size=(10, d)) + rng.normal(0, 0.1, (n, d))

full = PCA(n_components=k).fit(X)

ipca = IncrementalPCA(n_components=k)
for chunk in np.array_split(X, 40):                    # 40 batches of 500 rows, e.g. read from disk
    ipca.partial_fit(chunk)

print("variance ratios, PCA:  ", full.explained_variance_ratio_.round(4))
print("variance ratios, IPCA: ", ipca.explained_variance_ratio_.round(4))
cos = np.abs(np.sum(full.components_ * ipca.components_, axis=1))
print("|cosine| between components:", cos.round(4))   # close to 1 for well-separated components

path = "big_matrix.npy"                                # out-of-core pattern with a memory-mapped file
np.save(path, X)
X_mm = np.load(path, mmap_mode="r")
ipca2 = IncrementalPCA(n_components=k, batch_size=1000).fit(X_mm)
print("memory-mapped fit matches:", np.allclose(ipca2.explained_variance_ratio_, ipca.explained_variance_ratio_, atol=1e-3))

Common Pitfalls

  • Batches smaller than n_components (scikit-learn raises an error).

  • Feeding batches that are each from one class or time period, which can bias intermediate results; shuffle when possible.

  • Using IncrementalPCA when data fits in memory (slower, no benefit).

  • Forgetting that scaling must also be done incrementally (StandardScaler.partial_fit).

Interview Tips

  • Contrast whole-data SVD with mini-batch updates and memory use.

  • Name out-of-core and streaming use cases.

38

How is randomized PCA used and what are its benefits?

Randomized PCA approximates the top k principal components without a full SVD: it multiplies the centred data matrix by a random Gaussian matrix with k plus a few extra columns, so the result captures the range of the dominant directions; a few power iterations sharpen it when the spectrum decays slowly, an orthonormal basis Q of that sketch is computed, and an exact SVD of the small matrix QᵀX gives the components. Its cost is roughly O(n·d·k) instead of O(n·d·min(n, d)), it needs only a few passes over the data, it parallelizes as matrix multiplications, and its error is close to the best rank-k approximation with high probability, which is why scikit-learn’s PCA(svd_solver="randomized") (chosen automatically for large inputs when k is small) and TruncatedSVD use it. Use it when you want a few components from a large matrix; when you need many components or exact values, use the full solver.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+

Algorithm (Halko, Martinsson and Tropp)

Step Operation
1 Draw a random matrix Ω (d × (k + p)), p ≈ 5–10 oversampling
2 Y = XcΩ; optionally power iterations Y = (XcXcᵀ)^q XcΩ
3 Q = orthonormal basis of Y (QR decomposition)
4 B = QᵀXc (small (k + p) × d matrix)
5 SVD of B = ŨΣVᵀ; components are the top k rows of Vᵀ

Code Example

import time

import numpy as np
from sklearn.decomposition import PCA

def randomized_pca(X, k, oversample=10, power_iters=4, seed=42):
    rng = np.random.default_rng(seed)
    Xc = X - X.mean(axis=0)
    omega = rng.normal(size=(X.shape[1], k + oversample))
    Y = Xc @ omega
    for _ in range(power_iters):
        Y, _ = np.linalg.qr(Y)                          # re-orthonormalize for stability
        Y = Xc @ (Xc.T @ Y)
    Q, _ = np.linalg.qr(Y)
    B = Q.T @ Xc
    _, s, Vt = np.linalg.svd(B, full_matrices=False)
    return Vt[:k], s[:k] ** 2 / (len(X) - 1)

rng = np.random.default_rng(42)
n, d = 5000, 2000
spectrum = 1 / np.arange(1, d + 1) ** 0.7              # slowly decaying singular values
U, _ = np.linalg.qr(rng.normal(size=(n, 200)))
V, _ = np.linalg.qr(rng.normal(size=(d, 200)))
X = (U * spectrum[:200] * 100) @ V.T + rng.normal(0, 0.01, (n, d))

start = time.perf_counter()
exact = PCA(n_components=10, svd_solver="full").fit(X)
t_full = time.perf_counter() - start
start = time.perf_counter()
comps, variances = randomized_pca(X, 10)
t_rand = time.perf_counter() - start

print(f"full SVD {t_full:.2f}s, randomized {t_rand:.2f}s")
print("relative error of variances:", np.abs(variances / exact.explained_variance_ - 1).max().round(5))
print("min |cosine| with exact components:", np.abs(np.sum(comps * exact.components_, axis=1)).min().round(4))

The randomized version is typically several times faster here and matches the top variances to well under 1%.

Benefits and Trade-offs

Benefit Trade-off
Much faster for small k Approximate (tiny, controllable error)
Few passes over the data Slow spectral decay needs power iterations
Parallel, GPU-friendly Random seed affects tiny details
Works with sparse matrices (TruncatedSVD) Not for computing all components

Common Pitfalls

  • Skipping oversampling or power iterations on data with a flat spectrum, giving poor accuracy.

  • Forgetting to fix random_state for reproducibility.

  • Using it for k close to min(n, d), where it has no advantage.

Interview Tips

  • Describe the random projection → QR → small SVD steps.

  • Mention O(ndk) cost, power iterations and scikit-learn’s solver choice.

39

What recent advances have been made in PCA for big data?

PCA for big data has moved from exact SVDs on one machine to approximate, streaming, distributed and hardware-accelerated methods: randomized SVD and sketching (random projections, CountSketch, frequent directions) compute top components in a few passes with provable error bounds; streaming algorithms such as Oja’s rule, incremental SVD and frequent directions update components one record or batch at a time with bounded memory; distributed implementations in Spark MLlib, Dask and Ray split the work as map-reduce of sufficient statistics or parallel randomized SVD; and GPU libraries (RAPIDS cuML, PyTorch torch.pca_lowrank) make the dense matrix products very fast. Research has also strengthened the method itself: robust PCA (low-rank plus sparse decomposition) handles corrupted data, sparse PCA gives interpretable components with few non-zero loadings, federated and differentially private PCA learn components without centralizing raw data, and random matrix theory gives principled thresholds for separating signal from noise components in high dimensions.

Read full answer

Applies togeneral concepts; examples with NumPy 2.x and scikit-learn 1.5+

Advances at a Glance

Area Methods Benefit
Approximation Randomized SVD, sketching O(ndk) cost, few passes
Streaming Oja’s rule, frequent directions, IncrementalPCA Bounded memory, online updates
Distributed Spark MLlib, Dask-ML, TSQR Scale across clusters
Hardware GPU PCA (cuML, PyTorch) Large speedups for dense data
Robustness Principal Component Pursuit, ROBPCA Outliers and corruption
Interpretability Sparse PCA, structured sparsity Few features per component
Privacy Federated PCA, differentially private PCA No raw data sharing
High-dimensional theory Marchenko-Pastur thresholds, spiked models Choose k reliably

Code Example: Frequent Directions Sketch

import numpy as np
from sklearn.decomposition import PCA

def frequent_directions(rows, ell):
    """Streaming sketch B (ell x d) with ||X^T X - B^T B|| bounded by ||X||_F^2 / ell (Liberty, 2013)."""
    d = rows.shape[1]
    B = np.zeros((2 * ell, d))
    filled = 0
    for row in rows:
        if filled == 2 * ell:                              # sketch full: shrink
            _, s, Vt = np.linalg.svd(B, full_matrices=False)
            s = np.sqrt(np.maximum(s**2 - s[ell] ** 2, 0))
            B = np.zeros((2 * ell, d))
            B[:ell] = s[:ell, None] * Vt[:ell]
            filled = ell
        B[filled] = row
        filled += 1
    return B

rng = np.random.default_rng(42)
X = rng.normal(size=(20_000, 8)) @ rng.normal(size=(8, 200)) + rng.normal(0, 0.1, (20_000, 200))
Xc = X - X.mean(axis=0)

sketch = frequent_directions(Xc, ell=20)                  # 40 x 200 sketch from one pass
_, _, Vt_sketch = np.linalg.svd(sketch, full_matrices=False)
exact = PCA(n_components=8).fit(X)
cos = np.abs(np.sum(Vt_sketch[:8] * exact.components_, axis=1))
print("|cosine| of sketch components with exact PCA:", cos.round(3))   # near 1 for the 8 signal directions
print(f"sketch rows {sketch.shape[0]} vs data rows {X.shape[0]}")

Centring here uses the full-data mean for simplicity; a real stream keeps a running mean and corrects for it.

Common Pitfalls

  • Choosing an approximate method without checking its error on a sample.

  • Ignoring the communication cost in distributed settings.

  • Assuming privacy-preserving or federated versions give identical results to centralized PCA.

Interview Tips

  • Group advances into approximation, streaming, distribution, hardware, robustness and privacy.

  • Explain one method (randomized SVD or frequent directions) in detail.

40

How does robust PCA handle outliers, and what are its uses?

Robust PCA in its best-known form, Principal Component Pursuit (Candès, Li, Ma and Wright, 2011), assumes the data matrix is the sum of a low-rank matrix L (the clean structure) and a sparse matrix S (a few large errors or outliers), M = L + S, and recovers both by solving the convex problem minimize ‖L‖* + λ‖S‖₁ subject to L + S = M, where the nuclear norm ‖L‖* encourages low rank and the ℓ₁ norm encourages sparsity, typically with λ = 1/√max(n, d). Unlike standard PCA, where one corrupted value can rotate the components, it isolates arbitrarily large but sparse corruptions in S and keeps them out of L. Other robust approaches downweight whole outlying rows instead: PCA on a robust covariance (Minimum Covariance Determinant), ROBPCA, or M-estimators. Practical uses include separating the static background (L) from moving objects (S) in surveillance video, removing shadows and specular highlights from face images, cleaning sensor and network-traffic data, and anomaly detection, where S itself is the list of anomalies. The cost is iterative SVDs, which are slower than one PCA, and the model needs the corruption to be sparse and the clean part to be genuinely low-rank.

Read full answer

Applies toNumPy 2.x

Code Example: Principal Component Pursuit with ADMM

import numpy as np

def shrink(X, tau):
    return np.sign(X) * np.maximum(np.abs(X) - tau, 0)

def svd_shrink(X, tau):
    U, s, Vt = np.linalg.svd(X, full_matrices=False)
    return (U * shrink(s, tau)) @ Vt

def robust_pca(M, max_iter=500, tol=1e-7):
    n, d = M.shape
    lam = 1 / np.sqrt(max(n, d))
    mu = n * d / (4 * np.abs(M).sum())
    L, S, Y = np.zeros_like(M), np.zeros_like(M), np.zeros_like(M)
    for _ in range(max_iter):
        L = svd_shrink(M - S + Y / mu, 1 / mu)        # low-rank update
        S = shrink(M - L + Y / mu, lam / mu)          # sparse update
        residual = M - L - S
        Y += mu * residual
        if np.linalg.norm(residual) / np.linalg.norm(M) < tol:
            break
    return L, S

rng = np.random.default_rng(42)
n, d, rank = 200, 100, 3
L_true = rng.normal(size=(n, rank)) @ rng.normal(size=(rank, d))
S_true = np.zeros((n, d))
corrupt = rng.random((n, d)) < 0.05                   # 5% of entries grossly corrupted
S_true[corrupt] = rng.uniform(-20, 20, corrupt.sum())
M = L_true + S_true

L_hat, S_hat = robust_pca(M)
print(f"relative error of L: {np.linalg.norm(L_hat - L_true) / np.linalg.norm(L_true):.2e}")   # tiny
print("rank of recovered L:", np.linalg.matrix_rank(L_hat, tol=1e-3))                      # 3
print(f"corrupted entries found: {np.mean((np.abs(S_hat) > 1e-3) == corrupt):.4f} agreement")

U, s, Vt = np.linalg.svd(M - M.mean(axis=0), full_matrices=False)    # standard PCA for comparison
L_pca = (U[:, :rank] * s[:rank]) @ Vt[:rank] + M.mean(axis=0)
print(f"standard rank-3 PCA error: {np.linalg.norm(L_pca - L_true) / np.linalg.norm(L_true):.2f}")   # much larger

Approaches

Method Outlier model Notes
Principal Component Pursuit Sparse corrupted entries Convex; exact recovery under conditions
MCD-based PCA Whole outlying rows Robust covariance, moderate d
ROBPCA Whole rows, high d Projection pursuit + MCD
Outlier pursuit Corrupted columns Column-sparse S

Common Pitfalls

  • Applying PCP when corruption is dense rather than sparse.

  • Using the default λ blindly; tune it when the outlier share is high.

  • Expecting robust PCA to be as fast as standard PCA.

Interview Tips

  • Write M = L + S and the nuclear norm plus ℓ₁ objective.

  • Give video background subtraction and anomaly detection as uses.

41

How does Generalized PCA differ from standard PCA?

“Generalized PCA” names two related extensions of PCA, and a good answer distinguishes them: Generalized PCA for exponential-family data (Collins, Dasgupta and Schapire, 2001; also called exponential family PCA) replaces PCA’s squared-error loss, which implicitly assumes Gaussian noise, with the likelihood of a distribution that fits the data (Bernoulli for binary, Poisson for counts, multinomial for categories), fitting a low-rank matrix of natural parameters instead of the data itself; and Generalized PCA in the subspace-clustering sense (Vidal, Ma and Sastry, 2005) fits data lying on a union of several different subspaces, rather than one, by representing them as the roots of polynomials. Standard PCA is the special case of the first with Gaussian noise and of the second with one subspace. Applications include binary survey or click data (logistic PCA), word or gene counts (Poisson PCA), and motion segmentation and face clustering under varying lighting (subspace clustering); the cost is iterative, non-convex fitting instead of a single SVD.

Read full answer

Applies togeneral concepts; example with NumPy 2.x

Comparison

Standard PCA Exponential-family GPCA Subspace-clustering GPCA
Data model One subspace + Gaussian noise Low-rank natural parameters, any exponential family Union of several subspaces
Loss Squared error Negative log-likelihood (e.g. logistic, Poisson) Algebraic fit of polynomials
Solution SVD (closed form) Iterative optimization Polynomial fitting + clustering
Typical data Continuous Binary, counts, categories Mixed motions, faces under lighting

Code Example: Logistic PCA vs Standard PCA on Binary Data

import numpy as np

rng = np.random.default_rng(42)
n, d, k = 500, 40, 2
U_true = rng.normal(size=(n, k)) * 2
V_true = rng.normal(size=(d, k))
theta_true = U_true @ V_true.T                       # natural parameters (log-odds)
X = (rng.random((n, d)) < 1 / (1 + np.exp(-theta_true))).astype(float)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def logistic_pca(X, k, lr=0.05, iters=1500, seed=42):
    r = np.random.default_rng(seed)
    U = r.normal(0, 0.1, (X.shape[0], k))
    V = r.normal(0, 0.1, (X.shape[1], k))
    b = np.zeros(X.shape[1])
    for _ in range(iters):                           # gradient descent on the Bernoulli log-likelihood
        P = sigmoid(U @ V.T + b)
        G = P - X
        U -= lr * (G @ V) / X.shape[1]
        V -= lr * (G.T @ U) / X.shape[0]
        b -= lr * G.mean(axis=0)
    return sigmoid(U @ V.T + b)

P_logistic = logistic_pca(X, k)
Xc = X - X.mean(axis=0)
Us, s, Vt = np.linalg.svd(Xc, full_matrices=False)
raw_pca = (Us[:, :k] * s[:k]) @ Vt[:k] + X.mean(axis=0)
print(f"standard PCA reconstructions outside [0, 1]: {np.mean((raw_pca < 0) | (raw_pca > 1)):.1%}")
P_pca = np.clip(raw_pca, 1e-6, 1 - 1e-6)
P_true = sigmoid(theta_true)

for name, P in [("standard PCA", P_pca), ("logistic PCA", P_logistic)]:
    print(f"{name:13s}: mean abs error of probabilities {np.mean(np.abs(P - P_true)):.3f}")
# Logistic PCA's fitted values are proper probabilities; standard PCA's reconstructions are not
# limited to [0, 1] and usually fit the true probabilities less well

Common Pitfalls

  • Using standard PCA on binary or count data and reading reconstructions as probabilities or counts.

  • Confusing the two meanings of “Generalized PCA” in a discussion.

  • Expecting closed-form, unique solutions; these methods need good initialization.

Interview Tips

  • Explain PCA as Gaussian-likelihood fitting and the exponential-family generalization.

  • Mention subspace clustering as the other meaning and give one application of each.

42

Implement PCA in PyTorch and compare it with scikit-learn

PCA in PyTorch follows the same steps as in NumPy: centre the data tensor, compute torch.linalg.svd(Xc, full_matrices=False) (or the faster approximate torch.pca_lowrank for a few components), take the top k rows of Vᴴ as components and the squared singular values divided by n − 1 as explained variances, then project with Xc @ Vₖᵀ; running it on a GPU makes large dense problems much faster. To compare with scikit-learn fairly, use float64 (or loosen tolerances for float32), compare explained variances directly, and compare components and projections only up to sign, because each singular vector is defined up to multiplication by −1. Small differences around 1e-6 in float32 are expected and are not bugs.

Read full answer

Applies toPyTorch 2.x, scikit-learn 1.5+

Code Example

import numpy as np
import torch
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA

class TorchPCA:
    def __init__(self, n_components, device="cpu", dtype=torch.float64):
        self.k, self.device, self.dtype = n_components, device, dtype

    def fit(self, X):
        X = torch.as_tensor(X, dtype=self.dtype, device=self.device)
        self.mean_ = X.mean(dim=0)
        U, S, Vh = torch.linalg.svd(X - self.mean_, full_matrices=False)
        self.components_ = Vh[: self.k]
        self.explained_variance_ = S[: self.k] ** 2 / (X.shape[0] - 1)
        self.explained_variance_ratio_ = S[: self.k] ** 2 / (S ** 2).sum()
        return self

    def transform(self, X):
        X = torch.as_tensor(X, dtype=self.dtype, device=self.device)
        return (X - self.mean_) @ self.components_.T

device = "cuda" if torch.cuda.is_available() else "cpu"
X = load_digits().data
k = 10

tp = TorchPCA(k, device=device).fit(X)
sk = PCA(n_components=k, svd_solver="full").fit(X)

comps = tp.components_.cpu().numpy()
signs = np.sign(np.sum(comps * sk.components_, axis=1))
print("variances match:", np.allclose(tp.explained_variance_.cpu().numpy(), sk.explained_variance_))   # True
print("components match up to sign:", np.allclose(comps * signs[:, None], sk.components_, atol=1e-8))   # True
print("projections match:", np.allclose(tp.transform(X).cpu().numpy() * signs, sk.transform(X), atol=1e-8))   # True

torch.manual_seed(42)
Xt = torch.as_tensor(X, dtype=torch.float32, device=device)
U, S, V = torch.pca_lowrank(Xt, q=k, center=True, niter=4)     # randomized, float32
approx_var = (S ** 2 / (Xt.shape[0] - 1)).cpu().numpy()
print("pca_lowrank relative variance error:", np.abs(approx_var / sk.explained_variance_ - 1).max().round(5))

Notes

Topic Detail
Sign ambiguity Align signs before comparing
Precision float64 for exact checks; float32 with atol around 1e-4
torch.pca_lowrank Randomized; returns V (not Vᴴ), center=True by default
GPU Move tensors with device="cuda"; big speedups for large dense X
Autograd SVD is differentiable, but gradients are unstable for repeated singular values

Common Pitfalls

  • Forgetting to centre (torch.svd_lowrank doesn’t centre; pca_lowrank does by default).

  • Comparing float32 GPU results with float64 scikit-learn results using tight tolerances.

  • Using the deprecated torch.svd instead of torch.linalg.svd.

Interview Tips

  • Show centre → SVD → components → projection with tensors.

  • Mention sign alignment, precision and pca_lowrank.

43

How do you choose the number of PCA dimensions with the elbow method?

The elbow method plots the explained variance (or eigenvalue) of each principal component in order, a scree plot, and picks the number of components at the “elbow”, where the curve bends from a steep drop to a flat tail, because components after that point add little information each. Doing it in Python means fitting PCA with all components on standardized data, plotting explained_variance_ratio_ with its cumulative sum, and either reading the elbow by eye or detecting it automatically, for example by finding the point farthest from the straight line joining the first and last points of the curve (the “knee” or kneedle idea). Because elbows are often vague, confirm the choice with a second criterion such as cumulative variance (for example 90%) or downstream cross-validation.

Read full answer

Applies toNumPy 2.x, scikit-learn 1.5+, Matplotlib 3.8+

Code Example

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n, d, true_rank = 1000, 30, 5
latent = rng.normal(size=(n, true_rank)) * np.array([5, 4, 3, 2.5, 2])
X = latent @ rng.normal(size=(true_rank, d)) + rng.normal(0, 1.0, (n, d))
Xs = StandardScaler().fit_transform(X)

ratio = PCA().fit(Xs).explained_variance_ratio_
components = np.arange(1, d + 1)

def elbow_index(y):
    """Point farthest from the line joining the first and last points (both axes scaled to 0..1)."""
    x = np.linspace(0, 1, len(y))
    y_scaled = (y - y.min()) / (y.max() - y.min())
    start, end = np.array([x[0], y_scaled[0]]), np.array([x[-1], y_scaled[-1]])
    direction = (end - start) / np.linalg.norm(end - start)
    points = np.column_stack([x, y_scaled]) - start
    distances = np.abs(points[:, 0] * direction[1] - points[:, 1] * direction[0])
    return int(np.argmax(distances))

k_elbow = elbow_index(ratio) + 1
k_90 = int(np.searchsorted(np.cumsum(ratio), 0.90) + 1)
print(f"elbow at k = {k_elbow}, 90% variance at k = {k_90}")   # compare with the true rank 5; the rules can differ by a component or two

fig, ax = plt.subplots(figsize=(7, 4))
ax.plot(components, ratio, marker="o", label="explained variance ratio")
ax.plot(components, np.cumsum(ratio), marker=".", label="cumulative")
ax.axvline(k_elbow, color="black", linestyle="--", label=f"elbow (k={k_elbow})")
ax.set(xlabel="number of components", ylabel="variance ratio", title="Scree plot and elbow")
ax.legend()
fig.tight_layout()
fig.savefig("pca_elbow.png", dpi=120)

The distance-to-chord rule can land one component before or after where the eye would put the elbow, so treat it as a suggestion and check it against the plot.

Common Pitfalls

  • Relying on the elbow when the curve decays smoothly with no clear bend.

  • Detecting the elbow on the cumulative curve with a method meant for the per-component curve (or the reverse) without care.

  • Choosing k by the elbow for a predictive model without validating it.

Interview Tips

  • Explain the scree plot and what the bend means.

  • Show an automatic detection rule and a second criterion to confirm it.

44

Write a script to remove outliers automatically before PCA

A robust automatic approach flags outliers with a method that the outliers themselves can’t distort, removes (or sets aside) them, and only then fits PCA: standardize with the median and IQR (RobustScaler), estimate a robust covariance with the Minimum Covariance Determinant (MinCovDet), and flag points whose robust Mahalanobis distance exceeds a chi-squared cut-off such as the 97.5% quantile with d degrees of freedom; alternatives are IsolationForest or LocalOutlierFactor for non-Gaussian data, or a two-pass check of PCA reconstruction error. The script should report how many rows were removed and why, keep them for review instead of silently deleting them (they may be real rare events), cap the removed share, and apply the same fitted detector to new data inside the pipeline, so preprocessing is identical in training and production.

Read full answer

Applies toNumPy 2.x, SciPy 1.13+, scikit-learn 1.5+

Code Example

import numpy as np
from scipy import stats
from sklearn.covariance import MinCovDet
from sklearn.decomposition import PCA
from sklearn.preprocessing import RobustScaler

def remove_outliers_robust(X, quantile=0.975, max_fraction=0.1, random_state=42):
    """Return (X_clean, mask_kept, distances) using robust Mahalanobis distances."""
    Xs = RobustScaler().fit_transform(X)
    mcd = MinCovDet(random_state=random_state).fit(Xs)
    d2 = mcd.mahalanobis(Xs)                                  # squared robust distances
    cutoff = stats.chi2.ppf(quantile, df=X.shape[1])
    keep = d2 <= cutoff
    if (~keep).mean() > max_fraction:                         # never drop more than max_fraction
        keep = d2 <= np.quantile(d2, 1 - max_fraction)
    return X[keep], keep, d2

rng = np.random.default_rng(42)
n, d = 500, 6
X = rng.normal(size=(n, 2)) @ rng.normal(size=(2, d)) + rng.normal(0, 0.3, (n, d))
outlier_rows = rng.choice(n, 15, replace=False)
X[outlier_rows] += rng.normal(0, 8, (15, d))                  # 3% gross outliers

X_clean, keep, d2 = remove_outliers_robust(X)
removed = np.where(~keep)[0]
print(f"removed {len(removed)} rows; true outliers caught: {np.isin(outlier_rows, removed).sum()} of 15")

pca_dirty = PCA(n_components=2).fit(X)
pca_clean = PCA(n_components=2).fit(X_clean)
print("variance in 2 PCs: with outliers", pca_dirty.explained_variance_ratio_.sum().round(3),
      "after removal", pca_clean.explained_variance_ratio_.sum().round(3))
# After removal the two real components explain most of the variance again

review = np.column_stack([removed, d2[removed].round(1)])     # keep for human review, don't discard silently
print("rows set aside (index, squared distance):\n", review[:5])

The chi-squared cut-off assumes the clean data are roughly Gaussian; with a 97.5% cut-off about 2.5% of clean points are flagged as well, so a few false positives are expected.

Method Options

Method Works when Notes
MCD + chi-squared cut-off Roughly elliptical data, moderate d Classic, interpretable
IsolationForest Any shape, high d Needs a contamination setting
LocalOutlierFactor Local density differences Needs novelty=True for new data
PCA reconstruction error Low-rank data Use a robust first fit
Per-feature IQR rule Simple univariate checks Misses multivariate outliers

Common Pitfalls

  • Detecting outliers with a standard (non-robust) PCA or mean/covariance, which the outliers distort (masking).

  • Removing too much data with a strict cut-off.

  • Deleting rows silently that may be valid rare events.

  • Fitting the detector on the full dataset including the test set.

Interview Tips

  • Use robust distances with a chi-squared cut-off and explain masking.

  • Mention review instead of silent deletion and caps on removal.

45

Show the effect of PCA on classification accuracy with a synthetic dataset

A clear experiment creates a synthetic classification dataset with a known structure, a few informative features plus many redundant and noisy ones, then compares a classifier’s cross-validated accuracy and training time with no PCA against PCA with an increasing number of components, using a Pipeline so PCA is fitted inside each fold. The usual result is a curve: too few components lose class information and accuracy drops, a moderate number matches or beats the full feature set because PCA removes noise and redundancy (especially for kNN and SVMs, less for linear models with regularization), and many components bring back the noise. The experiment is honest only if the comparison includes the baseline and uses the same folds; a second synthetic case, where the class signal hides in a low-variance direction, shows PCA hurting.

Read full answer

Applies toscikit-learn 1.5+

Code Example

import time

import numpy as np
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(n_samples=2000, n_features=100, n_informative=8, n_redundant=12,
                           n_repeated=0, n_classes=3, n_clusters_per_class=1,
                           class_sep=1.5, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
knn = KNeighborsClassifier(n_neighbors=15)

def evaluate(model):
    start = time.perf_counter()
    score = cross_val_score(model, X, y, cv=cv).mean()
    return score, time.perf_counter() - start

base, base_t = evaluate(make_pipeline(StandardScaler(), knn))
print(f"no PCA (100 features): accuracy {base:.3f}, time {base_t:.2f}s")
for k in [2, 5, 10, 20, 40, 80]:
    score, t = evaluate(make_pipeline(StandardScaler(), PCA(n_components=k), knn))
    print(f"PCA k={k:>2}: accuracy {score:.3f}, time {t:.2f}s")
# Accuracy usually peaks at a moderate k above the no-PCA baseline: the 80 noise features hurt kNN distances

rng = np.random.default_rng(42)
y2 = rng.integers(0, 2, 2000)
X2 = np.column_stack([rng.normal(0, 5, (2000, 20)), (y2 - 0.5) + rng.normal(0, 0.2, 2000)])
logreg = LogisticRegression(max_iter=1000)
for name, model in [("no PCA", make_pipeline(logreg)), ("PCA k=5", make_pipeline(PCA(n_components=5), logreg))]:
    print(f"low-variance signal, {name}: {cross_val_score(model, X2, y2, cv=cv).mean():.3f}")
# No PCA: close to 1.0. PCA keeps 5 of the 20 high-variance noise directions, drops the signal and falls to about chance

What to Report

Item Why
Baseline without PCA The reference point
Accuracy vs k curve Shows the trade-off
Variance kept at the best k Links to explained variance
Training/prediction time Practical benefit
A failure case PCA is not always helpful

Common Pitfalls

  • Fitting PCA on all data before cross-validation.

  • Showing only the best k and not the curve or the baseline.

  • Generalizing from one synthetic setup to all data.

Interview Tips

  • Describe the experimental design (pipeline, same folds, baseline).

  • Explain why accuracy rises then falls, and show a case where PCA hurts.

46

How do you decide between PCA and a classification algorithm?

The question compares two things that do different jobs: PCA is an unsupervised dimensionality reduction step that never predicts labels, while a classification algorithm (logistic regression, random forest, gradient boosting, SVM, a neural network) learns to predict a label, so if the goal is to assign classes you always need a classifier, and the real decision is whether to put PCA in front of it. Use PCA before the classifier when there are many correlated or noisy features, when distance-based or linear models struggle with dimensionality, when training speed or memory is a constraint, or when you want a quick 2-D view of class structure; skip it when features are few or already informative, when you use tree ensembles, when predictions must be explained in terms of original features, or when the signal is in low-variance directions. Decide with evidence: cross-validate the classifier with and without PCA (tuning k), and if dimensionality reduction should use the labels, try LDA or supervised feature selection instead.

Read full answer

Applies toscikit-learn 1.5+

Decision Guide

Situation Recommendation
Goal is prediction of a label A classifier (always)
Many correlated features, kNN/SVM/linear model Try PCA + classifier
Tree ensembles Usually no PCA
Need explanations per original feature Avoid PCA or use selection
Exploring structure, no labels PCA alone (plus clustering)
Labels available for reduction LDA, PLS, supervised selection
Very wide sparse data (text) TruncatedSVD or none

Code Example: Let Cross-Validation Decide

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)
pipe = Pipeline([("scale", StandardScaler()), ("reduce", "passthrough"), ("clf", SVC())])
grid = [
    {"reduce": ["passthrough"], "clf": [SVC(), RandomForestClassifier(n_estimators=200, random_state=42)]},
    {"reduce": [PCA(n_components=20), PCA(n_components=40)],
     "clf": [SVC(), RandomForestClassifier(n_estimators=200, random_state=42)]},
    {"reduce": [LinearDiscriminantAnalysis(n_components=9)], "clf": [SVC()]},
]
search = GridSearchCV(pipe, grid, cv=5, n_jobs=-1).fit(X, y)
results = sorted(zip(search.cv_results_["mean_test_score"], search.cv_results_["params"]),
                 key=lambda r: r[0], reverse=True)
for score, params in results:
    print(f"{score:.3f}  reduce={params['reduce']}, clf={type(params['clf']).__name__}")

The ranking shows whether PCA helps for each classifier on this data; on digits the SVM does about as well with and without PCA, and the forest gains little from it.

Common Pitfalls

  • Treating PCA as a replacement for a classifier.

  • Adding PCA by habit without comparing.

  • Evaluating with leakage (PCA fitted outside cross-validation).

Interview Tips

  • Clarify that PCA and classifiers do different jobs.

  • Give the cases for and against PCA and decide by cross-validation.

47

Describe a case where PCA reduced overfitting

A typical case is a model with many correlated, noisy features and few training samples, such as predicting a disease outcome from 2,000 gene-expression measurements on 150 patients, a quality score from hundreds of near-infrared spectral wavelengths, or a failure from 300 correlated sensor readings: an unregularized or lightly regularized model fits noise in hundreds of directions, so training accuracy is near perfect while validation accuracy is poor. Projecting onto the top 10–30 principal components, chosen by cross-validation, keeps the shared structure (biological pathways, chemical composition, machine state) and discards most noise dimensions, so the model has far fewer effective parameters, the gap between training and validation scores shrinks, and validation accuracy improves; this is principal component regression, a close cousin of ridge regression, which shrinks low-variance directions instead of cutting them off. The proof is in the numbers: report training and validation scores before and after, chosen inside cross-validation.

Read full answer

Applies toscikit-learn 1.5+

Code Example: Spectral-Style Data with More Features than Samples

import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n, d, latent_dim = 120, 400, 4
wavelengths = np.linspace(0, 1, d)
peaks = np.vstack([np.exp(-((wavelengths - c) ** 2) / 0.005) for c in [0.2, 0.4, 0.6, 0.8]])
concentrations = rng.uniform(0, 1, (n, latent_dim))
X = concentrations @ peaks + rng.normal(0, 0.05, (n, d))      # smooth, highly correlated spectra
y = concentrations @ np.array([3.0, -2.0, 1.5, 0.5]) + rng.normal(0, 0.1, n)

models = {
    "linear regression, 400 features": make_pipeline(StandardScaler(), LinearRegression()),
    "PCA (8) + linear regression": make_pipeline(StandardScaler(), PCA(n_components=8), LinearRegression()),
    "ridge (alpha=10)": make_pipeline(StandardScaler(), Ridge(alpha=10)),
}
for name, model in models.items():
    res = cross_validate(model, X, y, cv=5, scoring="r2", return_train_score=True)
    print(f"{name:32s}: train R2 {res['train_score'].mean():.3f}, validation R2 {res['test_score'].mean():.3f}")
# Plain regression fits the training data perfectly (more features than rows), so its train R2 says nothing;
# the PCA and ridge versions keep train and validation scores close, which is the sign of less overfitting

Why It Works

Effect Explanation
Fewer parameters 8 coefficients instead of 400
Noise removal Discarded components are mostly noise
Collinearity removed Components are uncorrelated
Relation to ridge Ridge shrinks small-variance directions smoothly; PCR drops them

Common Pitfalls

  • Picking k with the test set.

  • Assuming PCA always reduces overfitting; it fails when the signal lives in low-variance directions (partial least squares helps there).

  • Not comparing with simpler regularization like ridge or Lasso.

Interview Tips

  • Describe the high-dimensional, correlated, small-sample setting.

  • Show the train/validation gap closing and mention ridge as an alternative.

48

Give an example of PCA applied incorrectly and how to fix it

A common mistake is to fit the scaler and PCA on the entire dataset, then split it into training and test sets and report the test accuracy: the components were learned using the test rows, so information leaks into training and the score is optimistic, which matters most with small datasets, high dimensions, or when the test data comes from a later time. The fix is to put StandardScaler and PCA inside a scikit-learn Pipeline and fit it only on training data (or within each cross-validation fold), then transform the test set with the already-fitted pipeline. Other frequent misuses, each with a fix: running PCA on unscaled features with different units (standardize first), applying it to categorical codes (use MCA or encoding), choosing components by variance when prediction is the goal (tune k by cross-validation), and interpreting the sign of a component as meaningful (only relative signs within a component matter).

Read full answer

Applies toscikit-learn 1.5+

Code Example: The Leak and the Fix

import numpy as np
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n, d = 60, 2000                                    # few samples, many features, pure noise
X = rng.normal(size=(n, d))
y = rng.integers(0, 2, n)                          # labels are random: true accuracy is 50%

# Wrong: unsupervised steps fitted on all rows, then feature selection by correlation with y on all rows
Xs_all = StandardScaler().fit_transform(X)
corr = np.abs((Xs_all * (y - y.mean())[:, None]).mean(axis=0))
top = np.argsort(corr)[-20:]                       # uses the labels of the test rows too
Z_all = PCA(n_components=5).fit_transform(Xs_all[:, top])
Z_tr, Z_te, y_tr, y_te = train_test_split(Z_all, y, test_size=0.3, random_state=42, stratify=y)
print(f"leaky test accuracy: {LogisticRegression().fit(Z_tr, y_tr).score(Z_te, y_te):.2f}")   # far above 0.5

# Right: every fitted step lives inside the pipeline, fitted per fold
pipe = make_pipeline(StandardScaler(), PCA(n_components=5), LogisticRegression())
print(f"honest CV accuracy: {cross_val_score(pipe, X, y, cv=5).mean():.2f}")   # about 0.5, as it should be

The example adds a label-based selection step to make the leak dramatic; PCA alone leaks less, because it doesn’t see labels, but fitting it on test rows still makes evaluation optimistic and is never correct practice.

Misuses and Fixes

Misuse Fix
Fit scaler/PCA before splitting Pipeline fitted on training folds only
No scaling with mixed units StandardScaler (or RobustScaler) first
PCA on categorical codes MCA/FAMD or proper encoding
k chosen by variance for a prediction task Tune k with cross-validation
Interpreting the sign of a component Interpret relative signs and magnitudes
PCA before tree ensembles by habit Compare with and without
Outliers driving components Robust scaling or robust PCA

Common Pitfalls

  • Re-fitting PCA on the test set separately, which gives different axes from training.

  • Applying a time-series PCA fitted on future data in a backtest.

Interview Tips

  • Lead with the leakage example and the Pipeline fix.

  • List two or three other misuses with their corrections.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 48 PCA answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects