What is dimensionality reduction in machine learning?
Dimensionality reduction transforms high-dimensional data into a lower-dimensional representation while retaining important information. It reduces computational cost, redundancy, and sometimes noise, improving many downstream models. The main trade-off is information loss, which can reduce predictive performance and interpretability.
Read full answerHide full answer
Applies tomachine-learning concepts (all versions); scikit-learn 1.5+ API names (Python 3.12)
How It Works
Dimensionality reduction creates new features through transformations or combinations of the original features. A common linear method is Principal Component Analysis (PCA), which finds directions along which the data has maximum variance.
For PCA, each principal component is calculated from centered data:
PC = Xw
-
X: the centered feature matrix. -
w: a unit direction vector containing component weights. -
PC: the resulting lower-dimensional coordinates.
PCA components are ordered so that the first explains the most variance and each later component explains the maximum remaining variance while being orthogonal to earlier components. The explained variance ratio is the proportion of total variance captured by each component.
Kernel methods, neighborhood embeddings, and neural-network encoders can capture nonlinear relationships.
Feature Selection vs. Feature Extraction
These approaches solve related but different problems.
| Approach | What It Does | Examples |
|---|---|---|
| Feature selection | Keeps a subset of the original features | Filter methods such as correlation or SelectKBest; wrapper methods such as recursive feature elimination; embedded methods such as L1 regression or tree-based importance |
| Feature extraction | Creates new features from the originals | PCA, LDA, truncated SVD, kernel PCA, and autoencoders |
Feature selection preserves the meanings of the selected original columns, whereas extracted features can be harder to interpret.
Why It Matters
-
Computational efficiency: Models process fewer values, reducing training time and storage requirements.
-
Curse of dimensionality: Pairwise distances become less informative as dimensionality grows, affecting methods such as k-NN.
-
Multicollinearity: PCA can replace correlated variables with a smaller set of uncorrelated components.
-
Noise reduction: Discarding low-variance directions may remove measurement noise, although it can also remove useful signal.
-
Visualization: t-SNE and UMAP can project complex datasets into two dimensions for exploration.
-
Storage and deployment: Compact representations can reduce memory, bandwidth, and serving costs.
Method Comparison
| Method | How It Works | Strengths | Weaknesses | When to Use | Typical Complexity or Scale |
|---|---|---|---|---|---|
| PCA | Linear, unsupervised variance maximization | Fast, deterministic, widely supported | Sensitive to scaling; components may be difficult to interpret | General preprocessing, compression, redundancy reduction | Full SVD is roughly O(n * min(n, p)^2); randomized PCA can be faster |
| LDA | Linear, supervised class separation | Uses labels to emphasize class differences | Assumes Gaussian classes with shared covariance; at most n_classes - 1 components |
Classification when classes are reasonably separated | Covariance-based path is commonly around O(n * p^2) |
| Truncated SVD | Factorizes the feature matrix without centering | Effective for sparse text data | Not equivalent to centered PCA; transformed features may be dense | Sparse matrices and text embeddings | Iterative methods are often around O(n * p * r) for rank r |
| Kernel PCA | Uses a kernel to model nonlinear structure | Captures nonlinear relationships | Expensive eigendecomposition; transformation is less convenient | Moderate-sized datasets with nonlinear structure | Approximately O(n^3) time and O(n^2) memory |
| t-SNE | Preserves local neighborhoods in a low-dimensional map | Effective for visualizing local groups | Distances and cluster sizes are unreliable; exact computation is expensive | Exploratory visualization | Exact methods use O(n^2) pairwise work; approximate methods are substantially faster |
| UMAP | Constructs and optimizes a fuzzy neighborhood graph | Often faster than t-SNE and may preserve more global structure | Parameter-sensitive; results depend on preprocessing | Large-scale visualization and embeddings | Sparse-neighbor graph; generally scales better than exact t-SNE |
| Autoencoder | Learns a neural encoder and decoder | Learns nonlinear representations | Requires training, tuning, and sufficient data | Complex nonlinear feature extraction | Depends on architecture and epochs; GPUs commonly help with larger models |
Practical Guidance
-
Scale when required: Standardize before PCA, kernel PCA, t-SNE, or UMAP when features use different units or have substantially different variances. Scaling is generally unnecessary for tree-based models.
-
Prevent leakage: Split the data first and use a scikit-learn
Pipelineso the imputer, scaler, and reducer are fitted only on training folds. -
Handle missing values: Most reducers require finite numeric values; include an imputer in the training pipeline when necessary.
-
Choose the target dimension:
n_components=0.95selects the smallest PCA component count that explains at least 95% of the training variance. -
Validate downstream performance: A compact representation is useful only if it preserves or improves the model’s task-appropriate metrics.
-
Avoid t-SNE for production inference: Although scikit-learn provides
TSNE.transform, its documentation warns that it is not intended to transform new observations reliably. -
Use reproducible settings: Set a fixed random seed for stochastic embedding methods and inspect stability across runs.
-
Consider supervision and class balance: LDA uses labels, so severe class imbalance can affect the learned separation directions.
Common Pitfalls
-
Fitting PCA or another reducer before splitting the data causes data leakage.
-
Forgetting to scale features can let large-magnitude variables dominate PCA.
-
Interpreting t-SNE cluster sizes or inter-cluster distances literally can produce misleading conclusions.
-
Reducing dimensions too aggressively can remove predictive signal.
-
Assuming dimensionality reduction prevents overfitting is incorrect; it can regularize a model but must still be validated.
-
Using t-SNE or UMAP by default overlooks faster linear methods such as PCA.
-
Selecting the best reduction using test-set results turns the test set into a validation set.
-
Confusing feature selection with feature extraction leads to incorrect expectations about interpretability.
Interviewers commonly ask which method suits sparse text data, why scaling matters for PCA, how the number of components is selected, whether t-SNE can transform new data, and how dimensionality reduction can introduce data leakage.