What is unsupervised learning and how does it differ from supervised learning?
KnownRevise
Unsupervised learning finds structure in data that has no labels: groups of similar items, a smaller set of underlying dimensions, unusual points or items that occur together. Supervised learning, by contrast, learns a mapping from inputs to known target labels. Unsupervised methods are useful when labels are expensive or unknown, but their results are harder to evaluate because there is no ground truth to compare against.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Key Differences
Aspect
Supervised learning
Unsupervised learning
Data
Inputs X with labels y
Inputs X only
Goal
Predict y for new inputs
Discover structure in X
Typical tasks
Classification, regression
Clustering, dimensionality reduction, density estimation, anomaly detection, association rules
Evaluation
Compare predictions with true labels (accuracy, RMSE)
Density estimation: Gaussian mixtures, kernel density estimation.
Anomaly detection: Isolation Forest, Local Outlier Factor.
Association rules: Apriori, FP-Growth for “items bought together”.
Self-supervised learning: a modern related approach that creates labels from the data itself (predicting masked words or image patches) to learn representations, as in large language models.
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, _ = load_iris(return_X_y=True) # labels are ignored: this is unsupervised
for k in range(2, 6):
pipe = make_pipeline(StandardScaler(), PCA(n_components=2, random_state=42),
KMeans(n_clusters=k, n_init=10, random_state=42))
labels = pipe.fit_predict(X)
X_reduced = pipe[:-1].transform(X) # data in the space where clustering happened
print(f"k={k}: silhouette={silhouette_score(X_reduced, labels):.3f}")
Common Pitfalls
Scaling and distance choice change the result completely; standardize features for distance-based methods.
Clusters can be artefacts: algorithms always return clusters, even in random data.
The number of clusters is a judgement supported by metrics, not a fact the data reveals.
Interview tip: mention semi-supervised learning (few labels plus many unlabelled points) as the middle ground.
2
What types of problems does unsupervised learning solve?
KnownRevise
Unsupervised learning addresses problems where you have data but no target labels, and you want to discover its structure. The main problem types are clustering, dimensionality reduction, density estimation, anomaly detection and association rule mining, with self-supervised representation learning as a modern extension. Each answers a different question about the data, and each is evaluated without ground truth, which makes validation the main challenge.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
What few directions capture most of the information?
PCA, t-SNE, UMAP, autoencoders
Visualization, compression, noise removal
Density estimation
How likely is a given point?
Gaussian mixtures, kernel density estimation
Probability models, sampling new data
Anomaly detection
Which points do not fit the usual pattern?
Isolation Forest, Local Outlier Factor, one-class SVM
Fraud, equipment faults, intrusion detection
Association rules
Which items occur together?
Apriori, FP-Growth
Market-basket analysis, cross-selling
Representation learning
What compact features describe the data?
Autoencoders, self-supervised models
Pretrained embeddings for later supervised tasks
How They Fit Together
Dimensionality reduction often comes before clustering, to remove noise and speed up distance calculations.
Clustering and density estimation are closely related: a Gaussian mixture does both.
Anomaly detection can be built from density (low-density points are anomalies) or from reconstruction error (points an autoencoder reconstructs poorly).
Learned representations are often used as features for supervised models.
Code Example
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.ensemble import IsolationForest
from sklearn.mixture import GaussianMixture
X, _ = make_blobs(n_samples=500, centers=3, random_state=42)
X = np.vstack([X, [[15, 15], [-15, 12]]]) # two obvious outliers
gmm = GaussianMixture(n_components=3, random_state=42).fit(X)
print("cluster sizes:", np.bincount(gmm.predict(X))) # clustering
print("log-density of first point:", gmm.score_samples(X[:1])) # density estimation
iso = IsolationForest(contamination=0.01, random_state=42).fit(X)
print("outlier rows:", np.where(iso.predict(X) == -1)[0]) # anomaly detection
Common Pitfalls
Choosing the method before stating the business question.
Treating clusters or anomalies as facts without checking them with domain experts or a downstream task.
Interview tip: give one concrete use case per problem type; it shows practical understanding.
3
What is dimensionality reduction and why is it important?
KnownRevise
Dimensionality reduction transforms data with many features into a representation with fewer features while keeping as much useful information as possible. It matters because high-dimensional data is slow to process, hard to visualize, prone to overfitting and affected by the curse of dimensionality, where distances become less meaningful. The trade-off is that some information is always lost, and the new features are often harder to interpret.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Two Families
Feature selection: keep a subset of the original features (variance threshold, correlation filtering, model-based selection). Interpretable, because features keep their meaning.
Feature extraction: build new features as combinations of the original ones (PCA, autoencoders, UMAP). Usually keeps more information in fewer dimensions.
Common Methods
Method
Type
Strengths
Limitations
PCA
Linear, unsupervised
Fast, keeps maximum variance, invertible
Misses non-linear structure; needs scaling
LDA
Linear, supervised
Maximizes class separation
Needs labels; at most classes - 1 components
t-SNE
Non-linear
Excellent 2-D visualization of local clusters
Slow; distances between clusters are not meaningful; no transform for new data in scikit-learn
UMAP
Non-linear
Faster than t-SNE, keeps more global structure, can transform new data
Separate library; hyperparameter-sensitive
Autoencoders
Non-linear, neural
Flexible, works on images and text
Needs more data and tuning
Why It Is Important
Visualization: project data to 2-D or 3-D to see clusters and outliers.
Speed and memory: fewer features mean faster training and smaller models.
Noise reduction: discarding low-variance components often removes noise.
Less overfitting: fewer parameters to fit, especially when features outnumber samples.
Multicollinearity: PCA components are uncorrelated, which helps linear models.
Code Example
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, _ = load_digits(return_X_y=True) # 64 pixel features
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95, random_state=42))
X_reduced = pipe.fit_transform(X)
pca = pipe.named_steps["pca"]
print("original features:", X.shape[1])
print("components for 95% variance:", pca.n_components_)
print("variance kept:", pca.explained_variance_ratio_.sum().round(3))
Common Pitfalls
Not scaling features before PCA, so large-unit features dominate.
Fitting the reducer on all data before a train/test split (leakage).
Reading t-SNE cluster sizes or distances as meaningful.
Reducing dimensions when interpretability of the original features is the main requirement.
4
What is clustering and how does it provide insights into data?
KnownRevise
Clustering is the unsupervised task of grouping data points so that points in the same group (cluster) are more similar to each other than to points in other groups. It gives insight by revealing natural segments, typical profiles and outliers that were not labelled in advance, such as customer types or topics in documents. The trade-off is that clusters depend heavily on the features, scaling, distance measure and algorithm, so they must be validated rather than taken as facts.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Main Families of Algorithms
Family
Example
Cluster shape
Notes
Centroid-based
k-means
Roughly spherical, similar size
Fast; needs k; sensitive to scaling and outliers
Hierarchical
Agglomerative (Ward, average linkage)
Depends on linkage
Dendrogram shows structure at every level; O(n^2) memory
Density-based
DBSCAN, HDBSCAN
Arbitrary shapes
Finds noise points; no k needed
Model-based
Gaussian mixture
Elliptical
Soft (probabilistic) assignments
How Clustering Produces Insights
Segment: group customers by behaviour (frequency, spend, product mix).
Profile: compute the average feature values per cluster and name them (“bargain hunters”, “loyal premium buyers”).
Act: tailor marketing, pricing or product decisions per segment.
Spot outliers: points far from every cluster, or labelled noise by DBSCAN, can be fraud or data errors.
Simplify: replace thousands of items with a handful of representative groups.
Code Example
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
customers = pd.DataFrame({
"orders_per_year": np.concatenate([rng.normal(2, 1, 200), rng.normal(12, 3, 150), rng.normal(30, 5, 50)]),
"avg_order_value": np.concatenate([rng.normal(40, 10, 200), rng.normal(60, 15, 150), rng.normal(150, 30, 50)]),
}).clip(lower=0)
X = StandardScaler().fit_transform(customers)
customers["segment"] = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
# Profile each segment in the original units
print(customers.groupby("segment").agg(["mean", "count"]).round(1))
Evaluating Clusters
Internal metrics: silhouette (higher is better), Davies-Bouldin (lower is better), inertia for k-means.
Stability: do the clusters persist with different seeds or bootstrap samples?
Usefulness: do segments behave differently on business outcomes, and do domain experts recognize them?
Common Pitfalls
Forgetting to scale features, so one large-unit feature decides the clusters.
Assuming clusters exist because the algorithm returned them.
Using k-means on elongated or nested shapes; try DBSCAN or Gaussian mixtures.
5
What is the difference between hard and soft clustering?
KnownRevise
Hard clustering assigns each data point to exactly one cluster, while soft clustering (fuzzy or probabilistic clustering) gives each point a degree of membership in every cluster, such as 70% cluster A and 30% cluster B. Soft clustering captures uncertainty and overlapping groups, which is realistic for many real-world segments. The trade-off is that soft assignments are harder to act on and the models are usually slower and have more parameters.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Lost: a borderline point looks as certain as a central one
Explicit
Interpretation
Simple segments
Richer, but needs thresholds for decisions
Cost
Usually faster
Usually slower (e.g. EM iterations over all components)
How Soft Clustering Works
In a Gaussian mixture model, each cluster is a Gaussian distribution. The membership (responsibility) of point x in cluster k is r_k(x) = pi_k * N(x | mu_k, Sigma_k) / sum_j pi_j * N(x | mu_j, Sigma_j), where pi_k is the cluster’s weight, mu_k its mean and Sigma_k its covariance.
Fuzzy c-means minimizes a weighted distance in which each point’s membership in each cluster is raised to a “fuzzifier” power that controls softness.
A soft clustering becomes hard by taking the cluster with the highest membership.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture
X, _ = make_blobs(n_samples=300, centers=[[0, 0], [3, 0]], cluster_std=1.2, random_state=42)
hard = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
gmm = GaussianMixture(n_components=2, random_state=42).fit(X)
soft = gmm.predict_proba(X) # shape (300, 2), rows sum to 1
borderline = np.where(np.abs(soft[:, 0] - 0.5) < 0.1)[0]
print("hard labels of first 5:", hard[:5])
print("soft memberships of first 5:\n", soft[:5].round(2))
print("points with roughly 50/50 membership:", len(borderline))
When to Use Which
Hard: clear, well-separated groups, or when each item must get one action (one marketing campaign per customer).
Soft: overlapping groups, documents covering several topics, risk scoring where uncertainty matters.
Common Pitfalls
Treating soft memberships as calibrated probabilities when the model’s assumptions (e.g. Gaussian shapes) do not hold.
Converting to hard labels too early and losing the uncertainty that motivated soft clustering.
6
How does the k-means clustering algorithm work?
KnownRevise
K-means is a centroid-based clustering algorithm that splits data into k groups by repeatedly assigning each point to its nearest centroid and moving each centroid to the mean of its assigned points. It minimizes the within-cluster sum of squared distances (inertia). It is fast and simple, but you must choose k, it assumes roughly spherical clusters of similar size and it is sensitive to scaling, outliers and the starting centroids.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
The Algorithm (Lloyd’s Algorithm)
Initializek centroids, usually with k-means++ (spreads initial centroids apart).
Assign each point to the nearest centroid (Euclidean distance).
Update each centroid to the mean of the points assigned to it.
Repeat steps 2-3 until assignments stop changing or the improvement is below a tolerance.
Objective
Inertia = sum_{i=1..n} || x_i - mu_{c(i)} ||^2
x_i is a data point, c(i) the cluster it is assigned to, and mu_{c(i)} that cluster’s centroid.
Each step can only lower (or keep) the inertia, so the algorithm always converges, but possibly to a local minimum. Running several initializations (n_init) and keeping the best reduces that risk.
Complexity: O(n * k * d) per iteration for n points, k clusters and d features.
Choosing k
Elbow method: plot inertia against k and look for the bend.
Silhouette score: pick the k with the highest average silhouette.
Business sense: the number of segments a team can actually act on.
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler
X, _ = make_blobs(n_samples=600, centers=4, cluster_std=1.0, random_state=42)
X = StandardScaler().fit_transform(X)
for k in range(2, 7):
km = KMeans(n_clusters=k, init="k-means++", n_init=10, random_state=42).fit(X)
print(f"k={k}: inertia={km.inertia_:.1f}, silhouette={silhouette_score(X, km.labels_):.3f}")
# The silhouette should peak at k=4 for this data.
Variants
MiniBatchKMeans: updates centroids from small batches; much faster on large data.
k-medoids: uses actual data points as centres; more robust to outliers.
Not scaling features: a feature in thousands dominates one in units.
Using k-means on non-spherical shapes (moons, rings) or clusters of very different density; use DBSCAN or Gaussian mixtures.
Comparing inertia across different k directly: it always decreases as k grows.
Categorical data: means are meaningless; use k-modes or other methods.
7
What does the silhouette coefficient measure in clustering?
KnownRevise
The silhouette coefficient measures how well each point fits its own cluster compared with the nearest other cluster, giving a score from -1 to 1. Averaged over all points, it summarizes clustering quality without labels and is widely used to compare different numbers of clusters or algorithms. Its trade-off is that it favours compact, convex, well-separated clusters, so it can undervalue good density-based clusters with irregular shapes, and it costs O(n^2) distance computations.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Formula
For a point i:
s(i) = (b(i) - a(i)) / max(a(i), b(i))
a(i): the average distance from i to the other points in its own cluster (cohesion).
b(i): the lowest average distance from i to the points of any other cluster (separation from the nearest neighbouring cluster).
A point alone in its cluster gets s(i) = 0 by convention.
The overall score is the mean of s(i) over all points.
Interpreting Values
Score
Meaning
Close to 1
Point is well inside its cluster and far from others
Around 0
Point is on the border between two clusters
Negative
Point is probably in the wrong cluster
As a rough guide, an average above 0.5 suggests clear structure and below 0.25 suggests weak or artificial structure, though these thresholds depend on the data.
How It Is Used
Choosing k: compute the average silhouette for several values of k and prefer the highest.
Comparing algorithms: k-means versus agglomerative clustering on the same features.
Silhouette plots: per-point scores grouped by cluster reveal clusters with many poorly assigned points.
Code Example
from sklearn.cluster import AgglomerativeClustering, KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_samples, silhouette_score
X, _ = make_blobs(n_samples=500, centers=3, cluster_std=1.5, random_state=42)
for name, model in [("kmeans", KMeans(n_clusters=3, n_init=10, random_state=42)),
("agglomerative", AgglomerativeClustering(n_clusters=3))]:
labels = model.fit_predict(X)
per_point = silhouette_samples(X, labels)
print(f"{name}: mean silhouette={silhouette_score(X, labels):.3f}, "
f"negative points={(per_point < 0).sum()}")
Common Pitfalls
Using it for DBSCAN results with noise points: decide whether to exclude the noise label (-1) first.
Comparing scores computed with different distance metrics or feature scalings.
On large datasets, use silhouette_score(X, labels, sample_size=10000, random_state=42) to keep it affordable.
Treating the highest silhouette as the “true” k; combine it with domain knowledge.
8
How does DBSCAN work, and what advantages does it have over k-means?
KnownRevise
DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that lie in dense regions and marks points in sparse regions as noise. A cluster grows from “core” points that have enough neighbours within a radius eps. Unlike k-means, it does not need the number of clusters, finds clusters of arbitrary shape and handles outliers explicitly; its trade-off is sensitivity to eps and trouble with clusters of very different densities.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
How It Works
Parameters: eps (neighbourhood radius) and min_samples (points needed within eps, including the point itself, to be a core point).
Point types:
Core point: has at least min_samples points within eps.
Border point: within eps of a core point but not a core point itself.
Noise point: neither; labelled -1.
Algorithm:
Pick an unvisited point and find its eps-neighbours.
If it is a core point, start a cluster and add all density-reachable points (neighbours of core points, recursively).
Otherwise mark it as noise for now (it may later become a border point).
Repeat until all points are visited.
DBSCAN Versus k-means
Aspect
k-means
DBSCAN
Number of clusters
Must be given
Found automatically
Cluster shape
Convex, roughly spherical
Arbitrary (moons, rings)
Outliers
Forced into a cluster, pulling centroids
Labelled as noise
Varying densities
Handles if shapes are spherical
Struggles: one eps fits all
Parameters
k
eps, min_samples
Predict on new data
Yes (predict)
No predict in scikit-learn; re-fit or assign by nearest core point
Complexity
O(n * k * d) per iteration
About O(n log n) with a spatial index; up to O(n^2) worst case
Code Example
import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score
from sklearn.preprocessing import StandardScaler
X, y_true = make_moons(n_samples=500, noise=0.06, random_state=42)
X = StandardScaler().fit_transform(X)
db = DBSCAN(eps=0.3, min_samples=5).fit(X)
km = KMeans(n_clusters=2, n_init=10, random_state=42).fit(X)
print("DBSCAN clusters:", len(set(db.labels_) - {-1}), "noise points:", (db.labels_ == -1).sum())
print("ARI DBSCAN:", round(adjusted_rand_score(y_true, db.labels_), 3)) # close to 1
print("ARI k-means:", round(adjusted_rand_score(y_true, km.labels_), 3)) # much lower on moons
Choosing eps
Plot the sorted distance of each point to its min_samples-th nearest neighbour (k-distance plot) and choose eps at the “knee”.
A common start for min_samples is 2 * n_features.
HDBSCAN (sklearn.cluster.HDBSCAN) removes the need for eps and handles varying densities better.
Common Pitfalls
Not scaling features: eps is a distance, so units matter.
High-dimensional data: distances concentrate and density loses meaning; reduce dimensions first.
9
How does hierarchical clustering work and when should you use it?
KnownRevise
Hierarchical clustering builds a tree of nested clusters (a dendrogram) instead of a single partition. The common agglomerative version starts with every point as its own cluster and repeatedly merges the two closest clusters until one remains; cutting the tree at a chosen height gives the final clusters. It is useful when you want to see structure at several levels or do not know the number of clusters, but it needs O(n^2) memory, so it does not scale to very large datasets.
Read full answerHide full answer
Applies toscikit-learn 1.5+ and SciPy 1.13+ (Python 3.12)
How Agglomerative Clustering Works
Compute the distance between every pair of points.
Treat each point as a cluster.
Merge the two clusters with the smallest linkage distance.
Update distances between the new cluster and the others.
Repeat until one cluster remains, recording each merge and its distance.
Linkage Criteria
Linkage
Distance between clusters
Behaviour
Ward
Increase in within-cluster variance after merging
Compact, similar-size clusters; Euclidean only; most common default
Complete
Largest pairwise distance
Compact clusters, sensitive to outliers
Average
Mean pairwise distance
Balanced compromise
Single
Smallest pairwise distance
Finds elongated shapes but suffers from “chaining”
When to Use It
Small to medium datasets (up to tens of thousands of points).
Unknown number of clusters: the dendrogram shows natural cut points (large vertical gaps).
Hierarchies matter: taxonomies of species, product categories, gene expression groups.
Any distance metric: works with precomputed distances such as cosine or edit distance (except Ward).
Code Example
from scipy.cluster.hierarchy import fcluster, linkage
from sklearn.cluster import AgglomerativeClustering
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler
X, _ = make_blobs(n_samples=200, centers=3, random_state=42)
X = StandardScaler().fit_transform(X)
# SciPy: full merge tree, then cut it into 3 clusters
Z = linkage(X, method="ward") # each row: [cluster_a, cluster_b, distance, size]
labels_scipy = fcluster(Z, t=3, criterion="maxclust")
print("last 3 merge distances:", Z[-3:, 2].round(2)) # big jumps suggest a good cut
# scikit-learn: same idea with an estimator API
labels = AgglomerativeClustering(n_clusters=3, linkage="ward").fit_predict(X)
print("silhouette:", round(silhouette_score(X, labels), 3))
# Plot with scipy.cluster.hierarchy.dendrogram(Z) in a notebook.
Advantages and Limitations
Advantages: no need to fix k in advance; deterministic; produces an interpretable hierarchy.
Limitations: O(n^2) memory and at least O(n^2) time; merges cannot be undone, so early mistakes persist; sensitive to noise and to the linkage choice.
Common Pitfalls
Not scaling features before computing distances.
Using Ward linkage with non-Euclidean distances.
Running it on hundreds of thousands of rows; use k-means, MiniBatchKMeans or HDBSCAN instead, or cluster a sample.
10
What is the difference between agglomerative and divisive hierarchical clustering?
KnownRevise
Agglomerative and divisive clustering are the two directions for building a cluster hierarchy. Agglomerative clustering is bottom-up: every point starts as its own cluster and the closest clusters are merged step by step. Divisive clustering is top-down: all points start in one cluster, which is split recursively. Agglomerative is far more common because it is simpler and widely implemented; divisive can capture the large-scale structure better but is expensive unless a heuristic split method is used.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Comparison
Aspect
Agglomerative (bottom-up)
Divisive (top-down)
Start
n clusters of one point
One cluster with all points
Step
Merge the two closest clusters
Split one cluster into two
Decisions focus on
Local similarity between small groups
Global structure first
Exact cost
O(n^2) memory, O(n^2 log n) to O(n^3) time
Exponential if all splits are considered
Practical method
Linkage rules (Ward, average, complete, single)
Heuristics such as DIANA or repeated 2-means (bisecting k-means)
Availability
AgglomerativeClustering, scipy.cluster.hierarchy
BisectingKMeans in scikit-learn (a divisive approach that returns a flat partition)
Typical use
Small/medium datasets, dendrograms
When a few top-level groups matter most
How Divisive Clustering Works
Start with all points in one cluster.
Choose a cluster to split (for example the largest, or the one with the highest variance).
Split it into two, e.g. with 2-means or by moving the most dissimilar points out (DIANA).
Repeat until the desired number of clusters or until each cluster is a single point.
Agglomerative: you want a full dendrogram, small or medium data, or a custom distance metric.
Divisive: larger data where only a few top-level splits are needed, since you can stop early instead of building the full tree.
Common Pitfalls
Saying divisive is always more accurate; it depends on the split heuristic.
Forgetting that both are greedy: a bad early merge or split is never undone.
11
How does Principal Component Analysis (PCA) work?
KnownRevise
Principal Component Analysis (PCA) is a linear dimensionality-reduction method that finds new, uncorrelated axes (principal components) ordered by how much of the data’s variance they capture, and projects the data onto the first few of them. It solves the problem of many correlated features by summarizing them in fewer dimensions with minimal information loss. The trade-off is that the components are linear combinations of the original features, so they can be hard to interpret and cannot capture non-linear structure.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Step by Step
Center the data: subtract each feature’s mean (and usually standardize to unit variance).
Covariance: compute C = (1/(n-1)) * X^T X for the centered data matrix X (n rows, d features).
Eigen-decomposition: find the eigenvectors v_j and eigenvalues lambda_j of C. Each eigenvector is a principal direction; its eigenvalue is the variance along it.
Sort components by eigenvalue, largest first.
Project: Z = X W_k, where W_k holds the first k eigenvectors as columns.
In practice, libraries use the singular value decomposition X = U S V^T, which is numerically more stable; the rows of V^T are the components and S^2/(n-1) gives the variances.
Key Quantities
Explained variance ratio: lambda_j / sum(lambda), the share of total variance a component captures.
Loadings: the weights of each original feature in a component; they show what a component “means”.
Reconstruction: X_approx = Z W_k^T + mean; the reconstruction error measures what was lost.
Code Example
import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
X, _ = load_wine(return_X_y=True) # 13 features
X_std = StandardScaler().fit_transform(X)
pca = PCA(random_state=42).fit(X_std)
cumulative = np.cumsum(pca.explained_variance_ratio_)
k = int(np.argmax(cumulative >= 0.90) + 1)
print("explained variance of first 3:", pca.explained_variance_ratio_[:3].round(3))
print(f"components needed for 90% variance: {k}")
Z = PCA(n_components=k, random_state=42).fit_transform(X_std)
print("reduced shape:", Z.shape)
Common Pitfalls
No scaling: features with large units dominate the first components.
Leakage: fit PCA on training data only (inside a Pipeline).
Assuming high variance means useful: the most variable directions are not always the most predictive.
Non-linear data: use kernel PCA, UMAP or autoencoders.
Sign of components is arbitrary: two runs can return the same component with flipped signs.
12
What is t-SNE and when should you use it?
KnownRevise
t-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear method for visualizing high-dimensional data in two or three dimensions. It keeps points that are close neighbours in the original space close in the plot, which makes local clusters stand out clearly. Its trade-off is that it is slow, stochastic and does not preserve global distances or cluster sizes, so it is a visualization tool rather than a general dimensionality-reduction step.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
How It Works
High-dimensional similarities: for each point i, define a Gaussian over its neighbours, giving conditional probabilities p_{j|i}. The Gaussian width is chosen so the distribution has a target perplexity (roughly, the effective number of neighbours; default 30). These are symmetrized into joint probabilities p_ij.
Low-dimensional similarities: in the 2-D map, similarities q_ij use a Student t-distribution with one degree of freedom, which has heavy tails. This lets moderately distant points sit far apart and avoids the “crowding problem”.
Optimization: gradient descent moves the 2-D points to minimize the KL divergence KL(P || Q) = sum p_ij * log(p_ij / q_ij). It penalizes placing true neighbours far apart much more than placing non-neighbours close.
Checking whether classes or clusters are separable before building a model.
Inspecting single-cell genomics data, where t-SNE plots are standard.
Debugging: finding mislabeled points that sit inside another class’s cluster.
Code Example
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True)
X_pca = PCA(n_components=30, random_state=42).fit_transform(StandardScaler().fit_transform(X)) # denoise and speed up
embedding = TSNE(n_components=2, perplexity=30, init="pca", random_state=42).fit_transform(X_pca)
print(embedding.shape) # (1797, 2)
# Plot: plt.scatter(embedding[:, 0], embedding[:, 1], c=y, s=5, cmap="tab10")
t-SNE Versus Alternatives
Method
Preserves
Speed
New data
PCA
Global variance, linear structure
Very fast
transform available
t-SNE
Local neighbourhoods
Slow on large data
No transform in scikit-learn
UMAP
Local and more global structure
Faster than t-SNE
transform available
Common Pitfalls
Over-interpreting the plot: cluster sizes, gaps between clusters and positions are not meaningful.
Single perplexity: try several values (5-50); structure that appears at all of them is more trustworthy.
Using t-SNE features for a model: it has no reliable transform for new points and is not designed for that.
Random seeds: different runs give different layouts; fix random_state.
13
How does Linear Discriminant Analysis differ from PCA?
KnownRevise
Linear Discriminant Analysis (LDA) and PCA both project data onto a few linear directions, but with different goals. PCA is unsupervised: it finds the directions of maximum variance and ignores labels. LDA is supervised: it finds the directions that best separate known classes, by maximizing between-class variance relative to within-class variance. Use PCA for compression and exploration without labels, and LDA when you want class separation for classification or visualization.
LDA finds a projection w that maximizes the Fisher criterion
J(w) = (w^T S_B w) / (w^T S_W w)
S_B: between-class scatter, how far class means are from the overall mean.
S_W: within-class scatter, how spread out points are inside each class.
A high J means the classes are far apart relative to their spread. The solutions are eigenvectors of S_W^-1 S_B, and there are at most C - 1 useful ones for C classes.
When to Use Each
PCA: no labels; many correlated features; the goal is to keep information.
LDA: labels available; you want a few features that separate classes; small datasets where a simple linear classifier works well.
Both: PCA first to remove noise when features outnumber samples (which makes S_W singular), then LDA.
Code Example
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_wine(return_X_y=True) # 3 classes, so LDA gives at most 2 components
for name, reducer in [("PCA", PCA(n_components=2)), ("LDA", LinearDiscriminantAnalysis(n_components=2))]:
pipe = make_pipeline(StandardScaler(), reducer, LogisticRegression(max_iter=1000))
print(name, "2-D accuracy:", cross_val_score(pipe, X, y, cv=5).mean().round(3))
# LDA's 2 components usually separate the wine classes better than PCA's 2.
Common Pitfalls
Using LDA without labels, or asking for more than C - 1 components.
Fitting LDA on the whole dataset before cross-validation, which leaks labels into the projection.
Confusing this LDA with Latent Dirichlet Allocation, the topic model.
14
What is the curse of dimensionality in machine learning?
KnownRevise
The curse of dimensionality is the set of problems that appear when data has many features: the space grows so fast that data becomes sparse, distances between points become similar, and models need far more data to learn reliable patterns. It mainly hurts distance-based methods (k-NN, k-means, DBSCAN) and any model that overfits easily. The usual remedies are dimensionality reduction, feature selection, regularization and more data, each with its own cost.
Read full answerHide full answer
Applies tomachine learning concepts (examples in NumPy 2.x, Python 3.12)
Why It Happens
Volume grows exponentially: to cover a unit cube with 10 points per axis takes 10^d points; at d = 10 that is ten billion.
Distances concentrate: as d grows, the ratio between the farthest and nearest neighbour distances approaches 1, so “nearest” loses meaning.
Most volume is near the edges: in high dimensions, almost all of a hypercube’s volume lies close to its surface, so most points are “outliers” in some direction.
More parameters than samples: models can fit noise perfectly (overfitting).
Effects on Models
Model type
Effect
k-NN, k-means, DBSCAN
Neighbours and clusters become unreliable
Density estimation
Needs exponentially more data
Linear models
Overfit when features approach or exceed samples, unless regularized
Trees and boosting
More robust, but irrelevant features still add noise and cost
Code Example
import numpy as np
rng = np.random.default_rng(42)
for d in (2, 10, 100, 1000):
X = rng.random((500, d))
q = rng.random(d)
dist = np.linalg.norm(X - q, axis=1)
print(f"d={d:5d}: (max - min) / min distance = {(dist.max() - dist.min()) / dist.min():.3f}")
# The relative contrast shrinks as d grows: all points look about equally far away.
Remedies
Dimensionality reduction: PCA, UMAP or autoencoders before distance-based methods.
Feature selection: drop irrelevant or redundant features.
Regularization: L1/L2 penalties, dropout, tree depth limits.
Better distance measures: cosine similarity for text and embeddings.
More data, or simpler models that need fewer parameters.
Interview Tips
Mention the “blessing of dimensionality” counterpoint: high dimensions can make classes linearly separable, which is why kernel methods and SVMs work.
Pitfall: assuming more features always help; irrelevant ones add noise and cost.
15
What is an autoencoder and how does it reduce dimensionality?
KnownRevise
An autoencoder is a neural network trained to reproduce its input at its output, with a narrow middle layer (the bottleneck) that forces it to learn a compressed representation. The encoder maps input to this low-dimensional code and the decoder reconstructs the input from it, so the code serves as a learned, non-linear dimensionality reduction. Compared with PCA it can capture curved structure, but it needs more data, tuning and compute, and its features are harder to interpret.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
Architecture
Encoder: z = f(x), layers that shrink from the input size d to the code size k (with k < d).
Bottleneck (latent code): the k-dimensional vector z, the reduced representation.
Decoder: x_hat = g(z), layers that expand back to d.
Loss: reconstruction error, e.g. mean squared error L = (1/n) * sum ||x - x_hat||^2.
A linear autoencoder with MSE loss learns the same subspace as PCA; non-linear activations let it go beyond PCA.
Variants
Variant
Idea
Use
Undercomplete
Small bottleneck
Dimensionality reduction
Denoising
Corrupt input, reconstruct the clean version
Robust features, denoising
Sparse
Penalty on code activations
Interpretable, sparse features
Variational (VAE)
Code is a probability distribution
Generating new samples
Convolutional
Convolution layers
Images
Code Example
import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import MinMaxScaler
torch.manual_seed(42)
X = torch.tensor(MinMaxScaler().fit_transform(load_digits().data), dtype=torch.float32) # (1797, 64)
class AutoEncoder(nn.Module):
def __init__(self, d=64, k=8):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(d, 32), nn.ReLU(), nn.Linear(32, k))
self.decoder = nn.Sequential(nn.Linear(k, 32), nn.ReLU(), nn.Linear(32, d), nn.Sigmoid())
def forward(self, x):
return self.decoder(self.encoder(x))
model = AutoEncoder()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
for epoch in range(200):
optimizer.zero_grad()
loss = loss_fn(model(X), X) # reconstruct the input itself
loss.backward()
optimizer.step()
with torch.no_grad():
codes = model.encoder(X) # the 8-dimensional representation
print("final reconstruction MSE:", round(loss.item(), 4))
print("reduced shape:", tuple(codes.shape)) # (1797, 8)
In practice, use mini-batches and a validation set to stop training before the model overfits.
Common Pitfalls
A bottleneck that is too wide lets the network copy the input without learning structure.
Not scaling inputs to match the output activation (e.g. sigmoid output needs inputs in [0, 1]).
Judging only the reconstruction loss; check that the codes are useful for the downstream task.
16
What is association rule mining in unsupervised learning?
KnownRevise
Association rule mining finds items or events that frequently occur together in transaction data and expresses them as rules such as {bread, butter} -> {milk}. It is unsupervised because no target is given: the algorithm discovers co-occurrence patterns on its own. It is useful for market-basket analysis, recommendations and finding related events, but it can produce huge numbers of rules, many of them obvious or coincidental, so filtering and domain review are essential.
Read full answerHide full answer
Applies tomachine learning concepts (examples in Python 3.12)
Key Terms
Itemset: a set of items, e.g. {diapers, beer}.
RuleA -> B: if a transaction contains A (antecedent), it tends to contain B (consequent).
Support(A): fraction of transactions that contain A.
Confidence(A -> B) = support(A and B) / support(A): how often the rule is right.
Lift(A -> B) = confidence(A -> B) / support(B): how much more likely B is when A is present than in general. Lift above 1 means a positive association.
How It Works
Find frequent itemsets: all itemsets whose support is at least a threshold (min_support), using Apriori, FP-Growth or Eclat.
Generate rules from each frequent itemset by splitting it into antecedent and consequent.
Filter rules by confidence and lift, then review them.
Why It Counts as Unsupervised Learning
There is no label to predict; the goal is to describe structure in the data.
Like clustering, results are judged by usefulness and interest rather than accuracy against ground truth.
Applications
Domain
Example
Retail
Products bought together, store layout, bundles
E-commerce
“Frequently bought together” recommendations
Web analytics
Pages visited in the same session
Healthcare
Co-occurring diagnoses or medications
IT operations
Alerts that fire together, root-cause hints
Code Example
transactions = [
{"bread", "milk"},
{"bread", "diapers", "beer", "eggs"},
{"milk", "diapers", "beer", "cola"},
{"bread", "milk", "diapers", "beer"},
{"bread", "milk", "diapers", "cola"},
]
def support(*items):
return sum(set(items) <= t for t in transactions) / len(transactions)
a, b = "diapers", "beer"
conf = support(a, b) / support(a)
lift = conf / support(b)
print(f"support={support(a, b):.2f}, confidence={conf:.2f}, lift={lift:.2f}")
# support=0.60, confidence=0.75, lift=1.25
Common Pitfalls
High confidence with lift close to 1: the consequent is simply popular (most baskets contain milk).
Setting min_support too low, which produces millions of rules.
Reading rules as causal: co-purchase does not mean one causes the other.
17
How does the Apriori algorithm find association rules?
KnownRevise
Apriori is the classic algorithm for finding frequent itemsets and association rules. It searches level by level, building itemsets of size k + 1 only from frequent itemsets of size k, and relies on the Apriori property: every subset of a frequent itemset must also be frequent, so any candidate with an infrequent subset can be discarded without counting it. It is simple and easy to explain, but it scans the data once per level and can generate many candidates, which makes it slow on large or dense datasets.
Read full answerHide full answer
Applies tomachine learning concepts (examples in Python 3.12)
The Algorithm
Level 1: count every single item; keep those with support of at least min_support.
Candidate generation: join frequent k-itemsets that share k - 1 items to form (k + 1)-candidates.
Pruning: remove any candidate that has an infrequent k-subset (the Apriori property).
Counting: scan the transactions to count the remaining candidates; keep the frequent ones.
Repeat steps 2-4 until no new frequent itemsets appear.
Rule generation: for each frequent itemset I and each non-empty proper subset A, form A -> I - A and keep it if its confidence meets min_confidence.
Frequent single items: bread (4), milk (4), diapers (4), beer (3). Eggs (1) and cola (2) are dropped.
Frequent pairs: {bread, milk}, {bread, diapers}, {milk, diapers}, {diapers, beer} (3 each). {bread, beer} and {milk, beer} appear only twice.
Triples: {bread, milk, diapers} has all subsets frequent but appears only twice; the other candidates are pruned because they contain {bread, beer} or {milk, beer}. So the search stops.
A rule: beer -> diapers has confidence 3/3 = 1.0 and lift 1.0 / 0.8 = 1.25.
Complexity and Limitations
Up to 2^m itemsets for m items in the worst case.
One full scan of the data per level.
Low min_support causes a candidate explosion, especially with many frequent items.
FP-Growth avoids candidate generation and is usually much faster.
Interview Tips
State the Apriori property clearly; it is the key idea.
Explain the support/confidence/lift trade-off and why lift is needed to filter popular-item rules.
Pitfall: forgetting that thresholds are business choices, not statistical guarantees.
18
What are support, confidence and lift in association rules?
KnownRevise
Support, confidence and lift are the three standard measures for judging association rules of the form A -> B. Support tells you how common an itemset is, confidence how often the rule is correct when A occurs, and lift whether A really raises the chance of B beyond its normal popularity. Using them together matters: high confidence alone can be misleading when B is very common.
Read full answerHide full answer
Applies tomachine learning concepts (examples in Python 3.12)
Definitions
For N transactions:
Support(A) = (number of transactions containing A) / N.
Support(A -> B) = support(A and B): the share of transactions containing both.
Confidence(A -> B) = support(A and B) / support(A): an estimate of P(B | A).
Lift(A -> B) = confidence(A -> B) / support(B) = support(A and B) / (support(A) * support(B)).
Interpreting Lift
Lift
Meaning
> 1
A and B occur together more than expected by chance (positive association)
= 1
Independent
< 1
They occur together less than expected (substitutes)
Lift is symmetric: lift(A -> B) = lift(B -> A). Confidence is not.
Leverage = support(A and B) - support(A) * support(B): the absolute difference from independence.
Conviction = (1 - support(B)) / (1 - confidence(A -> B)): how much more often the rule would be wrong if A and B were independent.
Common Pitfalls
Choosing rules by confidence only; a rule to a very popular item has high confidence but lift near 1.
High lift on tiny support: a coincidence in three baskets; always set a minimum support.
Treating lift as causal.
19
How does FP-Growth improve on the Apriori algorithm?
KnownRevise
FP-Growth (Frequent Pattern Growth) finds the same frequent itemsets as Apriori without generating candidate itemsets. It compresses the transactions into a prefix tree called an FP-tree in two scans of the data, then mines frequent itemsets recursively from that tree. This makes it much faster than Apriori on large or dense datasets; the trade-off is that the tree must fit in memory and the algorithm is harder to implement and explain.
Read full answerHide full answer
Applies tomachine learning concepts (examples in Python 3.12)
How FP-Growth Works
First scan: count every item and drop items below min_support.
Sort the frequent items in each transaction by descending global frequency.
Second scan: insert each sorted transaction into the FP-tree. Transactions with a common prefix share a path, and each node stores a count. A header table links all nodes of the same item.
Mine each item, starting from the least frequent:
Collect its conditional pattern base: the prefix paths that lead to it, with counts.
Build a conditional FP-tree from those paths.
Recursively mine that tree, adding the item as a suffix to every pattern found.
Generate rules from the frequent itemsets exactly as with Apriori.
Comparison With Apriori
Aspect
Apriori
FP-Growth
Candidate generation
Yes, level by level
None
Database scans
One per itemset length
Two
Memory
Candidate sets
FP-tree (compact when transactions overlap)
Speed on dense data or low support
Slow, candidate explosion
Much faster
Simplicity
Very easy to explain
More complex
Output
Frequent itemsets
Same frequent itemsets
Example of Compression
Transactions {a, b, c}, {a, b, d} and {a, b, c, e} with items sorted by frequency share the path a -> b. The tree stores a:3 -> b:3, then branches to c:2 -> e:1 and d:1, instead of storing every transaction separately.
Practical Use
Spark MLlib provides FPGrowth for distributed mining.
In Python, mlxtend.frequent_patterns.fpgrowth returns frequent itemsets from a one-hot encoded DataFrame, with the same interface as its apriori function.
Eclat is another alternative that uses vertical (item -> transaction ID list) data.
Common Pitfalls
Expecting different results from Apriori: both return the same itemsets for the same support threshold; only speed differs.
Very sparse data with little overlap gives a large tree, reducing FP-Growth’s advantage.
Setting min_support too low still produces an overwhelming number of itemsets, whichever algorithm you use.
20
How is association rule learning used in market-basket analysis?
KnownRevise
Market-basket analysis applies association rule learning to shopping transactions to discover which products customers buy together, such as {pasta, tomato sauce} -> {parmesan}. Retailers use these rules for cross-selling, store layout, bundles, promotions and “frequently bought together” recommendations. The main challenge is turning thousands of mined rules into a few actionable ones, using support, confidence and lift plus business judgement.
Read full answerHide full answer
Applies tomachine learning concepts (examples in pandas 2.2+, Python 3.12)
Workflow
Prepare data: one row per transaction (basket) and the list of items; aggregate SKUs to a sensible level (product or category).
Encode: build a binary basket-by-item matrix.
Mine frequent itemsets with Apriori or FP-Growth and a minimum support.
Generate rules and filter by confidence and lift (e.g. lift > 1.2).
Review with the business: drop trivial rules (bag -> anything) and check that rules make sense.
Act and test: run A/B tests of placements or recommendations, because rules show correlation, not causation.
Business Applications
Use
Example
Cross-selling
Suggest batteries when a toy is in the cart
Store layout
Place chips near beer, or deliberately apart to increase store traffic
Bundles and promotions
Discount on the second item of a strong pair
Inventory planning
Stock associated items together
Recommendations
“Customers who bought X also bought Y”
Code Example
from itertools import combinations
import pandas as pd
baskets = [
["pasta", "tomato sauce", "parmesan"], ["pasta", "tomato sauce"], ["bread", "butter"],
["pasta", "parmesan"], ["bread", "butter", "milk"], ["pasta", "tomato sauce", "parmesan", "wine"],
["milk", "bread"], ["tomato sauce", "pasta", "wine"], ["butter", "milk"], ["pasta", "tomato sauce", "parmesan"],
]
onehot = pd.DataFrame([{item: True for item in b} for b in baskets]).notna() # True where the basket has the item
support = onehot.mean() # support of single items
rows = []
for a, b in combinations(onehot.columns, 2):
s_ab = (onehot[a] & onehot[b]).mean()
if s_ab < 0.2: # minimum support: 2 of 10 baskets
continue
for x, y in ((a, b), (b, a)):
conf = s_ab / support[x]
rows.append({"rule": f"{x} -> {y}", "support": s_ab, "confidence": round(conf, 2),
"lift": round(conf / support[y], 2)})
rules = pd.DataFrame(rows).sort_values("lift", ascending=False)
print(rules.head(6).to_string(index=False))
For many items, use a library implementation (mlxtend Apriori/FP-Growth, or Spark’s FPGrowth) rather than looping over pairs.
Common Pitfalls
Rules driven by very popular items (milk, bags) with lift near 1.
Mixing time periods or seasons, which creates misleading associations.
Ignoring profitability: a strong rule on low-margin items may not be worth acting on.
21
How does the Expectation-Maximization algorithm work for clustering?
KnownRevise
Expectation-Maximization (EM) is an iterative algorithm for fitting models with hidden (latent) variables, such as which cluster generated each point. For clustering with a Gaussian mixture, it alternates between estimating each point’s probability of belonging to each cluster (E-step) and re-estimating the cluster parameters from those probabilities (M-step). It gives soft cluster assignments and flexible elliptical clusters; the trade-off is sensitivity to initialization and convergence to local optima.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
The Steps for a Gaussian Mixture
Model: each point comes from one of K Gaussians with weight pi_k, mean mu_k and covariance Sigma_k.
Initialize the parameters (often from k-means).
E-step: compute responsibilities, the probability that cluster k generated point x_i: r_ik = pi_k * N(x_i | mu_k, Sigma_k) / sum_j pi_j * N(x_i | mu_j, Sigma_j)
M-step: update parameters using the responsibilities as weights, with N_k = sum_i r_ik:
Repeat until the log-likelihood sum_i log(sum_k pi_k * N(x_i | mu_k, Sigma_k)) stops improving.
Each iteration never decreases the log-likelihood, so EM converges, but possibly to a local maximum.
Relation to k-means
k-means is a limiting case of EM: hard assignments instead of probabilities, and equal, spherical covariances. EM generalizes it to soft assignments and elliptical clusters of different sizes.
Code Example
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture
X, _ = make_blobs(n_samples=600, centers=3, cluster_std=[1.0, 2.0, 0.5], random_state=42)
gmm = GaussianMixture(n_components=3, covariance_type="full", n_init=5, random_state=42).fit(X)
print("converged:", gmm.converged_, "after", gmm.n_iter_, "iterations")
print("weights:", gmm.weights_.round(2))
print("means:\n", gmm.means_.round(2))
print("responsibilities of first point:", gmm.predict_proba(X[:1]).round(3))
Other Applications
Missing-data imputation, hidden Markov models (Baum-Welch), topic models and mixture models in general.
Common Pitfalls
A component can collapse onto a single point (infinite likelihood); scikit-learn adds reg_covar to prevent this.
Results depend on initialization; use n_init > 1.
The number of components must be chosen, for example with BIC.
22
What are Gaussian Mixture Models and how do they relate to clustering?
KnownRevise
A Gaussian Mixture Model (GMM) describes data as coming from a mixture of several Gaussian (normal) distributions, each with its own mean, covariance and weight. Used for clustering, each Gaussian is a cluster, and every point gets a probability of belonging to each one. GMMs are more flexible than k-means because clusters can be elliptical and of different sizes, and they give soft assignments and a density estimate; the trade-off is more parameters, slower fitting and sensitivity to initialization.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
The Model
p(x) = sum_{k=1..K} pi_k * N(x | mu_k, Sigma_k)
K: number of components (clusters).
pi_k: mixing weight of component k (weights sum to 1).
mu_k, Sigma_k: mean vector and covariance matrix of component k.
N(x | mu, Sigma): the multivariate normal density.
Parameters are fitted with the Expectation-Maximization (EM) algorithm.
GMM Versus k-means
Aspect
k-means
GMM
Assignment
Hard
Soft (probabilities)
Cluster shape
Spherical, similar size
Elliptical, different sizes and orientations
Output
Labels, centroids
Labels, probabilities, density p(x)
Model selection
Elbow, silhouette
BIC / AIC (likelihood-based)
Cost
Lower
Higher, especially with full covariances
Covariance Types in scikit-learn
full: each component has its own full covariance (most flexible).
spherical: one variance per component (closest to k-means).
Code Example
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler
X, _ = make_blobs(n_samples=800, centers=4, cluster_std=[0.6, 1.2, 0.8, 1.5], random_state=42)
X = X @ np.array([[0.6, -0.6], [-0.4, 0.8]]) # stretch the blobs into ellipses
X = StandardScaler().fit_transform(X)
bics = {k: GaussianMixture(k, covariance_type="full", n_init=3, random_state=42).fit(X).bic(X)
for k in range(1, 8)}
best_k = min(bics, key=bics.get)
print("BIC by k:", {k: round(v) for k, v in bics.items()})
print("best k by BIC:", best_k)
gmm = GaussianMixture(best_k, covariance_type="full", n_init=3, random_state=42).fit(X)
print("soft memberships of 3 points:\n", gmm.predict_proba(X[:3]).round(2))
print("log-density of 3 points:", gmm.score_samples(X[:3]).round(2)) # low values flag anomalies
Uses Beyond Clustering
Anomaly detection: points with low score_samples are unusual.
Density estimation and sampling: gmm.sample(n) generates new data.
Speaker and image segmentation, background modelling in video.
Common Pitfalls
Too many components or full covariance on small data, which overfits; compare BIC.
Forgetting to scale features.
Non-Gaussian cluster shapes (moons) still need many components or a different method.
23
How do you choose the optimal number of clusters?
KnownRevise
Choosing the number of clusters combines quantitative criteria with judgement: you fit the clustering for a range of k values and compare internal metrics such as the elbow in inertia, the silhouette score, the gap statistic or BIC, then check which result is stable and useful. There is rarely a single “true” k, because the answer depends on the features, the algorithm and the purpose. The trade-off is between finer, more specific clusters and simpler groups that people can act on.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Methods
Method
Idea
Pick k where
Elbow (inertia)
Within-cluster sum of squares falls as k grows
The curve bends and further gains are small
Silhouette
Cohesion versus separation per point, from -1 to 1
Average silhouette is highest
Davies-Bouldin
Average similarity of each cluster to its most similar one
Lowest value
Calinski-Harabasz
Ratio of between- to within-cluster dispersion
Highest value
Gap statistic
Compares inertia with that of random reference data
Gap is largest (or first within one standard error of the next)
BIC / AIC
Likelihood penalized by parameter count (Gaussian mixtures)
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler
X, _ = make_blobs(n_samples=1000, centers=5, cluster_std=1.1, random_state=42)
X = StandardScaler().fit_transform(X)
print(" k inertia silhouette DB CH BIC")
for k in range(2, 9):
km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
bic = GaussianMixture(k, random_state=42).fit(X).bic(X)
print(f"{k:2d} {km.inertia_:8.1f} {silhouette_score(X, km.labels_):10.3f} "
f"{davies_bouldin_score(X, km.labels_):6.3f} {calinski_harabasz_score(X, km.labels_):7.0f} {bic:8.0f}")
# Most criteria should point to k=5 for this synthetic data.
Practical Advice
Use several criteria: if the elbow, silhouette and BIC agree, you can be more confident.
Check stability: rerun with different seeds and bootstrap samples.
Profile the clusters: can the business describe and act on each one?
Algorithms that choose k themselves: DBSCAN and HDBSCAN find the number of clusters from density.
Common Pitfalls
Reading inertia alone: it always decreases as k grows, so the minimum is meaningless.
Unscaled features, which change every metric.
Treating the metric-optimal k as final when the business needs a different level of detail.
24
What are cluster validity indices?
KnownRevise
Cluster validity indices are numbers that score how good a clustering is, so you can compare algorithms, parameters or numbers of clusters. Internal indices use only the data and the cluster labels (cohesion and separation), external indices compare the clusters with known labels, and relative use means comparing several clusterings with the same index. They make clustering evaluation objective, but each index encodes assumptions about what a “good” cluster looks like, so none is universally right.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Internal Indices (No Labels Needed)
Index
Measures
Better
Note
Silhouette
(b - a) / max(a, b) per point: separation minus cohesion
Higher (max 1)
O(n^2); favours convex clusters
Davies-Bouldin
Average of each cluster’s worst ratio (spread_i + spread_j) / distance between centroids
Lower (min 0)
Centroid-based
Calinski-Harabasz
Between-cluster dispersion / within-cluster dispersion, adjusted for k and n
Higher
Fast; favours convex clusters
Dunn index
Smallest inter-cluster distance / largest cluster diameter
Higher
Sensitive to outliers; not in scikit-learn
Inertia (WCSS)
Sum of squared distances to centroids
Lower, but always falls with k
Use with the elbow method
External Indices (Ground Truth Needed)
Index
Range
Notes
Adjusted Rand Index (ARI)
About -0.5 to 1; 0 = random
Pair-counting, chance-adjusted
Normalized / Adjusted Mutual Information
0 to 1
Information-theoretic; AMI is chance-adjusted
Homogeneity, completeness, V-measure
0 to 1
Each cluster has one class / each class in one cluster
Fowlkes-Mallows
0 to 1
Geometric mean of pairwise precision and recall
External indices are used for benchmarking algorithms on labelled data, not in real unsupervised projects where labels are absent.
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.metrics import (adjusted_rand_score, calinski_harabasz_score,
davies_bouldin_score, normalized_mutual_info_score, silhouette_score)
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
print("internal silhouette:", round(silhouette_score(X, labels), 3))
print("internal Davies-Bouldin:", round(davies_bouldin_score(X, labels), 3))
print("internal Calinski-Harabasz:", round(calinski_harabasz_score(X, labels), 1))
print("external ARI:", round(adjusted_rand_score(y, labels), 3))
print("external NMI:", round(normalized_mutual_info_score(y, labels), 3))
Common Pitfalls
Using silhouette or Davies-Bouldin to judge DBSCAN on non-convex shapes; they penalize correct but curved clusters.
Comparing index values computed on differently scaled or reduced data.
Optimizing one index blindly; combine indices with stability checks and domain review.
25
What challenges arise when clustering high-dimensional data?
KnownRevise
Clustering high-dimensional data is hard because distance-based similarity breaks down: as the number of features grows, distances between all points become similar, data becomes sparse, and irrelevant features add noise that hides real structure. Clusters may also exist only in subsets of features. The usual answer is to reduce or select dimensions, choose a suitable distance and validate results carefully, accepting some loss of information and interpretability.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Main Challenges
Distance concentration: the nearest and farthest neighbours are almost equally far away, so k-means, DBSCAN and hierarchical clustering lose their signal.
Irrelevant and noisy features: each one adds random variation to every distance.
Subspace clusters: groups may differ on only a few features (e.g. genes active in one tissue), invisible in the full space.
Sparsity: density-based methods find almost no dense regions.
Cost: distance computations grow with the number of features, and O(n^2) methods become impractical.
Interpretation and visualization: it is hard to describe or plot clusters in 1,000 dimensions.
Correlated features: groups of correlated features count the same information several times.
Remedies
Remedy
How it helps
PCA / truncated SVD
Keep the main variance directions, discard noise
UMAP (then cluster)
Non-linear reduction that keeps neighbourhoods; common for embeddings
Feature selection
Remove low-variance, redundant or irrelevant features
Cosine distance
More meaningful for text and embeddings than Euclidean
Subspace / biclustering
Find clusters in feature subsets
Spectral clustering
Clusters a similarity graph rather than raw distances
Model-based with constraints
Gaussian mixtures with diagonal covariance on reduced data
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.metrics import adjusted_rand_score
from sklearn.preprocessing import StandardScaler
# 4 informative features hidden among 496 noise features
X, y = make_classification(n_samples=1500, n_features=500, n_informative=4, n_redundant=0,
n_clusters_per_class=1, n_classes=3, class_sep=2.0, random_state=42)
X = StandardScaler().fit_transform(X)
raw = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
X_pca = PCA(n_components=10, random_state=42).fit_transform(X)
reduced = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X_pca)
print("ARI on all 500 features:", round(adjusted_rand_score(y, raw), 3))
print("ARI after PCA to 10:", round(adjusted_rand_score(y, reduced), 3))
# Results vary with the data; removing noise dimensions usually helps here.
Common Pitfalls
Clustering raw one-hot or TF-IDF matrices with Euclidean k-means without normalization.
Reducing with t-SNE and then clustering, which can create artificial clusters; prefer UMAP or PCA for this.
Assuming PCA always keeps the cluster structure; high-variance directions may not be the separating ones.
26
Which preprocessing steps should you do before unsupervised learning?
KnownRevise
Preprocessing for unsupervised learning prepares data so that distances, variances and densities reflect real structure rather than units, noise or errors. The usual steps are cleaning, handling missing values, encoding categories, transforming skewed features, scaling, removing redundant features and optionally reducing dimensions. Because there is no label to catch mistakes, preprocessing choices strongly shape the result, so each one should be deliberate and documented.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Recommended Steps
Understand the data: types, ranges, distributions, duplicates and the business meaning of each feature.
Clean: remove duplicates, fix data entry errors, standardize formats.
Handle missing values: drop, impute, or add missing indicators (see what the missingness means).
Treat outliers: decide whether they are errors (remove or cap) or interesting (keep; they may be the anomalies you want).
Encode categorical features: one-hot encoding for nominal data, or use distance measures and algorithms made for mixed data (Gower distance, k-prototypes).
Transform skewed features: log or Yeo-Johnson transforms for amounts and counts.
Scale: StandardScaler or RobustScaler so every feature contributes fairly to distances.
Letting one-hot columns dominate distances; consider weighting them or using a mixed-data method.
Keeping ID or timestamp columns, which create meaningless clusters.
Different preprocessing at scoring time than at training time; save the whole pipeline.
27
How do you handle missing values in unsupervised learning?
KnownRevise
Handling missing values in unsupervised learning is harder than in supervised learning because there is no target to check whether an imputation helps. The main options are removing incomplete rows or columns, imputing values (mean, median, k-NN or iterative imputation), or using methods that tolerate missing data. The trade-off is that imputation can create artificial similarities between points and distort clusters, while removal throws information away.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Options
Approach
When it fits
Risk
Drop rows
Few rows affected, missing at random
Bias if missingness is related to the data
Drop columns
A feature is mostly empty
Losing a useful feature
Mean / median imputation
Few missing values, quick baseline
Pulls points toward the centre, shrinks variance, creates fake clusters at the mean
Most-frequent / “missing” category
Categorical features
Adds an artificial category
k-NN imputation (KNNImputer)
Moderate missingness, correlated features
Slow on large data; distances need scaling
Iterative imputation (IterativeImputer)
Features predict each other
Slower; experimental API in scikit-learn
Missing indicators
Missingness itself is informative
More dimensions
Algorithms that handle NaN
E.g. EM-based mixture models with missing data, matrix factorization
Fewer ready-made tools
Understand the Missingness First
MCAR (missing completely at random): dropping or simple imputation is unbiased.
MAR (depends on observed features): model-based imputation works well.
MNAR (depends on the missing value itself, e.g. high earners skip income): no imputation is fully safe; add indicators and be cautious.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.metrics import adjusted_rand_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_blobs(n_samples=600, n_features=5, centers=3, random_state=42)
rng = np.random.default_rng(42)
X_missing = X.copy()
X_missing[rng.random(X.shape) < 0.2] = np.nan # 20% missing
for name, imputer in [("mean", SimpleImputer(strategy="mean")), ("knn", KNNImputer(n_neighbors=5))]:
pipe = make_pipeline(StandardScaler(), imputer, KMeans(n_clusters=3, n_init=10, random_state=42))
labels = pipe.fit_predict(X_missing)
print(f"{name} imputation, ARI vs true groups: {adjusted_rand_score(y, labels):.3f}")
StandardScaler ignores NaN values when computing means and variances, so scaling before k-NN imputation is valid and keeps the distances fair.
Common Pitfalls
Mean-imputing many values, which creates a dense artificial cluster.
Imputing before scaling for distance-based imputers, so large-unit features dominate.
Not checking whether clusters are just “rows with many missing values”.
28
How do you scale and normalize data for clustering?
KnownRevise
Scaling and normalizing for clustering puts features on comparable ranges so that each contributes fairly to distances. Without it, a feature measured in thousands (income) completely outweighs one measured in units (number of children) in k-means, hierarchical clustering or DBSCAN. The choice of scaler matters too: standardization, min-max scaling, robust scaling and row normalization each suit different data, and the wrong one can distort clusters.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Steps
Inspect distributions: look for skew, outliers and different units.
Transform skewed features first: log1p for counts and amounts, or PowerTransformer (Yeo-Johnson).
Choose a scaler for the remaining features (table below).
Consider feature weights: after scaling, every feature counts equally; multiply important features by a weight if domain knowledge says they should count more.
Normalize rows if the direction of a vector matters more than its size (text, embeddings).
Fit the scaler once, inside a pipeline, and reuse it for new data.
Choosing a Scaler
Scaler
Formula
Use when
StandardScaler
(x - mean) / std
Default for k-means, PCA, Gaussian mixtures
MinMaxScaler
(x - min) / (max - min)
Bounded features, no extreme outliers
RobustScaler
(x - median) / IQR
Outliers present
PowerTransformer
Yeo-Johnson / Box-Cox, then standardize
Strongly skewed features
Normalizer
Each row divided by its L2 norm
Text vectors, embeddings, cosine-like clustering
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler, StandardScaler
rng = np.random.default_rng(42)
# Two groups that differ only in "children"; "income" is large-scale noise
children = np.concatenate([rng.normal(0.5, 0.3, 300), rng.normal(3, 0.3, 300)])
income = rng.normal(60_000, 15_000, 600)
X = np.column_stack([income, children])
y = np.repeat([0, 1], 300)
raw = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
print("unscaled ARI:", round(adjusted_rand_score(y, raw), 3)) # near 0: income dominates
for scaler in (StandardScaler(), RobustScaler()):
labels = make_pipeline(scaler, KMeans(n_clusters=2, n_init=10, random_state=42)).fit_predict(X)
print(type(scaler).__name__, "ARI:", round(adjusted_rand_score(y, labels), 3)) # near 1
Common Pitfalls
Scaling one-hot columns as if they were continuous; decide their weight deliberately.
Min-max scaling with outliers: one extreme value squeezes everyone else into a tiny range.
Scaling with statistics from a different period than the data being clustered without noticing drift.
Not scaling at all because “the features are already numeric”.
29
How do you evaluate the performance of a clustering algorithm?
KnownRevise
Evaluating a clustering algorithm is harder than evaluating a classifier because there are usually no true labels. It combines internal metrics (how compact and separated the clusters are), stability checks (do the clusters persist across samples and seeds), external metrics when some labels exist, and usefulness checks (do the clusters help a business decision or a downstream model). No single number is enough, so good evaluation uses several of these together.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Downstream model gain, business KPIs, expert review
Do the clusters matter?
Stability Check
Draw several bootstrap samples (or change the seed).
Re-cluster each one.
Compare the labels on shared points with the Adjusted Rand Index.
High average ARI means the structure is real, not an artefact of randomness.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score, davies_bouldin_score, silhouette_score
from sklearn.preprocessing import StandardScaler
X, _ = make_blobs(n_samples=800, centers=4, cluster_std=1.3, random_state=42)
X = StandardScaler().fit_transform(X)
rng = np.random.default_rng(42)
def stability(X, k, n_boot=20):
base = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X)
scores = []
for b in range(n_boot):
idx = rng.choice(len(X), size=len(X), replace=True)
boot = KMeans(n_clusters=k, n_init=10, random_state=b).fit(X[idx])
scores.append(adjusted_rand_score(base.predict(X), boot.predict(X))) # compare on all points
return np.mean(scores)
for k in (3, 4, 5, 6):
labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X)
print(f"k={k}: silhouette={silhouette_score(X, labels):.3f}, "
f"DB={davies_bouldin_score(X, labels):.3f}, stability ARI={stability(X, k):.3f}")
Usefulness Checks
Profiles: average feature values per cluster should tell a clear story.
Business outcomes: clusters should differ on revenue, churn or response rates.
Downstream models: adding the cluster label as a feature should help a supervised model, measured with cross-validation.
Expert review: domain experts should recognize the groups.
Common Pitfalls
Optimizing silhouette alone, which favours a few large spherical clusters.
Using external labels that are not what the clustering is meant to find.
Evaluating on the same sample used to choose parameters without any stability check.
30
Why is feature selection important in unsupervised learning?
KnownRevise
Feature selection in unsupervised learning chooses which features the algorithm sees, and it largely decides what structure is found: clustering on spending features finds spending segments, while adding demographic features finds different groups. Removing irrelevant, noisy or redundant features makes clusters clearer, more stable and easier to explain, and it reduces cost. The difficulty is that there is no target to measure relevance against, so selection relies on statistics, stability and domain knowledge.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Why It Matters
Noise drowns structure: every irrelevant feature adds random variation to distances.
Redundancy double-counts: ten correlated “spend” features give spending ten times the weight.
Curse of dimensionality: fewer features make distances more meaningful.
Interpretability: clusters defined by five features are easier to describe than by 200.
Cost: fewer features mean faster clustering and simpler pipelines.
Methods Without Labels
Method
Idea
Variance threshold
Remove near-constant features
Correlation filtering
Drop one of each highly correlated pair
Laplacian score
Prefer features that preserve the local neighbourhood structure
Wrapper search
Try feature subsets, keep the one with the best silhouette or stability
Domain-driven selection
Choose features that match the business question
Feature extraction
PCA or autoencoders combine features instead of selecting them
Post-hoc importance
Train a classifier to predict cluster labels; its important features explain the clusters
Code Example
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.feature_selection import VarianceThreshold
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
X_signal, _ = make_blobs(n_samples=500, n_features=3, centers=4, random_state=42)
X = pd.DataFrame(np.hstack([X_signal, rng.normal(size=(500, 7)), np.full((500, 1), 1.0)]),
columns=[f"s{i}" for i in range(3)] + [f"noise{i}" for i in range(7)] + ["constant"])
X["s0_copy"] = X["s0"] * 2 + rng.normal(0, 0.01, 500) # redundant feature
X_var = X.loc[:, VarianceThreshold(1e-6).fit(X).get_support()] # drop the constant column
corr = X_var.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
X_sel = X_var.drop(columns=[c for c in upper.columns if (upper[c] > 0.95).any()])
for name, data in [("all features", X_var), ("signal only", X_sel[["s0", "s1", "s2"]])]:
Z = StandardScaler().fit_transform(data)
labels = KMeans(n_clusters=4, n_init=10, random_state=42).fit_predict(Z)
print(f"{name}: silhouette={silhouette_score(Z, labels):.3f}")
Common Pitfalls
Choosing features by the silhouette score alone, which can favour trivial splits.
Removing a feature because it has low variance after scaling (scaling sets all variances to 1; check before scaling).
Forgetting that the selected features define the meaning of the clusters; state them clearly in reports.
31
How do you cluster a large distributed dataset?
KnownRevise
Clustering a large distributed dataset means running the clustering where the data lives, across many machines, using algorithms that need only a few passes and exchange small summaries instead of raw rows. The common choices are distributed k-means (Spark MLlib, Dask-ML), mini-batch k-means on samples or streams, and two-stage approaches that summarize data first and then cluster the summaries. The trade-off is that exact methods with O(n^2) cost, like hierarchical clustering or plain DBSCAN, do not scale, so you accept approximations.
Each worker assigns its points to centroids and sends partial sums; the driver updates centroids
Very large numeric data
| k-means|| initialization | Parallel version of k-means++ in Spark | Fast, good starting centroids | | Mini-batch k-means | Updates centroids from small random batches | One machine, data streamed from disk | | Sample, then assign | Cluster a representative sample, then assign all points to the nearest centroid in parallel | Expensive algorithms (HDBSCAN, GMM) | | Two-stage (BIRCH-style) | Compress data into many micro-clusters, then cluster those | Streaming or memory-limited settings | | Bisecting k-means | Divisive splits, distributed in Spark | Hierarchical-like results at scale |
Code Example
from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from pyspark.ml.clustering import KMeans
from pyspark.ml.evaluation import ClusteringEvaluator
from pyspark.ml.feature import StandardScaler, VectorAssembler
spark = SparkSession.builder.appName("distributed-kmeans").getOrCreate()
# In production: spark.read.parquet("s3://bucket/customers/") ; here a synthetic DataFrame
df = spark.range(0, 1_000_000).selectExpr(
"id", "rand(42) * 100 AS spend", "randn(7) * 5 + 20 AS visits", "rand(3) * 10 AS tenure")
pipeline = Pipeline(stages=[
VectorAssembler(inputCols=["spend", "visits", "tenure"], outputCol="raw"),
StandardScaler(inputCol="raw", outputCol="features", withMean=True, withStd=True),
KMeans(k=5, seed=42, initMode="k-means||", maxIter=30, featuresCol="features"),
])
model = pipeline.fit(df)
clustered = model.transform(df)
silhouette = ClusteringEvaluator(featuresCol="features").evaluate(clustered.sample(0.05, seed=42))
print("silhouette on a 5% sample:", round(silhouette, 3))
clustered.groupBy("prediction").count().show()
Practical Tips
Scale features in the distributed pipeline so every node uses the same statistics.
Cache the feature DataFrame: k-means passes over the data many times.
Evaluate on samples: silhouette is O(n^2) in general (Spark’s evaluator uses a cheaper approximation, but sampling still saves time).
Reduce dimensions first (PCA in Spark MLlib) for wide data.
Save the fitted pipeline to assign new data consistently.
Common Pitfalls
Collecting all data to the driver (toPandas()) to run scikit-learn.
Skewed partitions that leave one worker doing most of the work.
Running DBSCAN-style algorithms naively at scale; they need spatial partitioning or sampling.
32
Where can unsupervised learning add value to a business process?
KnownRevise
Unsupervised learning adds business value wherever a company has lots of data but no labels for the question it cares about. A typical example is customer support: thousands of tickets arrive daily, nobody has tagged them by root cause, and clustering their text reveals the main problem themes, their volume and new emerging issues. The value comes from faster insight and automation without labelling costs; the trade-off is that the discovered groups need human review before they drive decisions.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Represent each ticket as a vector (TF-IDF or sentence embeddings).
Cluster the vectors (k-means or HDBSCAN) to group similar tickets.
Label clusters by reading top terms and sample tickets (“password reset”, “billing double charge”, “app crash on login”).
Act:
Route clusters to the right team automatically.
Write help-centre articles for the largest clusters.
Alert engineering when a new cluster grows quickly (emerging bug).
Measure: resolution time, deflection rate and repeat contacts before and after.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer
tickets = [
"cannot reset my password", "password reset email never arrives", "forgot password link broken",
"charged twice this month", "double charge on my card", "refund for duplicate payment",
"app crashes when I log in", "login screen crashes on android", "app closes after login",
]
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(tickets) # rows are L2-normalized by default
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
terms = np.array(vec.get_feature_names_out())
for c in range(3):
top = terms[np.argsort(km.cluster_centers_[c])[::-1][:3]]
print(f"cluster {c}: {', '.join(top)} -> {np.sum(km.labels_ == c)} tickets")
Other High-Value Scenarios
Area
Unsupervised use
Marketing
Customer segmentation for targeted campaigns
Finance
Anomaly detection on transactions for fraud review
Manufacturing
Detecting abnormal sensor patterns before failures
Retail
Market-basket rules for bundles and store layout
Operations
Grouping similar suppliers, routes or incidents
Data quality
Finding duplicate or inconsistent records
Common Pitfalls
Starting without a decision the clusters will inform.
Presenting clusters as facts without validation by domain experts.
Not monitoring: clusters drift as products and customers change, so re-cluster periodically.
33
How is unsupervised learning used for image segmentation?
KnownRevise
Unsupervised image segmentation divides an image into regions of similar pixels without any labelled masks, by clustering pixels on colour, position or learned features. Classic methods include k-means on colours, mean shift, graph-based methods, superpixels (SLIC) and watershed; modern approaches cluster features from self-supervised vision models. It is cheap and needs no annotation, but the regions it finds are based on appearance, so they may not match meaningful objects.
Read full answerHide full answer
Applies toscikit-learn 1.5+ and scikit-image 0.24+ (Python 3.12)
Methods
Method
Groups pixels by
Notes
k-means on colours
Colour (RGB, Lab)
Simple colour quantization; ignores spatial layout unless coordinates are added
Gaussian mixture
Colour distribution
Soft assignments
Mean shift
Density peaks in colour + position
No fixed k; slower
SLIC superpixels
Colour + position, local k-means
Compact regions used as building blocks
Felzenszwalb (graph-based)
Edge weights between neighbouring pixels
Adapts to region detail
Watershed
Image gradient “basins”
Separates touching objects; needs markers
Self-supervised features + clustering
Deep features (e.g. from DINO-style models)
Regions closer to semantic objects
Code Example
import numpy as np
from skimage import data
from skimage.color import rgb2lab
from skimage.segmentation import slic
from sklearn.cluster import KMeans
image = data.astronaut() # (512, 512, 3) RGB
lab = rgb2lab(image) # perceptually more uniform colour space
# 1) k-means on colour + scaled position
h, w, _ = lab.shape
yy, xx = np.mgrid[0:h, 0:w]
features = np.column_stack([lab.reshape(-1, 3), 0.2 * yy.ravel(), 0.2 * xx.ravel()])
labels = KMeans(n_clusters=6, n_init=4, random_state=42).fit_predict(features)
segmentation = labels.reshape(h, w)
print("k-means segments:", np.unique(segmentation).size)
# 2) SLIC superpixels: many small, compact regions
superpixels = slic(image, n_segments=200, compactness=10, start_label=0)
print("superpixels:", superpixels.max() + 1)
The position weight (0.2 here) controls how much spatial closeness matters relative to colour.
Use Cases
Pre-segmenting images to speed up manual annotation.
Medical imaging: separating tissue types by intensity.
Satellite imagery: land-cover regions.
Background removal and colour quantization.
Common Pitfalls
Clustering only colour, so regions with the same colour far apart get one label.
Expecting object-level segments: shadows and textures split objects.
High resolution makes pixel-level k-means slow; work on superpixels or downsampled images.
34
How do recommendation systems use unsupervised learning?
KnownRevise
Recommendation systems use unsupervised learning to find structure in user behaviour without labelled “good recommendation” examples: matrix factorization discovers hidden taste factors, clustering groups similar users or items, nearest-neighbour search finds similar items, and association rules find items bought together. These methods are the backbone of collaborative filtering and candidate generation. The trade-off is that they struggle with new users and items (cold start) and reflect past behaviour, including its biases.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Techniques
Technique
What it learns
Recommendation use
Matrix factorization (SVD, ALS, NMF)
Latent factors for users and items from the interaction matrix
Predict missing ratings; embeddings for retrieval
Clustering users or items
Groups with similar behaviour
Recommend what is popular in a user’s cluster; cold-start fallbacks
k-nearest neighbours
Similar users or items by cosine similarity
“Users like you liked…” / “Similar items”
Association rules
Items that co-occur in baskets
“Frequently bought together”
Embeddings (word2vec on sessions, autoencoders)
Dense item vectors from sequences
Session-based recommendations
Topic models
Themes in item descriptions
Content-based similarity
How Matrix Factorization Works
The user-item matrix R (users x items, mostly empty) is approximated as R ≈ U V^T:
U holds a k-dimensional vector for each user and V one for each item.
The predicted score for user u and item i is the dot product u_u · v_i.
Factors are learned by minimizing the error on observed entries plus regularization; they often align with real tastes (genre, price level) without being told.
Code Example
import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.metrics.pairwise import cosine_similarity
from scipy.sparse import csr_matrix
rng = np.random.default_rng(42)
n_users, n_items = 300, 50
taste = rng.integers(0, 3, n_users) # hidden user groups
item_genre = rng.integers(0, 3, n_items)
prob = np.where(taste[:, None] == item_genre[None, :], 0.4, 0.03)
interactions = csr_matrix((rng.random((n_users, n_items)) < prob).astype(np.float32))
svd = TruncatedSVD(n_components=5, random_state=42)
user_vecs = svd.fit_transform(interactions) # users in latent space
item_vecs = svd.components_.T # items in latent space
user = 0
scores = user_vecs[user] @ item_vecs.T
scores[interactions[user].indices] = -np.inf # do not recommend what they already have
print("recommended items:", np.argsort(-scores)[:5], "| user's taste group:", taste[user])
print("item genres of recommendations:", item_genre[np.argsort(-scores)[:5]])
similar = cosine_similarity(item_vecs[[0]], item_vecs)[0].argsort()[::-1][1:4]
print("items similar to item 0:", similar)
Common Pitfalls
Treating missing interactions as dislikes; usually they are unknown.
Popularity bias: factorization favours already popular items.
Cold start: new items have no interactions, so combine with content features.
Evaluating only offline; confirm with online A/B tests.
35
How is unsupervised learning applied to anomaly detection?
KnownRevise
Unsupervised anomaly detection learns what “normal” data looks like and flags points that do not fit, without needing labelled examples of anomalies. Methods measure isolation (Isolation Forest), local density (Local Outlier Factor), distance to clusters, probability under a density model, or reconstruction error from an autoencoder. This is valuable because real anomalies such as fraud or faults are rare and often new; the trade-off is more false positives and the need to choose a threshold without labels.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Methods
Method
Anomaly score
Strengths
Weaknesses
Isolation Forest
Short average path length in random trees
Fast, scales well, no scaling needed
Weaker on local anomalies in dense regions
Local Outlier Factor
Density much lower than neighbours’ density
Finds local anomalies
O(n^2)-like cost; needs scaling
One-Class SVM
Outside a learned boundary
Flexible with kernels
Slow on large data; sensitive to parameters
Gaussian mixture / KDE
Low probability density
Probabilistic scores
Struggles in high dimensions
Clustering distance (k-means, DBSCAN noise)
Far from centroids or labelled noise
Simple, interpretable
Depends on clustering quality
Autoencoder
High reconstruction error
Images, sequences, complex data
Needs tuning and enough normal data
Workflow
Train on data that is mostly normal (clean it if you can).
Compute an anomaly score for each point.
Choose a threshold: an expected contamination rate, a review budget (top 100 per day) or a percentile.
Have experts review flagged cases; their feedback provides labels for later supervised or semi-supervised models.
Code Example
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
normal = rng.normal(0, 1, size=(1000, 4))
anomalies = rng.uniform(-6, 6, size=(15, 4))
X = StandardScaler().fit_transform(np.vstack([normal, anomalies]))
is_anomaly = np.r_[np.zeros(1000), np.ones(15)].astype(bool)
iso = IsolationForest(n_estimators=300, contamination=0.015, random_state=42).fit(X)
iso_flags = iso.predict(X) == -1 # -1 means anomaly
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.015)
lof_flags = lof.fit_predict(X) == -1
for name, flags in [("IsolationForest", iso_flags), ("LOF", lof_flags)]:
caught = (flags & is_anomaly).sum()
print(f"{name}: flagged {flags.sum()}, true anomalies caught {caught}/15")
Common Pitfalls
Training on data full of anomalies, which then look normal.
Setting contamination arbitrarily; it directly sets how many points are flagged.
Ignoring seasonality and context (a large transaction is normal on payday).
Evaluating without any labels at all; even a small labelled sample helps estimate precision.
36
What are Generative Adversarial Networks and how do they work?
KnownRevise
A Generative Adversarial Network (GAN) is a pair of neural networks trained against each other: a generator that turns random noise into synthetic samples, and a discriminator that tries to tell real samples from generated ones. As the discriminator gets better at spotting fakes, the generator learns to produce more realistic data. GANs can generate sharp, realistic images without labels, but training is unstable and can suffer from mode collapse, where the generator produces only a few kinds of output.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
How Training Works
The two networks play a minimax game:
min_G max_D E_x[log D(x)] + E_z[log(1 - D(G(z)))]
x: a real sample; z: a random noise vector; G(z): a generated sample.
D(x): the discriminator’s probability that x is real.
The discriminator maximizes the objective (classify correctly); the generator minimizes it (fool the discriminator).
Each training step: 1. Discriminator step: train on a batch of real samples (label 1) and generated samples (label 0). 2. Generator step: generate a batch and update G so that D labels it real. In practice the generator maximizes log D(G(z)) (the “non-saturating” loss), which gives stronger gradients early on.
Code Example
import torch
from torch import nn
torch.manual_seed(42)
real_data = torch.randn(2000, 1) * 0.5 + 3.0 # target: N(3, 0.5^2)
G = nn.Sequential(nn.Linear(8, 32), nn.ReLU(), nn.Linear(32, 1))
D = nn.Sequential(nn.Linear(1, 32), nn.LeakyReLU(0.2), nn.Linear(32, 1)) # outputs logits
opt_g = torch.optim.Adam(G.parameters(), lr=1e-3)
opt_d = torch.optim.Adam(D.parameters(), lr=1e-3)
bce = nn.BCEWithLogitsLoss()
for step in range(3000):
real = real_data[torch.randint(0, 2000, (64,))]
fake = G(torch.randn(64, 8))
# Discriminator: real -> 1, fake -> 0 (detach so G is not updated here)
d_loss = bce(D(real), torch.ones(64, 1)) + bce(D(fake.detach()), torch.zeros(64, 1))
opt_d.zero_grad(); d_loss.backward(); opt_d.step()
# Generator: make D output 1 for fakes
g_loss = bce(D(fake), torch.ones(64, 1))
opt_g.zero_grad(); g_loss.backward(); opt_g.step()
with torch.no_grad():
samples = G(torch.randn(5000, 8))
print("generated mean/std:", samples.mean().item(), samples.std().item()) # should approach 3 and 0.5
Variants
Variant
Idea
DCGAN
Convolutional generator and discriminator for images
Conditional GAN
Generate samples of a requested class
WGAN / WGAN-GP
Wasserstein loss with gradient penalty for more stable training
CycleGAN
Image-to-image translation without paired data
StyleGAN
High-quality, controllable face and image synthesis
Common Pitfalls
Mode collapse: limited variety; mitigations include WGAN-GP and minibatch discrimination.
Imbalance: a discriminator that wins too easily gives the generator no useful gradient.
Evaluation: losses do not show quality; use metrics such as FID plus visual inspection.
Diffusion models now often beat GANs on image quality, but GANs remain much faster at sampling.
37
What is a Variational Autoencoder (VAE)?
KnownRevise
A Variational Autoencoder (VAE) is a generative model built like an autoencoder, but its encoder outputs a probability distribution over a latent space (a mean and a variance) instead of a single point. Training balances reconstructing the input with keeping that distribution close to a standard normal, which makes the latent space smooth, so you can sample new points and decode them into new data. VAEs are stable to train and give meaningful latent spaces; their main trade-off is that generated images tend to be blurrier than those from GANs or diffusion models.
Read full answerHide full answer
Applies toPyTorch 2.x (Python 3.12)
How It Works
Encoderq(z | x): maps input x to a mean mu and log-variance log sigma^2 for each latent dimension.
Reparameterization trick: sample z = mu + sigma * eps with eps ~ N(0, I). Randomness is moved into eps, so gradients can flow through mu and sigma.
Decoderp(x | z): reconstructs x from z.
Loss (negative ELBO): L = reconstruction_loss(x, x_hat) + KL(q(z | x) || N(0, I))
The reconstruction term (MSE or binary cross-entropy) makes outputs match inputs.
The KL term, -0.5 * sum(1 + log sigma^2 - mu^2 - sigma^2), pulls each encoded distribution toward the standard normal, keeping the latent space continuous.
VAE Versus Autoencoder Versus GAN
Aspect
Autoencoder
VAE
GAN
Latent code
A point
A distribution
Random noise input
Can generate new data
Not reliably
Yes, sample z ~ N(0, I)
Yes
Training stability
Stable
Stable
Often unstable
Sample sharpness
n/a
Often blurry
Sharp
Has an encoder
Yes
Yes
No (standard GAN)
Code Example
import torch
from torch import nn
import torch.nn.functional as F
class VAE(nn.Module):
def __init__(self, d=784, h=256, k=16):
super().__init__()
self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU())
self.mu = nn.Linear(h, k)
self.logvar = nn.Linear(h, k)
self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d), nn.Sigmoid())
def forward(self, x):
hidden = self.enc(x)
mu, logvar = self.mu(hidden), self.logvar(hidden)
z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu) # reparameterization trick
return self.dec(z), mu, logvar
def vae_loss(x, x_hat, mu, logvar):
recon = F.binary_cross_entropy(x_hat, x, reduction="sum")
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return (recon + kl) / x.size(0)
torch.manual_seed(42)
model = VAE()
x = torch.rand(32, 784) # stand-in for a batch of MNIST images in [0, 1]
x_hat, mu, logvar = model(x)
print("loss:", vae_loss(x, x_hat, mu, logvar).item())
with torch.no_grad():
new_images = model.dec(torch.randn(8, 16)) # generate 8 new samples
print(new_images.shape) # torch.Size([8, 784])
Common Pitfalls
Posterior collapse: the decoder ignores z when the KL term dominates; use KL warm-up or a beta schedule.
Beta-VAE: weighting the KL term more gives more disentangled factors at the cost of reconstruction quality.
Using MSE on binary data or BCE on unbounded data; match the loss to the data.
38
How does unsupervised learning contribute to NLP?
KnownRevise
Unsupervised learning is central to modern NLP: most language knowledge is learned from huge amounts of raw, unlabelled text. Word embeddings, topic models, text clustering and, above all, self-supervised language-model pre-training (predicting the next or masked word) all learn from text without human labels. This lets small labelled datasets go much further; the trade-off is heavy compute and models that absorb the biases and errors present in their training text.
Read full answerHide full answer
Applies toNLP concepts (examples in scikit-learn 1.5+, Python 3.12)
Contributions
Technique
What it learns
Uses
Word embeddings (word2vec, GloVe, fastText)
Dense word vectors where similar words are close
Features, similarity, analogies
Self-supervised language models (BERT, GPT-style)
Contextual representations of tokens and sentences
Fine-tuning, zero-shot tasks, generation
Sentence embeddings
Vectors for whole sentences or documents
Semantic search, clustering, deduplication
Topic models (LDA, NMF)
Themes as distributions over words
Exploring document collections
Document clustering
Groups of similar texts
Ticket triage, news grouping
Subword tokenization (BPE, WordPiece)
Vocabulary from frequent character sequences
Handling rare and unknown words
Why It Works: the Distributional Hypothesis
“Words that occur in similar contexts tend to have similar meanings.” By predicting words from their context (or the reverse), a model has to encode meaning, syntax and some world knowledge into its representations.
Code Example
from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer
docs = [
"the striker scored a goal in the final match", "the team won the league after a late goal",
"the coach praised the defenders after the match", "the central bank raised interest rates",
"inflation and interest rates worry investors", "stock markets fell as the bank tightened policy",
"the new phone has a faster chip and better camera", "the laptop battery and chip got upgrades",
]
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(docs)
nmf = NMF(n_components=3, random_state=42, max_iter=500)
doc_topics = nmf.fit_transform(X) # document-topic weights
terms = vec.get_feature_names_out()
for t, weights in enumerate(nmf.components_):
print(f"topic {t}:", ", ".join(terms[i] for i in weights.argsort()[::-1][:4]))
print("topic of each doc:", doc_topics.argmax(axis=1))
Common Pitfalls
Bias: embeddings and language models reproduce stereotypes from their training text.
Topic models need careful preprocessing (stop words, lemmatization) and a chosen number of topics.
Static embeddings give one vector per word, so “bank” (river) and “bank” (money) share it; contextual models fix this.
39
What is the role of unsupervised pre-training in deep learning?
KnownRevise
Unsupervised pre-training trains a neural network on large amounts of unlabelled data first, so it learns general-purpose representations, and then fine-tunes it on a smaller labelled dataset for the actual task. Historically it helped initialize deep networks layer by layer; today it lives on as self-supervised pre-training, the basis of large language models and modern vision models. It greatly reduces the labelled data needed, at the cost of expensive pre-training compute and the risk of inheriting biases from the pre-training data.
Read full answerHide full answer
Applies todeep learning concepts (examples in PyTorch 2.x, Python 3.12)
Historical Role (around 2006-2012)
Deep networks were hard to train with random initialization (vanishing gradients).
Researchers pre-trained layers one at a time as restricted Boltzmann machines (deep belief networks) or autoencoders, then fine-tuned the whole network with labels.
Better activations (ReLU), initialization schemes, batch normalization and large labelled datasets later made this layer-wise approach unnecessary for supervised training.
Modern Role: Self-Supervised Pre-Training
The model creates its own targets from unlabelled data:
Domain
Pretext task
Example models
Text
Predict the next token, or masked tokens
GPT-style models, BERT
Images
Contrastive learning (match two augmented views), masked patch reconstruction
SimCLR, MoCo, DINO, MAE
Audio
Predict masked audio segments
wav2vec 2.0
Multimodal
Match images with their captions
CLIP
Why It Helps
Data efficiency: fine-tuning needs far fewer labels.
Better generalization: representations capture general structure (grammar, edges, shapes).
Transfer: one pre-trained model serves many downstream tasks.
Regularization effect: starting from good weights steers optimization toward solutions that generalize.
Code Example
import torch
from torch import nn
torch.manual_seed(42)
unlabelled = torch.rand(5000, 64) # plenty of unlabelled data
labelled_x, labelled_y = torch.rand(200, 64), torch.randint(0, 3, (200,)) # few labels
encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 16))
decoder = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 64))
# Stage 1: unsupervised pre-training (reconstruct the input)
opt = torch.optim.Adam([*encoder.parameters(), *decoder.parameters()], lr=1e-3)
for _ in range(200):
loss = nn.functional.mse_loss(decoder(encoder(unlabelled)), unlabelled)
opt.zero_grad(); loss.backward(); opt.step()
# Stage 2: fine-tune encoder + a new classification head on the labelled data
classifier = nn.Sequential(encoder, nn.ReLU(), nn.Linear(16, 3))
opt = torch.optim.Adam(classifier.parameters(), lr=1e-4) # smaller lr keeps pre-trained features
for _ in range(100):
loss = nn.functional.cross_entropy(classifier(labelled_x), labelled_y)
opt.zero_grad(); loss.backward(); opt.step()
print("fine-tuning loss:", round(loss.item(), 3))
Common Pitfalls
A domain mismatch between pre-training and target data reduces the benefit.
Fine-tuning with a high learning rate destroys the pre-trained features (“catastrophic forgetting”).
Interview tip: connect the historical answer (layer-wise RBM/autoencoder pre-training) to today’s self-supervised foundation models.
40
What are self-organizing maps in unsupervised learning?
KnownRevise
A self-organizing map (SOM), or Kohonen map, is a neural network that projects high-dimensional data onto a low-dimensional grid of neurons (usually 2-D) while preserving topology: inputs that are similar end up on the same or nearby grid cells. It is used for visualization, clustering and exploring structure in data. Its trade-off is that the grid size, learning rate and neighbourhood schedule must be tuned, and it is less common today than PCA, t-SNE or UMAP.
Read full answerHide full answer
Applies tomachine learning concepts (examples in NumPy 2.x, Python 3.12)
How It Works
Grid: create neurons on a grid (e.g. 10 x 10); each has a weight vector the same size as the input.
Initialize the weights randomly or from the first principal components.
For each training samplex:
Best matching unit (BMU): find the neuron whose weight vector is closest to x.
Update the BMU and its grid neighbours toward x: w_j = w_j + eta(t) * h(j, BMU, t) * (x - w_j) where eta(t) is the learning rate and h is a neighbourhood function (often Gaussian in grid distance) whose radius shrinks over time.
Decay the learning rate and neighbourhood radius as training proceeds: early steps organize the global layout, later steps fine-tune.
Using a Trained SOM
U-matrix: shows distances between neighbouring neurons; high values mark cluster boundaries.
Component planes: one map per input feature, showing how it varies across the grid.
Clustering: cluster the neuron weights (e.g. with k-means), then map each sample to its BMU’s cluster.
Code Example
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(42)
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
rows, cols, epochs = 8, 8, 30
weights = rng.normal(size=(rows, cols, X.shape[1]))
grid = np.stack(np.meshgrid(np.arange(rows), np.arange(cols), indexing="ij"), axis=-1)
n_steps, step = epochs * len(X), 0
for epoch in range(epochs):
for x in X[rng.permutation(len(X))]:
frac = step / n_steps
lr, radius = 0.5 * (1 - frac), max(4.0 * (1 - frac), 0.5) # both decay over time
bmu = np.unravel_index(np.argmin(((weights - x) ** 2).sum(axis=2)), (rows, cols))
grid_dist2 = ((grid - np.array(bmu)) ** 2).sum(axis=2)
h = np.exp(-grid_dist2 / (2 * radius ** 2))[..., None] # Gaussian neighbourhood
weights += lr * h * (x - weights)
step += 1
bmus = [np.unravel_index(np.argmin(((weights - x) ** 2).sum(axis=2)), (rows, cols)) for x in X]
for species in range(3):
cells = {bmus[i] for i in np.where(y == species)[0]}
print(f"species {species} occupies {len(cells)} grid cells")
Libraries such as MiniSom provide tested implementations for real use.
SOM Versus Other Methods
Method
Output
Notes
SOM
Discrete grid with prototypes
Topology-preserving, prototypes are interpretable
k-means
Unordered centroids
No neighbourhood relation between clusters
PCA
Linear projection
Fast, no grid
t-SNE / UMAP
Continuous 2-D embedding
Better local structure, no prototypes
Common Pitfalls
Not scaling features before training.
A grid that is too small merges distinct groups; too large leaves many empty cells.
Reading grid distances as exact distances in the original space.
41
How do you implement k-means clustering from scratch in Python?
KnownRevise
Implementing k-means from scratch takes three parts: initialize k centroids (k-means++ is best), then alternate between assigning each point to its nearest centroid and moving each centroid to the mean of its points, until the centroids stop moving. Writing it with NumPy shows how simple the algorithm is and where its weaknesses come from. In production, use sklearn.cluster.KMeans, which is faster and handles edge cases.
Read full answerHide full answer
Applies toNumPy 2.x (Python 3.12)
The Algorithm
k-means++ initialization: pick the first centroid at random; pick each next one with probability proportional to its squared distance from the nearest chosen centroid.
Assignment: label_i = argmin_j ||x_i - c_j||^2.
Update: c_j = mean of points with label j.
Stop when centroids move less than a tolerance or after max_iter iterations.
Restart several times and keep the run with the lowest inertia (sum of squared distances).
Code Example
import numpy as np
def kmeans_plus_plus(X, k, rng):
centroids = [X[rng.integers(len(X))]]
for _ in range(1, k):
d2 = np.min(((X[:, None, :] - np.array(centroids)[None]) ** 2).sum(axis=2), axis=1)
centroids.append(X[rng.choice(len(X), p=d2 / d2.sum())])
return np.array(centroids)
def kmeans(X, k, n_init=10, max_iter=300, tol=1e-6, seed=42):
"""Return (labels, centroids, inertia) of the best of n_init runs."""
rng = np.random.default_rng(seed)
best = None
for _ in range(n_init):
centroids = kmeans_plus_plus(X, k, rng)
for _ in range(max_iter):
d2 = ((X[:, None, :] - centroids[None]) ** 2).sum(axis=2) # (n, k) squared distances
labels = d2.argmin(axis=1)
new = np.array([X[labels == j].mean(axis=0) if np.any(labels == j)
else X[rng.integers(len(X))] # re-seed an empty cluster
for j in range(k)])
shift = np.sum((new - centroids) ** 2)
centroids = new
if shift < tol:
break
d2 = ((X[:, None, :] - centroids[None]) ** 2).sum(axis=2)
labels = d2.argmin(axis=1)
inertia = d2[np.arange(len(X)), labels].sum()
if best is None or inertia < best[2]:
best = (labels, centroids, inertia)
return best
rng = np.random.default_rng(0)
X = np.vstack([rng.normal(loc, 0.6, size=(100, 2)) for loc in ([0, 0], [4, 4], [0, 5])])
labels, centroids, inertia = kmeans(X, k=3)
print("centroids:\n", centroids.round(2)) # close to (0,0), (4,4), (0,5) in some order
print("cluster sizes:", np.bincount(labels), "inertia:", round(inertia, 1))
Complexity
O(n * k * d) per iteration for n points, k clusters and d features; this version builds an (n, k, d) array, so for large data compute distances in chunks.
Common Pitfalls
Random initialization without restarts, which often gets stuck in poor local minima.
Not handling empty clusters (they produce NaN centroids).
Forgetting to scale features before clustering.
Comparing your result with scikit-learn label by label: cluster numbering is arbitrary, so compare with the Adjusted Rand Index.
42
How do you compute the silhouette coefficient in Python?
KnownRevise
The silhouette coefficient of a clustering is the average, over all points, of s(i) = (b(i) - a(i)) / max(a(i), b(i)), where a(i) is the mean distance from point i to the other points in its cluster and b(i) the mean distance to the points of the nearest other cluster. Writing it yourself shows exactly what it measures; in practice you use sklearn.metrics.silhouette_score. The trade-off is O(n^2) distance computations, so large datasets need sampling.
Read full answerHide full answer
Applies toNumPy 2.x and scikit-learn 1.5+ (Python 3.12)
Algorithm
Compute the pairwise distance matrix D.
For each point i in cluster C:
If C has only one point, s(i) = 0.
a(i) = mean of D[i, j] for the other points j in C.
b(i) = the minimum, over other clusters K, of the mean of D[i, j] for j in K.
s(i) = (b(i) - a(i)) / max(a(i), b(i)).
Return the mean of s(i). It needs at least 2 clusters and fewer clusters than points.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import pairwise_distances, silhouette_score
def silhouette_coefficient(X, labels, metric="euclidean"):
"""Mean silhouette coefficient, matching sklearn.metrics.silhouette_score."""
X, labels = np.asarray(X), np.asarray(labels)
unique = np.unique(labels)
if not 2 <= len(unique) <= len(X) - 1:
raise ValueError("need 2 <= number of clusters <= n_samples - 1")
D = pairwise_distances(X, metric=metric)
scores = np.zeros(len(X))
for i in range(len(X)):
same = labels == labels[i]
n_same = same.sum()
if n_same == 1:
continue # singleton cluster: s(i) = 0
a = D[i, same].sum() / (n_same - 1) # exclude the point itself (distance 0)
b = min(D[i, labels == k].mean() for k in unique if k != labels[i])
scores[i] = (b - a) / max(a, b)
return scores.mean()
X, _ = make_blobs(n_samples=300, centers=3, random_state=42)
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
print("custom :", round(silhouette_coefficient(X, labels), 6))
print("sklearn:", round(silhouette_score(X, labels), 6)) # the two values match
Notes
Complexity: O(n^2) time and memory for the distance matrix; for large n, use silhouette_score(X, labels, sample_size=10000, random_state=42).
Per-point scores: sklearn.metrics.silhouette_samples returns s(i) for silhouette plots.
Metric: pass metric="cosine" for text or embeddings.
Common Pitfalls
Dividing a(i) by the full cluster size instead of size minus one.
Including DBSCAN noise points (label -1) as if they were a cluster.
Comparing silhouette values computed on differently scaled features.
43
How do you reduce dimensions with PCA in scikit-learn?
KnownRevise
PCA in scikit-learn is done with sklearn.decomposition.PCA: you standardize the features, fit PCA on the training data, and transform the data into a smaller number of principal components that keep most of the variance. You can set the number of components directly or as a target share of variance (e.g. n_components=0.95). The trade-off is between fewer dimensions (faster models, less noise) and the information that is discarded.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Steps
Split the data first, so PCA learns only from training data.
Standardize features with StandardScaler; PCA is variance-based, so units matter.
Fit PCA and inspect explained_variance_ratio_ to choose the number of components.
Transform training and test data with the same fitted PCA.
Use a Pipeline to combine scaling, PCA and a model safely.
Code Example
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True) # 30 features
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
# 1. Inspect variance to choose the number of components
scaler = StandardScaler().fit(X_train)
pca_full = PCA(random_state=42).fit(scaler.transform(X_train))
cumulative = np.cumsum(pca_full.explained_variance_ratio_)
print("variance kept by 2, 5, 10 components:", cumulative[[1, 4, 9]].round(3))
# 2. Keep enough components for 95% of the variance
pca = PCA(n_components=0.95, random_state=42).fit(scaler.transform(X_train))
X_train_pca = pca.transform(scaler.transform(X_train))
X_test_pca = pca.transform(scaler.transform(X_test))
print("components kept:", pca.n_components_, "| shapes:", X_train_pca.shape, X_test_pca.shape)
# 3. The same thing inside a pipeline, evaluated with cross-validation
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression(max_iter=1000))
print("CV accuracy with PCA:", cross_val_score(pipe, X_train, y_train, cv=5).mean().round(3))
# 4. Which original features drive the first component (loadings)
top = np.argsort(np.abs(pca.components_[0]))[::-1][:3]
print("top features in PC1:", top)
Useful Options
Option
Purpose
n_components=int
Exact number of components
n_components=0.95
Smallest number keeping 95% of variance
svd_solver="randomized"
Faster for large data and few components
whiten=True
Scales components to unit variance (some models benefit)
inverse_transform
Maps back to the original space (reconstruction)
IncrementalPCA
Fits in batches when data does not fit in memory
TruncatedSVD
For sparse matrices (no centering)
Common Pitfalls
Fitting PCA on all data before splitting (leakage).
Skipping standardization when features have different units.
Assuming the components are interpretable features; inspect components_ (loadings) instead.
44
How do you cluster spatial data with DBSCAN in Python?
KnownRevise
DBSCAN on spatial data groups locations that are densely packed (for example, pickup points, store visits or crime reports) and labels isolated points as noise, without needing the number of clusters. For latitude/longitude data, use the haversine distance so eps is a real distance on the Earth’s surface, and a ball-tree index to keep it fast. The main trade-off is choosing eps and min_samples, which set what “dense” means.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Steps
Prepare coordinates: latitude and longitude in degrees, converted to radians for haversine.
Choose eps: a physical radius, e.g. 300 metres, divided by the Earth’s radius (about 6,371 km) to get radians.
Choose min_samples: the minimum number of points within eps to form a dense area.
Inspect: number of clusters, noise share, and a representative point per cluster.
Code Example
import numpy as np
from sklearn.cluster import DBSCAN
rng = np.random.default_rng(42)
# Synthetic GPS points around three hotspots in a city, plus scattered noise
hotspots = np.array([[40.7580, -73.9855], [40.7128, -74.0060], [40.7306, -73.9352]])
points = np.vstack([h + rng.normal(0, 0.0015, size=(150, 2)) for h in hotspots] +
[rng.uniform([40.70, -74.02], [40.78, -73.93], size=(60, 2))])
EARTH_RADIUS_M = 6_371_000
eps_m = 300 # neighbourhood radius in metres
db = DBSCAN(eps=eps_m / EARTH_RADIUS_M, min_samples=10,
metric="haversine", algorithm="ball_tree")
labels = db.fit_predict(np.radians(points)) # haversine expects [lat, lon] in radians
n_clusters = len(set(labels) - {-1})
print("clusters:", n_clusters, "| noise points:", (labels == -1).sum())
for c in range(n_clusters):
centre = points[labels == c].mean(axis=0)
print(f"cluster {c}: {np.sum(labels == c)} points, centre {centre.round(4)}")
Choosing Parameters
k-distance plot: compute each point’s distance to its min_samples-th neighbour, sort, and pick eps at the knee.
Domain sense: 50 m for points inside one building, 1 km for neighbourhoods.
HDBSCAN (sklearn.cluster.HDBSCAN) handles clusters of varying density, such as dense downtown and sparse suburbs.
Common Pitfalls
Using Euclidean distance on raw degrees: one degree of longitude shrinks toward the poles, so distances are distorted.
Passing coordinates in [lon, lat] order; haversine in scikit-learn expects [lat, lon].
One eps for areas of very different density, which merges dense areas or labels sparse ones as noise.
Taking the centroid of a curved or ring-shaped cluster as a meaningful location; consider the medoid instead.
45
How do you implement the Apriori algorithm in Python?
KnownRevise
Implementing Apriori means finding all itemsets whose support meets a threshold by growing them one item at a time: start with frequent single items, join frequent k-itemsets into (k + 1)-candidates, prune candidates that have an infrequent subset, count the rest, and repeat. Association rules are then generated from the frequent itemsets using confidence and lift. A from-scratch version is great for understanding; for large data use FP-Growth or a library.
Read full answerHide full answer
Applies toPython 3.12 (standard library only)
Code Example
from itertools import combinations
def apriori(transactions, min_support):
"""Return {frozenset(itemset): support} for all frequent itemsets."""
transactions = [frozenset(t) for t in transactions]
n = len(transactions)
def frequent_of(candidates):
counts = {c: sum(c <= t for t in transactions) for c in candidates}
return {c: cnt / n for c, cnt in counts.items() if cnt / n >= min_support}
items = {frozenset([i]) for t in transactions for i in t}
current = frequent_of(items) # frequent 1-itemsets
result, k = dict(current), 1
while current:
prev = list(current)
# Join step: unions of two frequent k-itemsets that have size k + 1
candidates = {a | b for a, b in combinations(prev, 2) if len(a | b) == k + 1}
# Prune step: every k-subset must be frequent (the Apriori property)
candidates = {c for c in candidates
if all(frozenset(s) in current for s in combinations(c, k))}
current = frequent_of(candidates)
result.update(current)
k += 1
return result
def rules(frequent, min_confidence):
out = []
for itemset, sup in frequent.items():
for r in range(1, len(itemset)):
for antecedent in map(frozenset, combinations(itemset, r)):
consequent = itemset - antecedent
conf = sup / frequent[antecedent]
if conf >= min_confidence:
lift = conf / frequent[consequent]
out.append((set(antecedent), set(consequent), round(sup, 2), round(conf, 2), round(lift, 2)))
return sorted(out, key=lambda x: -x[4])
transactions = [
{"bread", "milk"},
{"bread", "diapers", "beer", "eggs"},
{"milk", "diapers", "beer", "cola"},
{"bread", "milk", "diapers", "beer"},
{"bread", "milk", "diapers", "cola"},
]
frequent = apriori(transactions, min_support=0.6)
for itemset, sup in sorted(frequent.items(), key=lambda x: (len(x[0]), -x[1])):
print(sorted(itemset), round(sup, 2))
# 4 single items (bread, milk, diapers 0.8; beer 0.6) and 4 pairs at 0.6; no triple reaches 0.6
for a, c, s, conf, lift in rules(frequent, min_confidence=0.7):
print(f"{a} -> {c}: support={s}, confidence={conf}, lift={lift}")
Every subset of a frequent itemset is itself frequent, so the lookups frequent[antecedent] and frequent[consequent] always succeed.
Complexity and Improvements
Each level scans all transactions for every candidate: roughly O(candidates x transactions x itemset size).
Speed-ups: transaction ID lists (Eclat), hash trees for counting, or FP-Growth, which avoids candidates.
For real data, use mlxtend.frequent_patterns or Spark’s FPGrowth.
Common Pitfalls
Generating candidates without the prune step, which counts many itemsets needlessly.
A very low min_support on many items, which explodes the number of candidates.
Reporting rules by confidence alone; always check lift.
46
How would you segment customers for targeted marketing with unsupervised learning?
KnownRevise
Customer segmentation groups customers with similar behaviour or needs so marketing can target each group with suitable messages, offers and channels. A solid strategy builds behavioural features such as recency, frequency and monetary value (RFM), scales them, clusters with k-means or a Gaussian mixture, and then profiles and validates each segment with the business. The trade-off is between many fine segments that are hard to act on and a few broad ones that may hide important differences.
Read full answerHide full answer
Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)
Strategy
Define the goal: retention, upselling, reactivation; it decides which features matter.
Build features per customer from a fixed time window:
Transform and scale: log-transform skewed spend and counts, then standardize.
Cluster: k-means for clear, sized segments; Gaussian mixtures for overlapping groups.
Choose k: silhouette, BIC and how many segments the marketing team can actually serve.
Profile and name each segment (“lapsed high-spenders”, “new bargain hunters”).
Validate: stability across months, different responses in A/B tests.
Operationalize: score customers regularly and push segments to the CRM.
Code Example
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler
rng = np.random.default_rng(42)
n = 2000
rfm = pd.DataFrame({
"recency_days": rng.exponential(60, n).round(),
"frequency": rng.poisson(5, n) + 1,
"monetary": rng.lognormal(5, 0.8, n).round(2),
})
pipe = make_pipeline(FunctionTransformer(np.log1p), StandardScaler())
X = pipe.fit_transform(rfm)
for k in range(3, 7):
labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X)
print(f"k={k}: silhouette={silhouette_score(X, labels):.3f}")
rfm["segment"] = KMeans(n_clusters=4, n_init=10, random_state=42).fit_predict(X)
profile = rfm.groupby("segment").agg(customers=("monetary", "size"), recency=("recency_days", "median"),
frequency=("frequency", "median"), spend=("monetary", "median"))
print(profile.round(1))
Turning Segments Into Actions
Segment profile
Example action
Recent, frequent, high spend
Loyalty perks, early access
High spend but long inactive
Win-back campaign
New, low frequency
Onboarding sequence, second-purchase incentive
Frequent, low spend
Bundles and upselling
Common Pitfalls
Including features the business cannot act on, or IDs and dates that create meaningless clusters.
Not log-transforming spend, so a few big spenders become their own cluster.
Never refreshing segments; behaviour changes over time.
47
How can clustering create features for a supervised model?
KnownRevise
Clustering for feature creation uses unsupervised structure as extra input for a supervised model: the cluster label, the distance to each cluster centre, or soft membership probabilities become new features. They can summarize complex interactions (a “type” of customer or location) that a simple model would struggle to learn. The trade-off is that the features can leak information if the clustering is fitted on test data, and they add little for flexible models like gradient boosting that already learn such groupings.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Ways to Use Clusters as Features
Feature
How
Benefit
Cluster ID
One-hot encode the k-means label
Captures group effects for linear models
Distances to centroids
KMeans.transform(X) gives k distances
Smooth, informative, works well with linear models
Soft memberships
GaussianMixture.predict_proba(X)
Expresses uncertainty between groups
Geographic clusters
DBSCAN or k-means on coordinates
Neighbourhood effects (prices, demand)
Cluster-level statistics
Mean target per cluster, computed out-of-fold
Strong signal, but needs cross-fitting to avoid leakage
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import FeatureUnion, make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler
X, y = make_moons(n_samples=1000, noise=0.25, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
baseline = make_pipeline(StandardScaler(), LogisticRegression())
# Original features + distances to 10 k-means centroids, all fitted inside each CV fold
with_clusters = make_pipeline(
StandardScaler(),
FeatureUnion([
("original", FunctionTransformer()), # identity: keep the inputs
("dist", KMeans(n_clusters=10, n_init=10, random_state=42)), # transform() -> distances
]),
LogisticRegression(max_iter=1000),
)
for name, model in [("logistic regression", baseline), ("+ cluster distances", with_clusters)]:
print(name, cross_val_score(model, X, y, cv=cv, scoring="roc_auc").mean().round(3))
# The cluster distances let a linear model capture the curved class boundary.
When It Helps Most
Linear models or k-NN on data with non-linear group structure.
High-cardinality spatial data (coordinates to neighbourhood clusters).
Combining many correlated behaviour features into a few profile features.
Common Pitfalls
Leakage: fitting the clustering on the full dataset before cross-validation. Put it inside the pipeline as above.
Too many clusters: the features start to memorize the training data.
Unstable clusters: if labels change between runs, the model sees a different feature each time; distances are more stable than IDs.
Expecting gains with tree ensembles; measure with cross-validation before keeping the features.
48
How would you group similar documents with unsupervised learning?
KnownRevise
Grouping similar documents means turning each document into a numeric vector that captures its meaning, then clustering those vectors so documents about the same subject end up together. A practical approach is TF-IDF or sentence embeddings, optional dimensionality reduction, then k-means or HDBSCAN, followed by labelling each cluster with its top terms. The trade-off is between simple, interpretable TF-IDF features and embeddings that understand synonyms but are heavier to compute.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
Approach
Clean text: lowercase, remove boilerplate, signatures and duplicates.
Represent:
TF-IDF: word weights; fast and interpretable.
Sentence embeddings (e.g. from a sentence-transformer model): capture meaning, so “car” and “automobile” are close.
Reduce dimensions: truncated SVD (LSA) for TF-IDF, or UMAP for embeddings.
Normalize vectors to unit length so Euclidean k-means behaves like cosine similarity.
Cluster: k-means when you want a fixed number of groups, HDBSCAN when you want the number found automatically and outliers left out.
Label and review: top terms per cluster and a few sample documents; merge or split clusters as needed.
Evaluate: silhouette with cosine distance, stability, and human review.
Code Example
import numpy as np
from sklearn.cluster import KMeans
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer
docs = [
"team wins championship after penalty shootout", "striker scores twice in derby match",
"coach signs new contract with football club", "central bank keeps interest rates unchanged",
"inflation slows as energy prices fall", "stocks rally after strong earnings reports",
"new smartphone launches with faster processor", "chip maker unveils energy efficient processor",
"software update improves laptop battery life",
]
tfidf = TfidfVectorizer(stop_words="english", sublinear_tf=True)
X = tfidf.fit_transform(docs)
lsa = make_pipeline(TruncatedSVD(n_components=5, random_state=42), Normalizer(copy=False))
X_lsa = lsa.fit_transform(X)
km = KMeans(n_clusters=3, n_init=20, random_state=42).fit(X_lsa)
print("silhouette (cosine):", round(silhouette_score(X_lsa, km.labels_, metric="cosine"), 3))
# Top terms: map centroids back from LSA space to term space
centroids_terms = lsa[0].inverse_transform(km.cluster_centers_)
terms = tfidf.get_feature_names_out()
for c in range(3):
top = terms[np.argsort(centroids_terms[c])[::-1][:3]]
members = [d[:30] for d, l in zip(docs, km.labels_) if l == c]
print(f"cluster {c} [{', '.join(top)}]: {members}")
Common Pitfalls
Clustering raw counts without TF-IDF weighting or normalization, so long documents dominate.
Ignoring duplicates and templates, which form artificial clusters.
Choosing k only by metrics; check that topics make sense to readers.
49
How do you detect communities in social networks with unsupervised learning?
KnownRevise
Community detection finds groups of nodes in a network that are densely connected to each other and sparsely connected to the rest, such as friend circles, interest groups or coordinated accounts. It is unsupervised: the algorithm uses only the graph structure (and optionally node features). A good framework covers graph construction, choosing an algorithm (modularity-based such as Louvain/Leiden, label propagation, spectral or embedding-based), evaluation and interpretation; the trade-off is that different algorithms and resolutions give different, equally plausible partitions.
Read full answerHide full answer
Applies tomachine learning concepts (examples in NetworkX 3.x, Python 3.12)
Framework
Build the graph: nodes are users; edges are follows, messages or interactions, weighted by frequency. Decide directed or undirected, and filter weak or spam edges.
Choose an algorithm:
Algorithm
Idea
Notes
Louvain / Leiden
Greedily maximize modularity, then aggregate communities
Fast, scales to millions of nodes; Leiden avoids badly connected communities
Label propagation
Nodes adopt the most common label among neighbours
Very fast, less stable
Spectral clustering
k-means on eigenvectors of the graph Laplacian
Needs the number of communities
Girvan-Newman
Remove high-betweenness edges
Intuitive but slow; small graphs only
Node embeddings + clustering
node2vec or graph neural networks, then k-means/HDBSCAN
Can use node features too
Overlapping methods
Clique percolation, mixed-membership models
People belong to several communities
Evaluate: modularity Q (fraction of edges inside communities minus the expected fraction in a random graph with the same degrees), conductance, community size distribution and stability across runs.
Interpret: describe communities by shared attributes, topics or central members (high degree or PageRank).
import networkx as nx
from networkx.algorithms import community
G = nx.karate_club_graph() # classic social network of 34 club members
communities = community.louvain_communities(G, resolution=1.0, seed=42)
print("communities found:", len(communities))
print("modularity:", round(community.modularity(G, communities), 3))
for i, c in enumerate(sorted(communities, key=len, reverse=True)):
hub = max(c, key=G.degree) # most connected member
print(f"community {i}: {len(c)} members, hub node {hub}")
lp = community.label_propagation_communities(G)
print("label propagation found:", len(list(lp)), "communities")
Resolution limit: modularity optimization can merge small but real communities.
Hubs and bots distort structure; consider removing or down-weighting them.
Single run: Louvain is randomized; check stability across seeds.
Ignoring time: communities change; use snapshots or temporal methods.
50
How does unsupervised learning help find patterns in genomic data?
KnownRevise
Unsupervised learning in genomics discovers structure in very high-dimensional biological data, such as gene expression for thousands of genes across many samples or cells, without predefined labels. Clustering finds cell types, disease subtypes and co-expressed gene modules; dimensionality reduction reveals population structure and developmental trajectories. It drives discovery, but results are sensitive to normalization and batch effects, so they need biological validation.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Typical Applications
Task
Method
Insight
Cell-type discovery (single-cell RNA-seq)
PCA, then graph clustering (Leiden/Louvain), visualized with UMAP
Ancestry groups; correcting confounding in association studies
Mutational signatures
NMF on mutation counts
Processes such as UV damage or smoking
Trajectory inference
Pseudotime on reduced data
How cells develop or differentiate
Typical Pipeline (Expression Data)
Quality control: remove low-quality samples or cells.
Normalize for sequencing depth and log-transform counts.
Select highly variable genes (often a few thousand).
Correct batch effects if samples came from different runs or labs.
Reduce with PCA (e.g. 50 components).
Cluster on the reduced data.
Interpret: find marker genes per cluster and compare with known biology.
Code Example
import numpy as np
from sklearn.cluster import AgglomerativeClustering
from sklearn.decomposition import PCA
from sklearn.metrics import adjusted_rand_score
rng = np.random.default_rng(42)
n_per_group, n_genes = 100, 2000
groups = np.repeat([0, 1, 2], n_per_group)
counts = rng.poisson(5, size=(len(groups), n_genes)).astype(float)
for g in range(3): # each group over-expresses 50 marker genes
counts[groups == g, g * 50:(g + 1) * 50] += rng.poisson(15, size=(n_per_group, 50))
lib_size = counts.sum(axis=1, keepdims=True)
log_norm = np.log1p(counts / lib_size * 1e4) # depth-normalize, then log
variable = np.argsort(log_norm.var(axis=0))[::-1][:500] # top variable genes
pcs = PCA(n_components=20, random_state=42).fit_transform(log_norm[:, variable])
labels = AgglomerativeClustering(n_clusters=3, linkage="ward").fit_predict(pcs)
print("ARI vs simulated groups:", round(adjusted_rand_score(groups, labels), 3))
for c in range(3): # marker genes: highest mean difference
diff = log_norm[labels == c].mean(axis=0) - log_norm[labels != c].mean(axis=0)
print(f"cluster {c} top marker genes:", np.argsort(diff)[::-1][:5])
Dedicated tools such as Scanpy or Seurat implement these workflows for real single-cell data.
Common Pitfalls
Batch effects that look like biological clusters.
p >> n: thousands of genes and few samples; reduce dimensions and validate stability.
Over-clustering: splitting one cell type into many clusters at high resolution.
Treating t-SNE or UMAP distances as biological distances.
51
What are recent advances in clustering algorithms?
KnownRevise
Recent advances in clustering have focused on handling varying densities without hand-tuned parameters, scaling to huge datasets, clustering learned representations instead of raw features, and graph-based methods for complex data. Notable examples include HDBSCAN, deep clustering on self-supervised embeddings, the Leiden algorithm for graph communities, UMAP-then-cluster pipelines and scalable approximate methods. The trade-off is often more moving parts: new methods depend on embeddings or graphs that themselves need tuning.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Key Developments
Advance
Idea
Why it matters
HDBSCAN
Builds a density hierarchy and keeps the most stable clusters
Varying densities, no eps, noise labelling; in scikit-learn since 1.3
Clustering on embeddings
Cluster vectors from pretrained models (sentence transformers, vision models)
Semantic text, image and audio clusters
Deep clustering
Train the representation and cluster assignments together (e.g. DEC, SCAN, DeepCluster)
Better clusters on images and text
Graph community detection
Leiden improves Louvain with guaranteed well-connected communities
Standard in single-cell genomics and networks
UMAP + density clustering
Reduce with UMAP, cluster with HDBSCAN
Popular for topic modelling (e.g. BERTopic)
Scalable methods
Mini-batch and streaming k-means, approximate nearest-neighbour graphs, GPU libraries
Millions to billions of points
Spectral clustering at scale
Approximations (Nystrom, landmarks)
Non-convex shapes on large data
Clustering with LLMs
Embeddings for grouping, LLMs to name and summarize clusters
Faster interpretation of results
Fair and constrained clustering
Balance protected groups or respect must-link/cannot-link constraints
Responsible, domain-aware results
Code Example
import numpy as np
from sklearn.cluster import DBSCAN, HDBSCAN
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score
# Clusters of very different densities: hard for one DBSCAN eps
X, y = make_blobs(n_samples=[400, 400, 100], centers=[[0, 0], [6, 6], [12, 0]],
cluster_std=[0.3, 1.5, 0.8], random_state=42)
hdb = HDBSCAN(min_cluster_size=20).fit(X)
db = DBSCAN(eps=0.5, min_samples=10).fit(X)
for name, labels in [("HDBSCAN", hdb.labels_), ("DBSCAN eps=0.5", db.labels_)]:
print(f"{name}: clusters={len(set(labels) - {-1})}, noise={np.sum(labels == -1)}, "
f"ARI={adjusted_rand_score(y, labels):.3f}")
Practical Takeaways
For text, images and audio, the biggest improvement usually comes from better embeddings, not a new clustering algorithm.
HDBSCAN is a strong default when the number of clusters is unknown and densities vary.
Always validate: newer methods still produce clusters on noise.
Common Pitfalls
Chasing state-of-the-art benchmarks that do not match your data.
Clustering on t-SNE output, which can create false gaps.
Assuming deep clustering works with small datasets; it needs large data or strong pretrained features.
52
How is unsupervised learning used in reinforcement learning?
KnownRevise
Unsupervised learning supports reinforcement learning (RL) by learning useful representations, models and exploration signals from experience without reward labels. Examples include learning compact state representations from pixels, world models that predict future observations, curiosity-driven exploration based on prediction error, discovering skills without rewards, and pre-training on unlabelled data. These methods make RL more sample-efficient, especially when rewards are sparse; the trade-off is extra components that can learn features irrelevant to the actual task.
Read full answerHide full answer
Applies tomachine learning concepts (deep RL research and practice)
Main Uses
Use
Idea
Examples
State representation learning
Compress high-dimensional observations into informative features
Autoencoders, contrastive learning (CURL), data augmentation methods
World models
Learn to predict the next observation or latent state, then plan or train in imagination
World Models (VAE + RNN), Dreamer, MuZero-style learned models
Intrinsic motivation / curiosity
Reward the agent for surprising or novel states
Intrinsic Curiosity Module (ICM), Random Network Distillation (RND), count-based bonuses
Unsupervised skill discovery
Learn diverse behaviours without external reward
DIAYN (Diversity Is All You Need)
Pre-training
Learn from unlabelled videos or offline logs before fine-tuning with rewards
Pretrained visual encoders, offline RL datasets
Auxiliary tasks
Extra self-supervised losses alongside the RL loss
UNREAL (pixel control, reward prediction)
State abstraction and clustering
Group similar states to simplify planning
Clustering of latent states, options discovery
How Curiosity Works (Example)
A model predicts the next state’s features from the current state and action.
The prediction error is added to the reward as an intrinsic bonus.
The agent seeks states it cannot yet predict, so it explores; as the model improves, the bonus fades.
RND makes this robust by predicting the output of a fixed random network, which avoids being distracted by inherently unpredictable noise to some extent.
Benefits
Sample efficiency: good representations reduce how many environment interactions are needed.
Sparse rewards: intrinsic rewards drive exploration when external rewards are rare (e.g. games where the first reward comes after many steps).
Transfer: skills and representations learned without rewards can be reused across tasks.
Common Pitfalls
The noisy TV problem: naive curiosity is attracted to random, unpredictable signals.
Irrelevant features: reconstruction losses spend capacity on background pixels that do not matter for control.
Model errors compound when planning with a learned world model over long horizons.
Interview tip: give one concrete method per category (e.g. RND for exploration, Dreamer for world models).
53
Why is interpretability challenging for unsupervised learning models?
KnownRevise
Interpretability in unsupervised learning is challenging because there is no target to anchor explanations: a cluster, component or anomaly score has no predefined meaning, so humans must infer what it represents. Results also depend on preprocessing, distance measures and random initialization, and methods like t-SNE, UMAP or autoencoders create abstract, non-linear representations. Good practice combines profiling, surrogate models and stability checks, accepting that some uncertainty about meaning always remains.
Read full answerHide full answer
Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)
Main Challenges
No ground truth: you cannot check an explanation against correct labels.
Arbitrary choices shape results: features, scaling, distance metric, k and algorithm all change the clusters.
Abstract representations: PCA components mix many features; autoencoder codes and embeddings have no direct meaning.
Instability: different seeds or samples can give different clusters with different “stories”.
Visualization traps: t-SNE and UMAP distort distances and cluster sizes.
Anomaly scores: “unusual” in which way? Scores rarely say why.
Confirmation bias: analysts can find a plausible narrative for almost any cluster.
Techniques That Help
Technique
What it explains
Cluster profiling
Mean or median of each feature per cluster versus the overall average
Surrogate model
A shallow decision tree trained to predict cluster labels; its rules describe clusters
SHAP on the surrogate
Which features separate each cluster
PCA loadings
Which original features drive each component
Prototypes / medoids
Representative real examples per cluster
Stability analysis
Which clusters are robust and worth interpreting
Feature attribution for anomalies
Which features deviate most for a flagged point
Code Example
from sklearn.cluster import KMeans
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier, export_text
data = load_wine(as_frame=True)
X = data.data
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(StandardScaler().fit_transform(X))
# 1. Profile: how each cluster differs from the overall mean (in standard deviations)
profile = (X.groupby(labels).mean() - X.mean()) / X.std()
print(profile[["alcohol", "flavanoids", "color_intensity", "proline"]].round(2))
# 2. Surrogate tree: human-readable rules that reproduce the clusters
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, labels)
print("surrogate fidelity:", round(tree.score(X, labels), 3)) # how well the rules match the clusters
print(export_text(tree, feature_names=list(X.columns)))
Common Pitfalls
Naming clusters from one or two features while ignoring the rest of the profile.
Explaining unstable clusters that disappear with another seed.
Presenting a surrogate’s rules without its fidelity score.
54
How is unsupervised learning used for multilingual text analysis?
KnownRevise
Unsupervised learning for multilingual text relies on representations that place texts with the same meaning close together regardless of language. Multilingual language models and cross-lingual embeddings are learned from unlabelled or loosely aligned text, so you can cluster, search, deduplicate and topic-model documents in many languages together without translation or labelled data. The trade-off is weaker quality for low-resource languages and possible language bias, where texts group by language instead of topic.
Read full answerHide full answer
Applies toNLP concepts (examples in sentence-transformers 3.x and scikit-learn 1.5+, Python 3.12)
Techniques
Technique
How it is learned
Use
Aligned word embeddings
Monolingual embeddings mapped to a shared space (e.g. adversarial alignment as in MUSE, or Procrustes with a small dictionary)
Bilingual lexicon induction, word translation
Multilingual language models
Self-supervised masked or next-token prediction on many languages at once (mBERT, XLM-R)
Shared contextual representations
Multilingual sentence embeddings
Trained so translations map to the same vector (e.g. LaBSE, multilingual sentence-transformers)
Cross-lingual search, clustering
Cross-lingual topic modelling
Cluster multilingual embeddings, then extract topic words per language
Topics across a global corpus
Unsupervised machine translation
Back-translation and denoising without parallel data
Translation for low-resource pairs
Example Workflow: Clustering Customer Feedback in Many Languages
Embed each message with a multilingual sentence-embedding model.
Normalize the vectors and optionally reduce them with UMAP.
Cluster with k-means or HDBSCAN.
Check that clusters mix languages (a sign they group by topic, not language).
Label clusters with top terms or an LLM summary per cluster.
Code Example
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
texts = [
"The delivery was very late", "La entrega llegó muy tarde", "Die Lieferung kam sehr spät",
"Great product quality", "Excelente calidad del producto", "Sehr gute Produktqualität",
"I want a refund", "Quiero un reembolso", "Ich möchte eine Rückerstattung",
]
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
emb = model.encode(texts, normalize_embeddings=True) # same meaning -> nearby vectors
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(emb)
for c in range(3):
print(c, [t for t, l in zip(texts, labels) if l == c])
# Expected: each cluster groups one topic across English, Spanish and German.
Common Pitfalls
Clusters forming by language because the embedding model is not truly cross-lingual.
Low-resource languages and code-switched text getting poor embeddings.
Tokenization and script differences when using TF-IDF; monolingual bag-of-words features do not align across languages.
Evaluating only in English; check examples in every language.
55
What role does unsupervised learning play in big data analytics?
KnownRevise
Unsupervised learning in big data analytics turns huge volumes of unlabelled data (logs, clicks, transactions, sensor streams, text) into structure people can use: segments, anomalies, topics, compressed features and associations. It matters because labelling at that scale is impractical, and most of the data has no target anyway. The trade-off is that algorithms must be scalable (distributed, streaming or approximate) and results must be validated, since patterns at scale can be spurious or driven by data quality issues.