Skip to content
Machine Learning & Data Science

Unsupervised Learning interview questions and answers

55 questions, 52 with code. About 2 h 46 min to read them all.

By TestLab · Updated

Plan for an interview
55 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is unsupervised learning and how does it differ from supervised learning?

Unsupervised learning finds structure in data that has no labels: groups of similar items, a smaller set of underlying dimensions, unusual points or items that occur together. Supervised learning, by contrast, learns a mapping from inputs to known target labels. Unsupervised methods are useful when labels are expensive or unknown, but their results are harder to evaluate because there is no ground truth to compare against.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Key Differences

Aspect Supervised learning Unsupervised learning
Data Inputs X with labels y Inputs X only
Goal Predict y for new inputs Discover structure in X
Typical tasks Classification, regression Clustering, dimensionality reduction, density estimation, anomaly detection, association rules
Evaluation Compare predictions with true labels (accuracy, RMSE) Internal metrics (silhouette, reconstruction error), stability, usefulness downstream, expert review
Examples Spam filter, price prediction Customer segments, topic discovery, fraud outliers

Main Unsupervised Task Types

  • Clustering: k-means, hierarchical clustering, DBSCAN/HDBSCAN, Gaussian mixtures.

  • Dimensionality reduction: PCA, t-SNE, UMAP, autoencoders.

  • Density estimation: Gaussian mixtures, kernel density estimation.

  • Anomaly detection: Isolation Forest, Local Outlier Factor.

  • Association rules: Apriori, FP-Growth for “items bought together”.

  • Self-supervised learning: a modern related approach that creates labels from the data itself (predicting masked words or image patches) to learn representations, as in large language models.

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, _ = load_iris(return_X_y=True)      # labels are ignored: this is unsupervised

for k in range(2, 6):
    pipe = make_pipeline(StandardScaler(), PCA(n_components=2, random_state=42),
                         KMeans(n_clusters=k, n_init=10, random_state=42))
    labels = pipe.fit_predict(X)
    X_reduced = pipe[:-1].transform(X)  # data in the space where clustering happened
    print(f"k={k}: silhouette={silhouette_score(X_reduced, labels):.3f}")

Common Pitfalls

  • Scaling and distance choice change the result completely; standardize features for distance-based methods.

  • Clusters can be artefacts: algorithms always return clusters, even in random data.

  • The number of clusters is a judgement supported by metrics, not a fact the data reveals.

  • Interview tip: mention semi-supervised learning (few labels plus many unlabelled points) as the middle ground.

2

What types of problems does unsupervised learning solve?

Unsupervised learning addresses problems where you have data but no target labels, and you want to discover its structure. The main problem types are clustering, dimensionality reduction, density estimation, anomaly detection and association rule mining, with self-supervised representation learning as a modern extension. Each answers a different question about the data, and each is evaluated without ground truth, which makes validation the main challenge.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Problem Types

Problem Question it answers Typical methods Example use
Clustering Which items are similar to each other? k-means, hierarchical, DBSCAN/HDBSCAN, Gaussian mixtures Customer segments, grouping documents
Dimensionality reduction What few directions capture most of the information? PCA, t-SNE, UMAP, autoencoders Visualization, compression, noise removal
Density estimation How likely is a given point? Gaussian mixtures, kernel density estimation Probability models, sampling new data
Anomaly detection Which points do not fit the usual pattern? Isolation Forest, Local Outlier Factor, one-class SVM Fraud, equipment faults, intrusion detection
Association rules Which items occur together? Apriori, FP-Growth Market-basket analysis, cross-selling
Representation learning What compact features describe the data? Autoencoders, self-supervised models Pretrained embeddings for later supervised tasks

How They Fit Together

  • Dimensionality reduction often comes before clustering, to remove noise and speed up distance calculations.

  • Clustering and density estimation are closely related: a Gaussian mixture does both.

  • Anomaly detection can be built from density (low-density points are anomalies) or from reconstruction error (points an autoencoder reconstructs poorly).

  • Learned representations are often used as features for supervised models.

Code Example

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.ensemble import IsolationForest
from sklearn.mixture import GaussianMixture

X, _ = make_blobs(n_samples=500, centers=3, random_state=42)
X = np.vstack([X, [[15, 15], [-15, 12]]])          # two obvious outliers

gmm = GaussianMixture(n_components=3, random_state=42).fit(X)
print("cluster sizes:", np.bincount(gmm.predict(X)))           # clustering
print("log-density of first point:", gmm.score_samples(X[:1]))  # density estimation

iso = IsolationForest(contamination=0.01, random_state=42).fit(X)
print("outlier rows:", np.where(iso.predict(X) == -1)[0])        # anomaly detection

Common Pitfalls

  • Choosing the method before stating the business question.

  • Treating clusters or anomalies as facts without checking them with domain experts or a downstream task.

  • Interview tip: give one concrete use case per problem type; it shows practical understanding.

3

What is dimensionality reduction and why is it important?

Dimensionality reduction transforms data with many features into a representation with fewer features while keeping as much useful information as possible. It matters because high-dimensional data is slow to process, hard to visualize, prone to overfitting and affected by the curse of dimensionality, where distances become less meaningful. The trade-off is that some information is always lost, and the new features are often harder to interpret.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Two Families

  • Feature selection: keep a subset of the original features (variance threshold, correlation filtering, model-based selection). Interpretable, because features keep their meaning.

  • Feature extraction: build new features as combinations of the original ones (PCA, autoencoders, UMAP). Usually keeps more information in fewer dimensions.

Common Methods

Method Type Strengths Limitations
PCA Linear, unsupervised Fast, keeps maximum variance, invertible Misses non-linear structure; needs scaling
LDA Linear, supervised Maximizes class separation Needs labels; at most classes - 1 components
t-SNE Non-linear Excellent 2-D visualization of local clusters Slow; distances between clusters are not meaningful; no transform for new data in scikit-learn
UMAP Non-linear Faster than t-SNE, keeps more global structure, can transform new data Separate library; hyperparameter-sensitive
Autoencoders Non-linear, neural Flexible, works on images and text Needs more data and tuning

Why It Is Important

  • Visualization: project data to 2-D or 3-D to see clusters and outliers.

  • Speed and memory: fewer features mean faster training and smaller models.

  • Noise reduction: discarding low-variance components often removes noise.

  • Less overfitting: fewer parameters to fit, especially when features outnumber samples.

  • Multicollinearity: PCA components are uncorrelated, which helps linear models.

Code Example

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, _ = load_digits(return_X_y=True)          # 64 pixel features
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95, random_state=42))
X_reduced = pipe.fit_transform(X)

pca = pipe.named_steps["pca"]
print("original features:", X.shape[1])
print("components for 95% variance:", pca.n_components_)
print("variance kept:", pca.explained_variance_ratio_.sum().round(3))

Common Pitfalls

  • Not scaling features before PCA, so large-unit features dominate.

  • Fitting the reducer on all data before a train/test split (leakage).

  • Reading t-SNE cluster sizes or distances as meaningful.

  • Reducing dimensions when interpretability of the original features is the main requirement.

4

What is clustering and how does it provide insights into data?

Clustering is the unsupervised task of grouping data points so that points in the same group (cluster) are more similar to each other than to points in other groups. It gives insight by revealing natural segments, typical profiles and outliers that were not labelled in advance, such as customer types or topics in documents. The trade-off is that clusters depend heavily on the features, scaling, distance measure and algorithm, so they must be validated rather than taken as facts.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Main Families of Algorithms

Family Example Cluster shape Notes
Centroid-based k-means Roughly spherical, similar size Fast; needs k; sensitive to scaling and outliers
Hierarchical Agglomerative (Ward, average linkage) Depends on linkage Dendrogram shows structure at every level; O(n^2) memory
Density-based DBSCAN, HDBSCAN Arbitrary shapes Finds noise points; no k needed
Model-based Gaussian mixture Elliptical Soft (probabilistic) assignments

How Clustering Produces Insights

  1. Segment: group customers by behaviour (frequency, spend, product mix).

  2. Profile: compute the average feature values per cluster and name them (“bargain hunters”, “loyal premium buyers”).

  3. Act: tailor marketing, pricing or product decisions per segment.

  4. Spot outliers: points far from every cluster, or labelled noise by DBSCAN, can be fraud or data errors.

  5. Simplify: replace thousands of items with a handful of representative groups.

Code Example

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
customers = pd.DataFrame({
    "orders_per_year": np.concatenate([rng.normal(2, 1, 200), rng.normal(12, 3, 150), rng.normal(30, 5, 50)]),
    "avg_order_value": np.concatenate([rng.normal(40, 10, 200), rng.normal(60, 15, 150), rng.normal(150, 30, 50)]),
}).clip(lower=0)

X = StandardScaler().fit_transform(customers)
customers["segment"] = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)

# Profile each segment in the original units
print(customers.groupby("segment").agg(["mean", "count"]).round(1))

Evaluating Clusters

  • Internal metrics: silhouette (higher is better), Davies-Bouldin (lower is better), inertia for k-means.

  • Stability: do the clusters persist with different seeds or bootstrap samples?

  • Usefulness: do segments behave differently on business outcomes, and do domain experts recognize them?

Common Pitfalls

  • Forgetting to scale features, so one large-unit feature decides the clusters.

  • Assuming clusters exist because the algorithm returned them.

  • Using k-means on elongated or nested shapes; try DBSCAN or Gaussian mixtures.

5

What is the difference between hard and soft clustering?

Hard clustering assigns each data point to exactly one cluster, while soft clustering (fuzzy or probabilistic clustering) gives each point a degree of membership in every cluster, such as 70% cluster A and 30% cluster B. Soft clustering captures uncertainty and overlapping groups, which is realistic for many real-world segments. The trade-off is that soft assignments are harder to act on and the models are usually slower and have more parameters.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Comparison

Aspect Hard clustering Soft clustering
Output One label per point A membership vector per point (sums to 1)
Examples k-means, DBSCAN, agglomerative clustering Gaussian mixture models, fuzzy c-means, LDA topic models
Overlapping groups Not represented Represented naturally
Uncertainty Lost: a borderline point looks as certain as a central one Explicit
Interpretation Simple segments Richer, but needs thresholds for decisions
Cost Usually faster Usually slower (e.g. EM iterations over all components)

How Soft Clustering Works

  • In a Gaussian mixture model, each cluster is a Gaussian distribution. The membership (responsibility) of point x in cluster k is r_k(x) = pi_k * N(x | mu_k, Sigma_k) / sum_j pi_j * N(x | mu_j, Sigma_j), where pi_k is the cluster’s weight, mu_k its mean and Sigma_k its covariance.

  • Fuzzy c-means minimizes a weighted distance in which each point’s membership in each cluster is raised to a “fuzzifier” power that controls softness.

  • A soft clustering becomes hard by taking the cluster with the highest membership.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture

X, _ = make_blobs(n_samples=300, centers=[[0, 0], [3, 0]], cluster_std=1.2, random_state=42)

hard = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
gmm = GaussianMixture(n_components=2, random_state=42).fit(X)
soft = gmm.predict_proba(X)                 # shape (300, 2), rows sum to 1

borderline = np.where(np.abs(soft[:, 0] - 0.5) < 0.1)[0]
print("hard labels of first 5:", hard[:5])
print("soft memberships of first 5:\n", soft[:5].round(2))
print("points with roughly 50/50 membership:", len(borderline))

When to Use Which

  • Hard: clear, well-separated groups, or when each item must get one action (one marketing campaign per customer).

  • Soft: overlapping groups, documents covering several topics, risk scoring where uncertainty matters.

Common Pitfalls

  • Treating soft memberships as calibrated probabilities when the model’s assumptions (e.g. Gaussian shapes) do not hold.

  • Converting to hard labels too early and losing the uncertainty that motivated soft clustering.

6

How does the k-means clustering algorithm work?

K-means is a centroid-based clustering algorithm that splits data into k groups by repeatedly assigning each point to its nearest centroid and moving each centroid to the mean of its assigned points. It minimizes the within-cluster sum of squared distances (inertia). It is fast and simple, but you must choose k, it assumes roughly spherical clusters of similar size and it is sensitive to scaling, outliers and the starting centroids.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

The Algorithm (Lloyd’s Algorithm)

  1. Initialize k centroids, usually with k-means++ (spreads initial centroids apart).

  2. Assign each point to the nearest centroid (Euclidean distance).

  3. Update each centroid to the mean of the points assigned to it.

  4. Repeat steps 2-3 until assignments stop changing or the improvement is below a tolerance.

Objective

Inertia = sum_{i=1..n} || x_i - mu_{c(i)} ||^2

  • x_i is a data point, c(i) the cluster it is assigned to, and mu_{c(i)} that cluster’s centroid.

  • Each step can only lower (or keep) the inertia, so the algorithm always converges, but possibly to a local minimum. Running several initializations (n_init) and keeping the best reduces that risk.

Complexity: O(n * k * d) per iteration for n points, k clusters and d features.

Choosing k

  • Elbow method: plot inertia against k and look for the bend.

  • Silhouette score: pick the k with the highest average silhouette.

  • Business sense: the number of segments a team can actually act on.

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=600, centers=4, cluster_std=1.0, random_state=42)
X = StandardScaler().fit_transform(X)

for k in range(2, 7):
    km = KMeans(n_clusters=k, init="k-means++", n_init=10, random_state=42).fit(X)
    print(f"k={k}: inertia={km.inertia_:.1f}, silhouette={silhouette_score(X, km.labels_):.3f}")
# The silhouette should peak at k=4 for this data.

Variants

  • MiniBatchKMeans: updates centroids from small batches; much faster on large data.

  • k-medoids: uses actual data points as centres; more robust to outliers.

  • Bisecting k-means (BisectingKMeans): splits clusters recursively.

Common Pitfalls

  • Not scaling features: a feature in thousands dominates one in units.

  • Using k-means on non-spherical shapes (moons, rings) or clusters of very different density; use DBSCAN or Gaussian mixtures.

  • Comparing inertia across different k directly: it always decreases as k grows.

  • Categorical data: means are meaningless; use k-modes or other methods.

7

What does the silhouette coefficient measure in clustering?

The silhouette coefficient measures how well each point fits its own cluster compared with the nearest other cluster, giving a score from -1 to 1. Averaged over all points, it summarizes clustering quality without labels and is widely used to compare different numbers of clusters or algorithms. Its trade-off is that it favours compact, convex, well-separated clusters, so it can undervalue good density-based clusters with irregular shapes, and it costs O(n^2) distance computations.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Formula

For a point i:

s(i) = (b(i) - a(i)) / max(a(i), b(i))

  • a(i): the average distance from i to the other points in its own cluster (cohesion).

  • b(i): the lowest average distance from i to the points of any other cluster (separation from the nearest neighbouring cluster).

  • A point alone in its cluster gets s(i) = 0 by convention.

The overall score is the mean of s(i) over all points.

Interpreting Values

Score Meaning
Close to 1 Point is well inside its cluster and far from others
Around 0 Point is on the border between two clusters
Negative Point is probably in the wrong cluster

As a rough guide, an average above 0.5 suggests clear structure and below 0.25 suggests weak or artificial structure, though these thresholds depend on the data.

How It Is Used

  • Choosing k: compute the average silhouette for several values of k and prefer the highest.

  • Comparing algorithms: k-means versus agglomerative clustering on the same features.

  • Silhouette plots: per-point scores grouped by cluster reveal clusters with many poorly assigned points.

Code Example

from sklearn.cluster import AgglomerativeClustering, KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_samples, silhouette_score

X, _ = make_blobs(n_samples=500, centers=3, cluster_std=1.5, random_state=42)

for name, model in [("kmeans", KMeans(n_clusters=3, n_init=10, random_state=42)),
                    ("agglomerative", AgglomerativeClustering(n_clusters=3))]:
    labels = model.fit_predict(X)
    per_point = silhouette_samples(X, labels)
    print(f"{name}: mean silhouette={silhouette_score(X, labels):.3f}, "
          f"negative points={(per_point < 0).sum()}")

Common Pitfalls

  • Using it for DBSCAN results with noise points: decide whether to exclude the noise label (-1) first.

  • Comparing scores computed with different distance metrics or feature scalings.

  • On large datasets, use silhouette_score(X, labels, sample_size=10000, random_state=42) to keep it affordable.

  • Treating the highest silhouette as the “true” k; combine it with domain knowledge.

8

How does DBSCAN work, and what advantages does it have over k-means?

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that lie in dense regions and marks points in sparse regions as noise. A cluster grows from “core” points that have enough neighbours within a radius eps. Unlike k-means, it does not need the number of clusters, finds clusters of arbitrary shape and handles outliers explicitly; its trade-off is sensitivity to eps and trouble with clusters of very different densities.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

  • Parameters: eps (neighbourhood radius) and min_samples (points needed within eps, including the point itself, to be a core point).

  • Point types:

  • Core point: has at least min_samples points within eps.

  • Border point: within eps of a core point but not a core point itself.

  • Noise point: neither; labelled -1.

  • Algorithm:

  • Pick an unvisited point and find its eps-neighbours.

  • If it is a core point, start a cluster and add all density-reachable points (neighbours of core points, recursively).

  • Otherwise mark it as noise for now (it may later become a border point).

  • Repeat until all points are visited.

DBSCAN Versus k-means

Aspect k-means DBSCAN
Number of clusters Must be given Found automatically
Cluster shape Convex, roughly spherical Arbitrary (moons, rings)
Outliers Forced into a cluster, pulling centroids Labelled as noise
Varying densities Handles if shapes are spherical Struggles: one eps fits all
Parameters k eps, min_samples
Predict on new data Yes (predict) No predict in scikit-learn; re-fit or assign by nearest core point
Complexity O(n * k * d) per iteration About O(n log n) with a spatial index; up to O(n^2) worst case

Code Example

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score
from sklearn.preprocessing import StandardScaler

X, y_true = make_moons(n_samples=500, noise=0.06, random_state=42)
X = StandardScaler().fit_transform(X)

db = DBSCAN(eps=0.3, min_samples=5).fit(X)
km = KMeans(n_clusters=2, n_init=10, random_state=42).fit(X)

print("DBSCAN clusters:", len(set(db.labels_) - {-1}), "noise points:", (db.labels_ == -1).sum())
print("ARI DBSCAN:", round(adjusted_rand_score(y_true, db.labels_), 3))   # close to 1
print("ARI k-means:", round(adjusted_rand_score(y_true, km.labels_), 3))  # much lower on moons

Choosing eps

  • Plot the sorted distance of each point to its min_samples-th nearest neighbour (k-distance plot) and choose eps at the “knee”.

  • A common start for min_samples is 2 * n_features.

  • HDBSCAN (sklearn.cluster.HDBSCAN) removes the need for eps and handles varying densities better.

Common Pitfalls

  • Not scaling features: eps is a distance, so units matter.

  • High-dimensional data: distances concentrate and density loses meaning; reduce dimensions first.

9

How does hierarchical clustering work and when should you use it?

Hierarchical clustering builds a tree of nested clusters (a dendrogram) instead of a single partition. The common agglomerative version starts with every point as its own cluster and repeatedly merges the two closest clusters until one remains; cutting the tree at a chosen height gives the final clusters. It is useful when you want to see structure at several levels or do not know the number of clusters, but it needs O(n^2) memory, so it does not scale to very large datasets.

Read full answer

Applies toscikit-learn 1.5+ and SciPy 1.13+ (Python 3.12)

How Agglomerative Clustering Works

  1. Compute the distance between every pair of points.

  2. Treat each point as a cluster.

  3. Merge the two clusters with the smallest linkage distance.

  4. Update distances between the new cluster and the others.

  5. Repeat until one cluster remains, recording each merge and its distance.

Linkage Criteria

Linkage Distance between clusters Behaviour
Ward Increase in within-cluster variance after merging Compact, similar-size clusters; Euclidean only; most common default
Complete Largest pairwise distance Compact clusters, sensitive to outliers
Average Mean pairwise distance Balanced compromise
Single Smallest pairwise distance Finds elongated shapes but suffers from “chaining”

When to Use It

  • Small to medium datasets (up to tens of thousands of points).

  • Unknown number of clusters: the dendrogram shows natural cut points (large vertical gaps).

  • Hierarchies matter: taxonomies of species, product categories, gene expression groups.

  • Any distance metric: works with precomputed distances such as cosine or edit distance (except Ward).

Code Example

from scipy.cluster.hierarchy import fcluster, linkage
from sklearn.cluster import AgglomerativeClustering
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=200, centers=3, random_state=42)
X = StandardScaler().fit_transform(X)

# SciPy: full merge tree, then cut it into 3 clusters
Z = linkage(X, method="ward")              # each row: [cluster_a, cluster_b, distance, size]
labels_scipy = fcluster(Z, t=3, criterion="maxclust")
print("last 3 merge distances:", Z[-3:, 2].round(2))   # big jumps suggest a good cut

# scikit-learn: same idea with an estimator API
labels = AgglomerativeClustering(n_clusters=3, linkage="ward").fit_predict(X)
print("silhouette:", round(silhouette_score(X, labels), 3))
# Plot with scipy.cluster.hierarchy.dendrogram(Z) in a notebook.

Advantages and Limitations

  • Advantages: no need to fix k in advance; deterministic; produces an interpretable hierarchy.

  • Limitations: O(n^2) memory and at least O(n^2) time; merges cannot be undone, so early mistakes persist; sensitive to noise and to the linkage choice.

Common Pitfalls

  • Not scaling features before computing distances.

  • Using Ward linkage with non-Euclidean distances.

  • Running it on hundreds of thousands of rows; use k-means, MiniBatchKMeans or HDBSCAN instead, or cluster a sample.

10

What is the difference between agglomerative and divisive hierarchical clustering?

Agglomerative and divisive clustering are the two directions for building a cluster hierarchy. Agglomerative clustering is bottom-up: every point starts as its own cluster and the closest clusters are merged step by step. Divisive clustering is top-down: all points start in one cluster, which is split recursively. Agglomerative is far more common because it is simpler and widely implemented; divisive can capture the large-scale structure better but is expensive unless a heuristic split method is used.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Comparison

Aspect Agglomerative (bottom-up) Divisive (top-down)
Start n clusters of one point One cluster with all points
Step Merge the two closest clusters Split one cluster into two
Decisions focus on Local similarity between small groups Global structure first
Exact cost O(n^2) memory, O(n^2 log n) to O(n^3) time Exponential if all splits are considered
Practical method Linkage rules (Ward, average, complete, single) Heuristics such as DIANA or repeated 2-means (bisecting k-means)
Availability AgglomerativeClustering, scipy.cluster.hierarchy BisectingKMeans in scikit-learn (a divisive approach that returns a flat partition)
Typical use Small/medium datasets, dendrograms When a few top-level groups matter most

How Divisive Clustering Works

  1. Start with all points in one cluster.

  2. Choose a cluster to split (for example the largest, or the one with the highest variance).

  3. Split it into two, e.g. with 2-means or by moving the most dissimilar points out (DIANA).

  4. Repeat until the desired number of clusters or until each cluster is a single point.

Code Example

from sklearn.cluster import AgglomerativeClustering, BisectingKMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score

X, _ = make_blobs(n_samples=1000, centers=5, cluster_std=1.2, random_state=42)

agglo = AgglomerativeClustering(n_clusters=5, linkage="ward").fit_predict(X)              # bottom-up
divisive = BisectingKMeans(n_clusters=5, bisecting_strategy="biggest_inertia",
                           random_state=42).fit_predict(X)                                 # top-down

print("agglomerative silhouette:", round(silhouette_score(X, agglo), 3))
print("bisecting k-means silhouette:", round(silhouette_score(X, divisive), 3))

When to Choose Which

  • Agglomerative: you want a full dendrogram, small or medium data, or a custom distance metric.

  • Divisive: larger data where only a few top-level splits are needed, since you can stop early instead of building the full tree.

Common Pitfalls

  • Saying divisive is always more accurate; it depends on the split heuristic.

  • Forgetting that both are greedy: a bad early merge or split is never undone.

11

How does Principal Component Analysis (PCA) work?

Principal Component Analysis (PCA) is a linear dimensionality-reduction method that finds new, uncorrelated axes (principal components) ordered by how much of the data’s variance they capture, and projects the data onto the first few of them. It solves the problem of many correlated features by summarizing them in fewer dimensions with minimal information loss. The trade-off is that the components are linear combinations of the original features, so they can be hard to interpret and cannot capture non-linear structure.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Step by Step

  1. Center the data: subtract each feature’s mean (and usually standardize to unit variance).

  2. Covariance: compute C = (1/(n-1)) * X^T X for the centered data matrix X (n rows, d features).

  3. Eigen-decomposition: find the eigenvectors v_j and eigenvalues lambda_j of C. Each eigenvector is a principal direction; its eigenvalue is the variance along it.

  4. Sort components by eigenvalue, largest first.

  5. Project: Z = X W_k, where W_k holds the first k eigenvectors as columns.

In practice, libraries use the singular value decomposition X = U S V^T, which is numerically more stable; the rows of V^T are the components and S^2/(n-1) gives the variances.

Key Quantities

  • Explained variance ratio: lambda_j / sum(lambda), the share of total variance a component captures.

  • Loadings: the weights of each original feature in a component; they show what a component “means”.

  • Reconstruction: X_approx = Z W_k^T + mean; the reconstruction error measures what was lost.

Code Example

import numpy as np
from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X, _ = load_wine(return_X_y=True)                 # 13 features
X_std = StandardScaler().fit_transform(X)

pca = PCA(random_state=42).fit(X_std)
cumulative = np.cumsum(pca.explained_variance_ratio_)
k = int(np.argmax(cumulative >= 0.90) + 1)
print("explained variance of first 3:", pca.explained_variance_ratio_[:3].round(3))
print(f"components needed for 90% variance: {k}")

Z = PCA(n_components=k, random_state=42).fit_transform(X_std)
print("reduced shape:", Z.shape)

Common Pitfalls

  • No scaling: features with large units dominate the first components.

  • Leakage: fit PCA on training data only (inside a Pipeline).

  • Assuming high variance means useful: the most variable directions are not always the most predictive.

  • Non-linear data: use kernel PCA, UMAP or autoencoders.

  • Sign of components is arbitrary: two runs can return the same component with flipped signs.

12

What is t-SNE and when should you use it?

t-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear method for visualizing high-dimensional data in two or three dimensions. It keeps points that are close neighbours in the original space close in the plot, which makes local clusters stand out clearly. Its trade-off is that it is slow, stochastic and does not preserve global distances or cluster sizes, so it is a visualization tool rather than a general dimensionality-reduction step.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

  1. High-dimensional similarities: for each point i, define a Gaussian over its neighbours, giving conditional probabilities p_{j|i}. The Gaussian width is chosen so the distribution has a target perplexity (roughly, the effective number of neighbours; default 30). These are symmetrized into joint probabilities p_ij.

  2. Low-dimensional similarities: in the 2-D map, similarities q_ij use a Student t-distribution with one degree of freedom, which has heavy tails. This lets moderately distant points sit far apart and avoids the “crowding problem”.

  3. Optimization: gradient descent moves the 2-D points to minimize the KL divergence KL(P || Q) = sum p_ij * log(p_ij / q_ij). It penalizes placing true neighbours far apart much more than placing non-neighbours close.

Use Cases

  • Exploring embeddings (word vectors, image features, LLM embeddings).

  • Checking whether classes or clusters are separable before building a model.

  • Inspecting single-cell genomics data, where t-SNE plots are standard.

  • Debugging: finding mislabeled points that sit inside another class’s cluster.

Code Example

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_pca = PCA(n_components=30, random_state=42).fit_transform(StandardScaler().fit_transform(X))  # denoise and speed up

embedding = TSNE(n_components=2, perplexity=30, init="pca", random_state=42).fit_transform(X_pca)
print(embedding.shape)   # (1797, 2)
# Plot: plt.scatter(embedding[:, 0], embedding[:, 1], c=y, s=5, cmap="tab10")

t-SNE Versus Alternatives

Method Preserves Speed New data
PCA Global variance, linear structure Very fast transform available
t-SNE Local neighbourhoods Slow on large data No transform in scikit-learn
UMAP Local and more global structure Faster than t-SNE transform available

Common Pitfalls

  • Over-interpreting the plot: cluster sizes, gaps between clusters and positions are not meaningful.

  • Single perplexity: try several values (5-50); structure that appears at all of them is more trustworthy.

  • Using t-SNE features for a model: it has no reliable transform for new points and is not designed for that.

  • Random seeds: different runs give different layouts; fix random_state.

13

How does Linear Discriminant Analysis differ from PCA?

Linear Discriminant Analysis (LDA) and PCA both project data onto a few linear directions, but with different goals. PCA is unsupervised: it finds the directions of maximum variance and ignores labels. LDA is supervised: it finds the directions that best separate known classes, by maximizing between-class variance relative to within-class variance. Use PCA for compression and exploration without labels, and LDA when you want class separation for classification or visualization.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Comparison

Aspect PCA LDA
Learning type Unsupervised Supervised (needs class labels)
Objective Maximize total variance Maximize between-class / within-class scatter
Maximum components min(n_samples, n_features) number of classes - 1
Assumptions None about classes Classes roughly Gaussian with a shared covariance
Also a classifier? No Yes (predict)
Typical use Compression, denoising, visualization, preprocessing Class-separating projection, simple classifier

LDA’s Objective

LDA finds a projection w that maximizes the Fisher criterion

J(w) = (w^T S_B w) / (w^T S_W w)

  • S_B: between-class scatter, how far class means are from the overall mean.

  • S_W: within-class scatter, how spread out points are inside each class.

A high J means the classes are far apart relative to their spread. The solutions are eigenvectors of S_W^-1 S_B, and there are at most C - 1 useful ones for C classes.

When to Use Each

  • PCA: no labels; many correlated features; the goal is to keep information.

  • LDA: labels available; you want a few features that separate classes; small datasets where a simple linear classifier works well.

  • Both: PCA first to remove noise when features outnumber samples (which makes S_W singular), then LDA.

Code Example

from sklearn.datasets import load_wine
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_wine(return_X_y=True)     # 3 classes, so LDA gives at most 2 components

for name, reducer in [("PCA", PCA(n_components=2)), ("LDA", LinearDiscriminantAnalysis(n_components=2))]:
    pipe = make_pipeline(StandardScaler(), reducer, LogisticRegression(max_iter=1000))
    print(name, "2-D accuracy:", cross_val_score(pipe, X, y, cv=5).mean().round(3))
# LDA's 2 components usually separate the wine classes better than PCA's 2.

Common Pitfalls

  • Using LDA without labels, or asking for more than C - 1 components.

  • Fitting LDA on the whole dataset before cross-validation, which leaks labels into the projection.

  • Confusing this LDA with Latent Dirichlet Allocation, the topic model.

14

What is the curse of dimensionality in machine learning?

The curse of dimensionality is the set of problems that appear when data has many features: the space grows so fast that data becomes sparse, distances between points become similar, and models need far more data to learn reliable patterns. It mainly hurts distance-based methods (k-NN, k-means, DBSCAN) and any model that overfits easily. The usual remedies are dimensionality reduction, feature selection, regularization and more data, each with its own cost.

Read full answer

Applies tomachine learning concepts (examples in NumPy 2.x, Python 3.12)

Why It Happens

  • Volume grows exponentially: to cover a unit cube with 10 points per axis takes 10^d points; at d = 10 that is ten billion.

  • Distances concentrate: as d grows, the ratio between the farthest and nearest neighbour distances approaches 1, so “nearest” loses meaning.

  • Most volume is near the edges: in high dimensions, almost all of a hypercube’s volume lies close to its surface, so most points are “outliers” in some direction.

  • More parameters than samples: models can fit noise perfectly (overfitting).

Effects on Models

Model type Effect
k-NN, k-means, DBSCAN Neighbours and clusters become unreliable
Density estimation Needs exponentially more data
Linear models Overfit when features approach or exceed samples, unless regularized
Trees and boosting More robust, but irrelevant features still add noise and cost

Code Example

import numpy as np

rng = np.random.default_rng(42)
for d in (2, 10, 100, 1000):
    X = rng.random((500, d))
    q = rng.random(d)
    dist = np.linalg.norm(X - q, axis=1)
    print(f"d={d:5d}: (max - min) / min distance = {(dist.max() - dist.min()) / dist.min():.3f}")
# The relative contrast shrinks as d grows: all points look about equally far away.

Remedies

  • Dimensionality reduction: PCA, UMAP or autoencoders before distance-based methods.

  • Feature selection: drop irrelevant or redundant features.

  • Regularization: L1/L2 penalties, dropout, tree depth limits.

  • Better distance measures: cosine similarity for text and embeddings.

  • More data, or simpler models that need fewer parameters.

Interview Tips

  • Mention the “blessing of dimensionality” counterpoint: high dimensions can make classes linearly separable, which is why kernel methods and SVMs work.

  • Pitfall: assuming more features always help; irrelevant ones add noise and cost.

15

What is an autoencoder and how does it reduce dimensionality?

An autoencoder is a neural network trained to reproduce its input at its output, with a narrow middle layer (the bottleneck) that forces it to learn a compressed representation. The encoder maps input to this low-dimensional code and the decoder reconstructs the input from it, so the code serves as a learned, non-linear dimensionality reduction. Compared with PCA it can capture curved structure, but it needs more data, tuning and compute, and its features are harder to interpret.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

Architecture

  • Encoder: z = f(x), layers that shrink from the input size d to the code size k (with k < d).

  • Bottleneck (latent code): the k-dimensional vector z, the reduced representation.

  • Decoder: x_hat = g(z), layers that expand back to d.

  • Loss: reconstruction error, e.g. mean squared error L = (1/n) * sum ||x - x_hat||^2.

A linear autoencoder with MSE loss learns the same subspace as PCA; non-linear activations let it go beyond PCA.

Variants

Variant Idea Use
Undercomplete Small bottleneck Dimensionality reduction
Denoising Corrupt input, reconstruct the clean version Robust features, denoising
Sparse Penalty on code activations Interpretable, sparse features
Variational (VAE) Code is a probability distribution Generating new samples
Convolutional Convolution layers Images

Code Example

import torch
from torch import nn
from sklearn.datasets import load_digits
from sklearn.preprocessing import MinMaxScaler

torch.manual_seed(42)
X = torch.tensor(MinMaxScaler().fit_transform(load_digits().data), dtype=torch.float32)  # (1797, 64)

class AutoEncoder(nn.Module):
    def __init__(self, d=64, k=8):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(d, 32), nn.ReLU(), nn.Linear(32, k))
        self.decoder = nn.Sequential(nn.Linear(k, 32), nn.ReLU(), nn.Linear(32, d), nn.Sigmoid())

    def forward(self, x):
        return self.decoder(self.encoder(x))

model = AutoEncoder()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()

for epoch in range(200):
    optimizer.zero_grad()
    loss = loss_fn(model(X), X)      # reconstruct the input itself
    loss.backward()
    optimizer.step()

with torch.no_grad():
    codes = model.encoder(X)         # the 8-dimensional representation
print("final reconstruction MSE:", round(loss.item(), 4))
print("reduced shape:", tuple(codes.shape))   # (1797, 8)

In practice, use mini-batches and a validation set to stop training before the model overfits.

Common Pitfalls

  • A bottleneck that is too wide lets the network copy the input without learning structure.

  • Not scaling inputs to match the output activation (e.g. sigmoid output needs inputs in [0, 1]).

  • Judging only the reconstruction loss; check that the codes are useful for the downstream task.

16

What is association rule mining in unsupervised learning?

Association rule mining finds items or events that frequently occur together in transaction data and expresses them as rules such as {bread, butter} -> {milk}. It is unsupervised because no target is given: the algorithm discovers co-occurrence patterns on its own. It is useful for market-basket analysis, recommendations and finding related events, but it can produce huge numbers of rules, many of them obvious or coincidental, so filtering and domain review are essential.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Key Terms

  • Itemset: a set of items, e.g. {diapers, beer}.

  • Rule A -> B: if a transaction contains A (antecedent), it tends to contain B (consequent).

  • Support(A): fraction of transactions that contain A.

  • Confidence(A -> B) = support(A and B) / support(A): how often the rule is right.

  • Lift(A -> B) = confidence(A -> B) / support(B): how much more likely B is when A is present than in general. Lift above 1 means a positive association.

How It Works

  1. Find frequent itemsets: all itemsets whose support is at least a threshold (min_support), using Apriori, FP-Growth or Eclat.

  2. Generate rules from each frequent itemset by splitting it into antecedent and consequent.

  3. Filter rules by confidence and lift, then review them.

Why It Counts as Unsupervised Learning

  • There is no label to predict; the goal is to describe structure in the data.

  • Like clustering, results are judged by usefulness and interest rather than accuracy against ground truth.

Applications

Domain Example
Retail Products bought together, store layout, bundles
E-commerce “Frequently bought together” recommendations
Web analytics Pages visited in the same session
Healthcare Co-occurring diagnoses or medications
IT operations Alerts that fire together, root-cause hints

Code Example

transactions = [
    {"bread", "milk"},
    {"bread", "diapers", "beer", "eggs"},
    {"milk", "diapers", "beer", "cola"},
    {"bread", "milk", "diapers", "beer"},
    {"bread", "milk", "diapers", "cola"},
]
def support(*items):
    return sum(set(items) <= t for t in transactions) / len(transactions)

a, b = "diapers", "beer"
conf = support(a, b) / support(a)
lift = conf / support(b)
print(f"support={support(a, b):.2f}, confidence={conf:.2f}, lift={lift:.2f}")
# support=0.60, confidence=0.75, lift=1.25

Common Pitfalls

  • High confidence with lift close to 1: the consequent is simply popular (most baskets contain milk).

  • Setting min_support too low, which produces millions of rules.

  • Reading rules as causal: co-purchase does not mean one causes the other.

17

How does the Apriori algorithm find association rules?

Apriori is the classic algorithm for finding frequent itemsets and association rules. It searches level by level, building itemsets of size k + 1 only from frequent itemsets of size k, and relies on the Apriori property: every subset of a frequent itemset must also be frequent, so any candidate with an infrequent subset can be discarded without counting it. It is simple and easy to explain, but it scans the data once per level and can generate many candidates, which makes it slow on large or dense datasets.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

The Algorithm

  1. Level 1: count every single item; keep those with support of at least min_support.

  2. Candidate generation: join frequent k-itemsets that share k - 1 items to form (k + 1)-candidates.

  3. Pruning: remove any candidate that has an infrequent k-subset (the Apriori property).

  4. Counting: scan the transactions to count the remaining candidates; keep the frequent ones.

  5. Repeat steps 2-4 until no new frequent itemsets appear.

  6. Rule generation: for each frequent itemset I and each non-empty proper subset A, form A -> I - A and keep it if its confidence meets min_confidence.

Worked Example

Transactions: {bread, milk}, {bread, diapers, beer, eggs}, {milk, diapers, beer, cola}, {bread, milk, diapers, beer}, {bread, milk, diapers, cola}. With min_support = 0.6 (3 of 5):

  • Frequent single items: bread (4), milk (4), diapers (4), beer (3). Eggs (1) and cola (2) are dropped.

  • Frequent pairs: {bread, milk}, {bread, diapers}, {milk, diapers}, {diapers, beer} (3 each). {bread, beer} and {milk, beer} appear only twice.

  • Triples: {bread, milk, diapers} has all subsets frequent but appears only twice; the other candidates are pruned because they contain {bread, beer} or {milk, beer}. So the search stops.

  • A rule: beer -> diapers has confidence 3/3 = 1.0 and lift 1.0 / 0.8 = 1.25.

Complexity and Limitations

  • Up to 2^m itemsets for m items in the worst case.

  • One full scan of the data per level.

  • Low min_support causes a candidate explosion, especially with many frequent items.

  • FP-Growth avoids candidate generation and is usually much faster.

Interview Tips

  • State the Apriori property clearly; it is the key idea.

  • Explain the support/confidence/lift trade-off and why lift is needed to filter popular-item rules.

  • Pitfall: forgetting that thresholds are business choices, not statistical guarantees.

18

What are support, confidence and lift in association rules?

Support, confidence and lift are the three standard measures for judging association rules of the form A -> B. Support tells you how common an itemset is, confidence how often the rule is correct when A occurs, and lift whether A really raises the chance of B beyond its normal popularity. Using them together matters: high confidence alone can be misleading when B is very common.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Definitions

For N transactions:

  • Support(A) = (number of transactions containing A) / N.

  • Support(A -> B) = support(A and B): the share of transactions containing both.

  • Confidence(A -> B) = support(A and B) / support(A): an estimate of P(B | A).

  • Lift(A -> B) = confidence(A -> B) / support(B) = support(A and B) / (support(A) * support(B)).

Interpreting Lift

Lift Meaning
> 1 A and B occur together more than expected by chance (positive association)
= 1 Independent
< 1 They occur together less than expected (substitutes)

Lift is symmetric: lift(A -> B) = lift(B -> A). Confidence is not.

Worked Example

Five transactions: {bread, milk}, {bread, diapers, beer, eggs}, {milk, diapers, beer, cola}, {bread, milk, diapers, beer}, {bread, milk, diapers, cola}.

  • support(diapers) = 4/5 = 0.8; support(beer) = 3/5 = 0.6; support(diapers and beer) = 3/5 = 0.6.

  • confidence(diapers -> beer) = 0.6 / 0.8 = 0.75.

  • confidence(beer -> diapers) = 0.6 / 0.6 = 1.0.

  • lift = 0.6 / (0.8 * 0.6) = 1.25 for both directions.

Code Example

transactions = [
    {"bread", "milk"},
    {"bread", "diapers", "beer", "eggs"},
    {"milk", "diapers", "beer", "cola"},
    {"bread", "milk", "diapers", "beer"},
    {"bread", "milk", "diapers", "cola"},
]

def support(items):
    return sum(items <= t for t in transactions) / len(transactions)

def rule_metrics(a, b):
    s_ab = support(a | b)
    conf = s_ab / support(a)
    return {"support": s_ab, "confidence": conf, "lift": conf / support(b)}

print(rule_metrics({"diapers"}, {"beer"}))   # support 0.6, confidence 0.75, lift 1.25
print(rule_metrics({"beer"}, {"diapers"}))   # support 0.6, confidence 1.0, lift 1.25

Other Useful Measures

  • Leverage = support(A and B) - support(A) * support(B): the absolute difference from independence.

  • Conviction = (1 - support(B)) / (1 - confidence(A -> B)): how much more often the rule would be wrong if A and B were independent.

Common Pitfalls

  • Choosing rules by confidence only; a rule to a very popular item has high confidence but lift near 1.

  • High lift on tiny support: a coincidence in three baskets; always set a minimum support.

  • Treating lift as causal.

19

How does FP-Growth improve on the Apriori algorithm?

FP-Growth (Frequent Pattern Growth) finds the same frequent itemsets as Apriori without generating candidate itemsets. It compresses the transactions into a prefix tree called an FP-tree in two scans of the data, then mines frequent itemsets recursively from that tree. This makes it much faster than Apriori on large or dense datasets; the trade-off is that the tree must fit in memory and the algorithm is harder to implement and explain.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

How FP-Growth Works

  1. First scan: count every item and drop items below min_support.

  2. Sort the frequent items in each transaction by descending global frequency.

  3. Second scan: insert each sorted transaction into the FP-tree. Transactions with a common prefix share a path, and each node stores a count. A header table links all nodes of the same item.

  4. Mine each item, starting from the least frequent:

  5. Collect its conditional pattern base: the prefix paths that lead to it, with counts.

  6. Build a conditional FP-tree from those paths.

  7. Recursively mine that tree, adding the item as a suffix to every pattern found.

  8. Generate rules from the frequent itemsets exactly as with Apriori.

Comparison With Apriori

Aspect Apriori FP-Growth
Candidate generation Yes, level by level None
Database scans One per itemset length Two
Memory Candidate sets FP-tree (compact when transactions overlap)
Speed on dense data or low support Slow, candidate explosion Much faster
Simplicity Very easy to explain More complex
Output Frequent itemsets Same frequent itemsets

Example of Compression

Transactions {a, b, c}, {a, b, d} and {a, b, c, e} with items sorted by frequency share the path a -> b. The tree stores a:3 -> b:3, then branches to c:2 -> e:1 and d:1, instead of storing every transaction separately.

Practical Use

  • Spark MLlib provides FPGrowth for distributed mining.

  • In Python, mlxtend.frequent_patterns.fpgrowth returns frequent itemsets from a one-hot encoded DataFrame, with the same interface as its apriori function.

  • Eclat is another alternative that uses vertical (item -> transaction ID list) data.

Common Pitfalls

  • Expecting different results from Apriori: both return the same itemsets for the same support threshold; only speed differs.

  • Very sparse data with little overlap gives a large tree, reducing FP-Growth’s advantage.

  • Setting min_support too low still produces an overwhelming number of itemsets, whichever algorithm you use.

20

How is association rule learning used in market-basket analysis?

Market-basket analysis applies association rule learning to shopping transactions to discover which products customers buy together, such as {pasta, tomato sauce} -> {parmesan}. Retailers use these rules for cross-selling, store layout, bundles, promotions and “frequently bought together” recommendations. The main challenge is turning thousands of mined rules into a few actionable ones, using support, confidence and lift plus business judgement.

Read full answer

Applies tomachine learning concepts (examples in pandas 2.2+, Python 3.12)

Workflow

  1. Prepare data: one row per transaction (basket) and the list of items; aggregate SKUs to a sensible level (product or category).

  2. Encode: build a binary basket-by-item matrix.

  3. Mine frequent itemsets with Apriori or FP-Growth and a minimum support.

  4. Generate rules and filter by confidence and lift (e.g. lift > 1.2).

  5. Review with the business: drop trivial rules (bag -> anything) and check that rules make sense.

  6. Act and test: run A/B tests of placements or recommendations, because rules show correlation, not causation.

Business Applications

Use Example
Cross-selling Suggest batteries when a toy is in the cart
Store layout Place chips near beer, or deliberately apart to increase store traffic
Bundles and promotions Discount on the second item of a strong pair
Inventory planning Stock associated items together
Recommendations “Customers who bought X also bought Y”

Code Example

from itertools import combinations
import pandas as pd

baskets = [
    ["pasta", "tomato sauce", "parmesan"], ["pasta", "tomato sauce"], ["bread", "butter"],
    ["pasta", "parmesan"], ["bread", "butter", "milk"], ["pasta", "tomato sauce", "parmesan", "wine"],
    ["milk", "bread"], ["tomato sauce", "pasta", "wine"], ["butter", "milk"], ["pasta", "tomato sauce", "parmesan"],
]
onehot = pd.DataFrame([{item: True for item in b} for b in baskets]).notna()  # True where the basket has the item
support = onehot.mean()                          # support of single items

rows = []
for a, b in combinations(onehot.columns, 2):
    s_ab = (onehot[a] & onehot[b]).mean()
    if s_ab < 0.2:                               # minimum support: 2 of 10 baskets
        continue
    for x, y in ((a, b), (b, a)):
        conf = s_ab / support[x]
        rows.append({"rule": f"{x} -> {y}", "support": s_ab, "confidence": round(conf, 2),
                     "lift": round(conf / support[y], 2)})

rules = pd.DataFrame(rows).sort_values("lift", ascending=False)
print(rules.head(6).to_string(index=False))

For many items, use a library implementation (mlxtend Apriori/FP-Growth, or Spark’s FPGrowth) rather than looping over pairs.

Common Pitfalls

  • Rules driven by very popular items (milk, bags) with lift near 1.

  • Mixing time periods or seasons, which creates misleading associations.

  • Ignoring profitability: a strong rule on low-margin items may not be worth acting on.

21

How does the Expectation-Maximization algorithm work for clustering?

Expectation-Maximization (EM) is an iterative algorithm for fitting models with hidden (latent) variables, such as which cluster generated each point. For clustering with a Gaussian mixture, it alternates between estimating each point’s probability of belonging to each cluster (E-step) and re-estimating the cluster parameters from those probabilities (M-step). It gives soft cluster assignments and flexible elliptical clusters; the trade-off is sensitivity to initialization and convergence to local optima.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

The Steps for a Gaussian Mixture

Model: each point comes from one of K Gaussians with weight pi_k, mean mu_k and covariance Sigma_k.

  1. Initialize the parameters (often from k-means).

  2. E-step: compute responsibilities, the probability that cluster k generated point x_i: r_ik = pi_k * N(x_i | mu_k, Sigma_k) / sum_j pi_j * N(x_i | mu_j, Sigma_j)

  3. M-step: update parameters using the responsibilities as weights, with N_k = sum_i r_ik:

  4. pi_k = N_k / n

  5. mu_k = (1/N_k) * sum_i r_ik * x_i

  6. Sigma_k = (1/N_k) * sum_i r_ik * (x_i - mu_k)(x_i - mu_k)^T

  7. Repeat until the log-likelihood sum_i log(sum_k pi_k * N(x_i | mu_k, Sigma_k)) stops improving.

Each iteration never decreases the log-likelihood, so EM converges, but possibly to a local maximum.

Relation to k-means

k-means is a limiting case of EM: hard assignments instead of probabilities, and equal, spherical covariances. EM generalizes it to soft assignments and elliptical clusters of different sizes.

Code Example

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture

X, _ = make_blobs(n_samples=600, centers=3, cluster_std=[1.0, 2.0, 0.5], random_state=42)

gmm = GaussianMixture(n_components=3, covariance_type="full", n_init=5, random_state=42).fit(X)
print("converged:", gmm.converged_, "after", gmm.n_iter_, "iterations")
print("weights:", gmm.weights_.round(2))
print("means:\n", gmm.means_.round(2))
print("responsibilities of first point:", gmm.predict_proba(X[:1]).round(3))

Other Applications

  • Missing-data imputation, hidden Markov models (Baum-Welch), topic models and mixture models in general.

Common Pitfalls

  • A component can collapse onto a single point (infinite likelihood); scikit-learn adds reg_covar to prevent this.

  • Results depend on initialization; use n_init > 1.

  • The number of components must be chosen, for example with BIC.

22

What are Gaussian Mixture Models and how do they relate to clustering?

A Gaussian Mixture Model (GMM) describes data as coming from a mixture of several Gaussian (normal) distributions, each with its own mean, covariance and weight. Used for clustering, each Gaussian is a cluster, and every point gets a probability of belonging to each one. GMMs are more flexible than k-means because clusters can be elliptical and of different sizes, and they give soft assignments and a density estimate; the trade-off is more parameters, slower fitting and sensitivity to initialization.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

The Model

p(x) = sum_{k=1..K} pi_k * N(x | mu_k, Sigma_k)

  • K: number of components (clusters).

  • pi_k: mixing weight of component k (weights sum to 1).

  • mu_k, Sigma_k: mean vector and covariance matrix of component k.

  • N(x | mu, Sigma): the multivariate normal density.

Parameters are fitted with the Expectation-Maximization (EM) algorithm.

GMM Versus k-means

Aspect k-means GMM
Assignment Hard Soft (probabilities)
Cluster shape Spherical, similar size Elliptical, different sizes and orientations
Output Labels, centroids Labels, probabilities, density p(x)
Model selection Elbow, silhouette BIC / AIC (likelihood-based)
Cost Lower Higher, especially with full covariances

Covariance Types in scikit-learn

  • full: each component has its own full covariance (most flexible).

  • tied: all components share one covariance.

  • diag: diagonal covariances (axis-aligned ellipses).

  • spherical: one variance per component (closest to k-means).

Code Example

import numpy as np
from sklearn.datasets import make_blobs
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=800, centers=4, cluster_std=[0.6, 1.2, 0.8, 1.5], random_state=42)
X = X @ np.array([[0.6, -0.6], [-0.4, 0.8]])   # stretch the blobs into ellipses
X = StandardScaler().fit_transform(X)

bics = {k: GaussianMixture(k, covariance_type="full", n_init=3, random_state=42).fit(X).bic(X)
        for k in range(1, 8)}
best_k = min(bics, key=bics.get)
print("BIC by k:", {k: round(v) for k, v in bics.items()})
print("best k by BIC:", best_k)

gmm = GaussianMixture(best_k, covariance_type="full", n_init=3, random_state=42).fit(X)
print("soft memberships of 3 points:\n", gmm.predict_proba(X[:3]).round(2))
print("log-density of 3 points:", gmm.score_samples(X[:3]).round(2))   # low values flag anomalies

Uses Beyond Clustering

  • Anomaly detection: points with low score_samples are unusual.

  • Density estimation and sampling: gmm.sample(n) generates new data.

  • Speaker and image segmentation, background modelling in video.

Common Pitfalls

  • Too many components or full covariance on small data, which overfits; compare BIC.

  • Forgetting to scale features.

  • Non-Gaussian cluster shapes (moons) still need many components or a different method.

23

How do you choose the optimal number of clusters?

Choosing the number of clusters combines quantitative criteria with judgement: you fit the clustering for a range of k values and compare internal metrics such as the elbow in inertia, the silhouette score, the gap statistic or BIC, then check which result is stable and useful. There is rarely a single “true” k, because the answer depends on the features, the algorithm and the purpose. The trade-off is between finer, more specific clusters and simpler groups that people can act on.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Methods

Method Idea Pick k where
Elbow (inertia) Within-cluster sum of squares falls as k grows The curve bends and further gains are small
Silhouette Cohesion versus separation per point, from -1 to 1 Average silhouette is highest
Davies-Bouldin Average similarity of each cluster to its most similar one Lowest value
Calinski-Harabasz Ratio of between- to within-cluster dispersion Highest value
Gap statistic Compares inertia with that of random reference data Gap is largest (or first within one standard error of the next)
BIC / AIC Likelihood penalized by parameter count (Gaussian mixtures) Lowest BIC
Dendrogram Merge distances in hierarchical clustering A large vertical gap before a merge
Stability Re-cluster bootstrap samples, compare labels (ARI) Assignments stay consistent

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=1000, centers=5, cluster_std=1.1, random_state=42)
X = StandardScaler().fit_transform(X)

print(" k  inertia  silhouette  DB     CH      BIC")
for k in range(2, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    bic = GaussianMixture(k, random_state=42).fit(X).bic(X)
    print(f"{k:2d} {km.inertia_:8.1f} {silhouette_score(X, km.labels_):10.3f} "
          f"{davies_bouldin_score(X, km.labels_):6.3f} {calinski_harabasz_score(X, km.labels_):7.0f} {bic:8.0f}")
# Most criteria should point to k=5 for this synthetic data.

Practical Advice

  • Use several criteria: if the elbow, silhouette and BIC agree, you can be more confident.

  • Check stability: rerun with different seeds and bootstrap samples.

  • Profile the clusters: can the business describe and act on each one?

  • Algorithms that choose k themselves: DBSCAN and HDBSCAN find the number of clusters from density.

Common Pitfalls

  • Reading inertia alone: it always decreases as k grows, so the minimum is meaningless.

  • Unscaled features, which change every metric.

  • Treating the metric-optimal k as final when the business needs a different level of detail.

24

What are cluster validity indices?

Cluster validity indices are numbers that score how good a clustering is, so you can compare algorithms, parameters or numbers of clusters. Internal indices use only the data and the cluster labels (cohesion and separation), external indices compare the clusters with known labels, and relative use means comparing several clusterings with the same index. They make clustering evaluation objective, but each index encodes assumptions about what a “good” cluster looks like, so none is universally right.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Internal Indices (No Labels Needed)

Index Measures Better Note
Silhouette (b - a) / max(a, b) per point: separation minus cohesion Higher (max 1) O(n^2); favours convex clusters
Davies-Bouldin Average of each cluster’s worst ratio (spread_i + spread_j) / distance between centroids Lower (min 0) Centroid-based
Calinski-Harabasz Between-cluster dispersion / within-cluster dispersion, adjusted for k and n Higher Fast; favours convex clusters
Dunn index Smallest inter-cluster distance / largest cluster diameter Higher Sensitive to outliers; not in scikit-learn
Inertia (WCSS) Sum of squared distances to centroids Lower, but always falls with k Use with the elbow method

External Indices (Ground Truth Needed)

Index Range Notes
Adjusted Rand Index (ARI) About -0.5 to 1; 0 = random Pair-counting, chance-adjusted
Normalized / Adjusted Mutual Information 0 to 1 Information-theoretic; AMI is chance-adjusted
Homogeneity, completeness, V-measure 0 to 1 Each cluster has one class / each class in one cluster
Fowlkes-Mallows 0 to 1 Geometric mean of pairwise precision and recall

External indices are used for benchmarking algorithms on labelled data, not in real unsupervised projects where labels are absent.

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.metrics import (adjusted_rand_score, calinski_harabasz_score,
                             davies_bouldin_score, normalized_mutual_info_score, silhouette_score)
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)

print("internal  silhouette:", round(silhouette_score(X, labels), 3))
print("internal  Davies-Bouldin:", round(davies_bouldin_score(X, labels), 3))
print("internal  Calinski-Harabasz:", round(calinski_harabasz_score(X, labels), 1))
print("external  ARI:", round(adjusted_rand_score(y, labels), 3))
print("external  NMI:", round(normalized_mutual_info_score(y, labels), 3))

Common Pitfalls

  • Using silhouette or Davies-Bouldin to judge DBSCAN on non-convex shapes; they penalize correct but curved clusters.

  • Comparing index values computed on differently scaled or reduced data.

  • Optimizing one index blindly; combine indices with stability checks and domain review.

25

What challenges arise when clustering high-dimensional data?

Clustering high-dimensional data is hard because distance-based similarity breaks down: as the number of features grows, distances between all points become similar, data becomes sparse, and irrelevant features add noise that hides real structure. Clusters may also exist only in subsets of features. The usual answer is to reduce or select dimensions, choose a suitable distance and validate results carefully, accepting some loss of information and interpretability.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Main Challenges

  • Distance concentration: the nearest and farthest neighbours are almost equally far away, so k-means, DBSCAN and hierarchical clustering lose their signal.

  • Irrelevant and noisy features: each one adds random variation to every distance.

  • Subspace clusters: groups may differ on only a few features (e.g. genes active in one tissue), invisible in the full space.

  • Sparsity: density-based methods find almost no dense regions.

  • Cost: distance computations grow with the number of features, and O(n^2) methods become impractical.

  • Interpretation and visualization: it is hard to describe or plot clusters in 1,000 dimensions.

  • Correlated features: groups of correlated features count the same information several times.

Remedies

Remedy How it helps
PCA / truncated SVD Keep the main variance directions, discard noise
UMAP (then cluster) Non-linear reduction that keeps neighbourhoods; common for embeddings
Feature selection Remove low-variance, redundant or irrelevant features
Cosine distance More meaningful for text and embeddings than Euclidean
Subspace / biclustering Find clusters in feature subsets
Spectral clustering Clusters a similarity graph rather than raw distances
Model-based with constraints Gaussian mixtures with diagonal covariance on reduced data

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.metrics import adjusted_rand_score
from sklearn.preprocessing import StandardScaler

# 4 informative features hidden among 496 noise features
X, y = make_classification(n_samples=1500, n_features=500, n_informative=4, n_redundant=0,
                           n_clusters_per_class=1, n_classes=3, class_sep=2.0, random_state=42)
X = StandardScaler().fit_transform(X)

raw = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)
X_pca = PCA(n_components=10, random_state=42).fit_transform(X)
reduced = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X_pca)

print("ARI on all 500 features:", round(adjusted_rand_score(y, raw), 3))
print("ARI after PCA to 10:", round(adjusted_rand_score(y, reduced), 3))
# Results vary with the data; removing noise dimensions usually helps here.

Common Pitfalls

  • Clustering raw one-hot or TF-IDF matrices with Euclidean k-means without normalization.

  • Reducing with t-SNE and then clustering, which can create artificial clusters; prefer UMAP or PCA for this.

  • Assuming PCA always keeps the cluster structure; high-variance directions may not be the separating ones.

26

Which preprocessing steps should you do before unsupervised learning?

Preprocessing for unsupervised learning prepares data so that distances, variances and densities reflect real structure rather than units, noise or errors. The usual steps are cleaning, handling missing values, encoding categories, transforming skewed features, scaling, removing redundant features and optionally reducing dimensions. Because there is no label to catch mistakes, preprocessing choices strongly shape the result, so each one should be deliberate and documented.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Recommended Steps

  1. Understand the data: types, ranges, distributions, duplicates and the business meaning of each feature.

  2. Clean: remove duplicates, fix data entry errors, standardize formats.

  3. Handle missing values: drop, impute, or add missing indicators (see what the missingness means).

  4. Treat outliers: decide whether they are errors (remove or cap) or interesting (keep; they may be the anomalies you want).

  5. Encode categorical features: one-hot encoding for nominal data, or use distance measures and algorithms made for mixed data (Gower distance, k-prototypes).

  6. Transform skewed features: log or Yeo-Johnson transforms for amounts and counts.

  7. Scale: StandardScaler or RobustScaler so every feature contributes fairly to distances.

  8. Remove redundant or irrelevant features: near-constant columns, IDs, highly correlated duplicates.

  9. Reduce dimensions (optional): PCA or UMAP for high-dimensional data.

  10. Build a pipeline so the same steps apply to new data.

Which Methods Need What

Method Scaling needed? Notes
k-means, hierarchical, DBSCAN, k-NN-based Yes Distance-based
PCA Yes (standardize) Variance-based
Gaussian mixtures Usually yes Helps numerical stability
Isolation Forest No Tree-based
Apriori / FP-Growth No Needs binary basket data instead

Code Example

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
df = pd.DataFrame({
    "annual_spend": rng.lognormal(6, 1, 500),          # skewed
    "visits": rng.poisson(10, 500).astype(float),
    "channel": rng.choice(["web", "app", "store"], 500),
})
df.loc[rng.random(500) < 0.05, "visits"] = np.nan

numeric = make_pipeline(SimpleImputer(strategy="median"),
                        FunctionTransformer(np.log1p), StandardScaler())
categorical = OneHotEncoder(handle_unknown="ignore")
prep = ColumnTransformer([("num", numeric, ["annual_spend", "visits"]),
                          ("cat", categorical, ["channel"])])

pipe = Pipeline([("prep", prep), ("cluster", KMeans(n_clusters=4, n_init=10, random_state=42))])
df["segment"] = pipe.fit_predict(df)
print(df.groupby("segment")[["annual_spend", "visits"]].median().round(1))

Common Pitfalls

  • Letting one-hot columns dominate distances; consider weighting them or using a mixed-data method.

  • Keeping ID or timestamp columns, which create meaningless clusters.

  • Different preprocessing at scoring time than at training time; save the whole pipeline.

27

How do you handle missing values in unsupervised learning?

Handling missing values in unsupervised learning is harder than in supervised learning because there is no target to check whether an imputation helps. The main options are removing incomplete rows or columns, imputing values (mean, median, k-NN or iterative imputation), or using methods that tolerate missing data. The trade-off is that imputation can create artificial similarities between points and distort clusters, while removal throws information away.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Options

Approach When it fits Risk
Drop rows Few rows affected, missing at random Bias if missingness is related to the data
Drop columns A feature is mostly empty Losing a useful feature
Mean / median imputation Few missing values, quick baseline Pulls points toward the centre, shrinks variance, creates fake clusters at the mean
Most-frequent / “missing” category Categorical features Adds an artificial category
k-NN imputation (KNNImputer) Moderate missingness, correlated features Slow on large data; distances need scaling
Iterative imputation (IterativeImputer) Features predict each other Slower; experimental API in scikit-learn
Missing indicators Missingness itself is informative More dimensions
Algorithms that handle NaN E.g. EM-based mixture models with missing data, matrix factorization Fewer ready-made tools

Understand the Missingness First

  • MCAR (missing completely at random): dropping or simple imputation is unbiased.

  • MAR (depends on observed features): model-based imputation works well.

  • MNAR (depends on the missing value itself, e.g. high earners skip income): no imputation is fully safe; add indicators and be cautious.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.metrics import adjusted_rand_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_blobs(n_samples=600, n_features=5, centers=3, random_state=42)
rng = np.random.default_rng(42)
X_missing = X.copy()
X_missing[rng.random(X.shape) < 0.2] = np.nan        # 20% missing

for name, imputer in [("mean", SimpleImputer(strategy="mean")), ("knn", KNNImputer(n_neighbors=5))]:
    pipe = make_pipeline(StandardScaler(), imputer, KMeans(n_clusters=3, n_init=10, random_state=42))
    labels = pipe.fit_predict(X_missing)
    print(f"{name} imputation, ARI vs true groups: {adjusted_rand_score(y, labels):.3f}")

StandardScaler ignores NaN values when computing means and variances, so scaling before k-NN imputation is valid and keeps the distances fair.

Common Pitfalls

  • Mean-imputing many values, which creates a dense artificial cluster.

  • Imputing before scaling for distance-based imputers, so large-unit features dominate.

  • Not checking whether clusters are just “rows with many missing values”.

28

How do you scale and normalize data for clustering?

Scaling and normalizing for clustering puts features on comparable ranges so that each contributes fairly to distances. Without it, a feature measured in thousands (income) completely outweighs one measured in units (number of children) in k-means, hierarchical clustering or DBSCAN. The choice of scaler matters too: standardization, min-max scaling, robust scaling and row normalization each suit different data, and the wrong one can distort clusters.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Steps

  1. Inspect distributions: look for skew, outliers and different units.

  2. Transform skewed features first: log1p for counts and amounts, or PowerTransformer (Yeo-Johnson).

  3. Choose a scaler for the remaining features (table below).

  4. Consider feature weights: after scaling, every feature counts equally; multiply important features by a weight if domain knowledge says they should count more.

  5. Normalize rows if the direction of a vector matters more than its size (text, embeddings).

  6. Fit the scaler once, inside a pipeline, and reuse it for new data.

Choosing a Scaler

Scaler Formula Use when
StandardScaler (x - mean) / std Default for k-means, PCA, Gaussian mixtures
MinMaxScaler (x - min) / (max - min) Bounded features, no extreme outliers
RobustScaler (x - median) / IQR Outliers present
PowerTransformer Yeo-Johnson / Box-Cox, then standardize Strongly skewed features
Normalizer Each row divided by its L2 norm Text vectors, embeddings, cosine-like clustering

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler, StandardScaler

rng = np.random.default_rng(42)
# Two groups that differ only in "children"; "income" is large-scale noise
children = np.concatenate([rng.normal(0.5, 0.3, 300), rng.normal(3, 0.3, 300)])
income = rng.normal(60_000, 15_000, 600)
X = np.column_stack([income, children])
y = np.repeat([0, 1], 300)

raw = KMeans(n_clusters=2, n_init=10, random_state=42).fit_predict(X)
print("unscaled ARI:", round(adjusted_rand_score(y, raw), 3))          # near 0: income dominates

for scaler in (StandardScaler(), RobustScaler()):
    labels = make_pipeline(scaler, KMeans(n_clusters=2, n_init=10, random_state=42)).fit_predict(X)
    print(type(scaler).__name__, "ARI:", round(adjusted_rand_score(y, labels), 3))   # near 1

Common Pitfalls

  • Scaling one-hot columns as if they were continuous; decide their weight deliberately.

  • Min-max scaling with outliers: one extreme value squeezes everyone else into a tiny range.

  • Scaling with statistics from a different period than the data being clustered without noticing drift.

  • Not scaling at all because “the features are already numeric”.

29

How do you evaluate the performance of a clustering algorithm?

Evaluating a clustering algorithm is harder than evaluating a classifier because there are usually no true labels. It combines internal metrics (how compact and separated the clusters are), stability checks (do the clusters persist across samples and seeds), external metrics when some labels exist, and usefulness checks (do the clusters help a business decision or a downstream model). No single number is enough, so good evaluation uses several of these together.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Four Ways to Evaluate

Type Examples Answers
Internal Silhouette, Davies-Bouldin, Calinski-Harabasz, inertia, BIC Are clusters compact and well separated?
External ARI, NMI, V-measure (needs labels) Do clusters match known categories?
Stability Bootstrap or seed variation, compared with ARI Are the clusters reproducible?
Usefulness Downstream model gain, business KPIs, expert review Do the clusters matter?

Stability Check

  1. Draw several bootstrap samples (or change the seed).

  2. Re-cluster each one.

  3. Compare the labels on shared points with the Adjusted Rand Index.

  4. High average ARI means the structure is real, not an artefact of randomness.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score, davies_bouldin_score, silhouette_score
from sklearn.preprocessing import StandardScaler

X, _ = make_blobs(n_samples=800, centers=4, cluster_std=1.3, random_state=42)
X = StandardScaler().fit_transform(X)
rng = np.random.default_rng(42)

def stability(X, k, n_boot=20):
    base = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X)
    scores = []
    for b in range(n_boot):
        idx = rng.choice(len(X), size=len(X), replace=True)
        boot = KMeans(n_clusters=k, n_init=10, random_state=b).fit(X[idx])
        scores.append(adjusted_rand_score(base.predict(X), boot.predict(X)))   # compare on all points
    return np.mean(scores)

for k in (3, 4, 5, 6):
    labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X)
    print(f"k={k}: silhouette={silhouette_score(X, labels):.3f}, "
          f"DB={davies_bouldin_score(X, labels):.3f}, stability ARI={stability(X, k):.3f}")

Usefulness Checks

  • Profiles: average feature values per cluster should tell a clear story.

  • Business outcomes: clusters should differ on revenue, churn or response rates.

  • Downstream models: adding the cluster label as a feature should help a supervised model, measured with cross-validation.

  • Expert review: domain experts should recognize the groups.

Common Pitfalls

  • Optimizing silhouette alone, which favours a few large spherical clusters.

  • Using external labels that are not what the clustering is meant to find.

  • Evaluating on the same sample used to choose parameters without any stability check.

30

Why is feature selection important in unsupervised learning?

Feature selection in unsupervised learning chooses which features the algorithm sees, and it largely decides what structure is found: clustering on spending features finds spending segments, while adding demographic features finds different groups. Removing irrelevant, noisy or redundant features makes clusters clearer, more stable and easier to explain, and it reduces cost. The difficulty is that there is no target to measure relevance against, so selection relies on statistics, stability and domain knowledge.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Why It Matters

  • Noise drowns structure: every irrelevant feature adds random variation to distances.

  • Redundancy double-counts: ten correlated “spend” features give spending ten times the weight.

  • Curse of dimensionality: fewer features make distances more meaningful.

  • Interpretability: clusters defined by five features are easier to describe than by 200.

  • Cost: fewer features mean faster clustering and simpler pipelines.

Methods Without Labels

Method Idea
Variance threshold Remove near-constant features
Correlation filtering Drop one of each highly correlated pair
Laplacian score Prefer features that preserve the local neighbourhood structure
Wrapper search Try feature subsets, keep the one with the best silhouette or stability
Domain-driven selection Choose features that match the business question
Feature extraction PCA or autoencoders combine features instead of selecting them
Post-hoc importance Train a classifier to predict cluster labels; its important features explain the clusters

Code Example

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.feature_selection import VarianceThreshold
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
X_signal, _ = make_blobs(n_samples=500, n_features=3, centers=4, random_state=42)
X = pd.DataFrame(np.hstack([X_signal, rng.normal(size=(500, 7)), np.full((500, 1), 1.0)]),
                 columns=[f"s{i}" for i in range(3)] + [f"noise{i}" for i in range(7)] + ["constant"])
X["s0_copy"] = X["s0"] * 2 + rng.normal(0, 0.01, 500)          # redundant feature

X_var = X.loc[:, VarianceThreshold(1e-6).fit(X).get_support()]  # drop the constant column
corr = X_var.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
X_sel = X_var.drop(columns=[c for c in upper.columns if (upper[c] > 0.95).any()])

for name, data in [("all features", X_var), ("signal only", X_sel[["s0", "s1", "s2"]])]:
    Z = StandardScaler().fit_transform(data)
    labels = KMeans(n_clusters=4, n_init=10, random_state=42).fit_predict(Z)
    print(f"{name}: silhouette={silhouette_score(Z, labels):.3f}")

Common Pitfalls

  • Choosing features by the silhouette score alone, which can favour trivial splits.

  • Removing a feature because it has low variance after scaling (scaling sets all variances to 1; check before scaling).

  • Forgetting that the selected features define the meaning of the clusters; state them clearly in reports.

31

How do you cluster a large distributed dataset?

Clustering a large distributed dataset means running the clustering where the data lives, across many machines, using algorithms that need only a few passes and exchange small summaries instead of raw rows. The common choices are distributed k-means (Spark MLlib, Dask-ML), mini-batch k-means on samples or streams, and two-stage approaches that summarize data first and then cluster the summaries. The trade-off is that exact methods with O(n^2) cost, like hierarchical clustering or plain DBSCAN, do not scale, so you accept approximations.

Read full answer

Applies toSpark 3.5 (PySpark), scikit-learn 1.5+ (Python 3.12)

Strategies

Strategy How it works Good for
Distributed k-means Each worker assigns its points to centroids and sends partial sums; the driver updates centroids Very large numeric data

| k-means|| initialization | Parallel version of k-means++ in Spark | Fast, good starting centroids | | Mini-batch k-means | Updates centroids from small random batches | One machine, data streamed from disk | | Sample, then assign | Cluster a representative sample, then assign all points to the nearest centroid in parallel | Expensive algorithms (HDBSCAN, GMM) | | Two-stage (BIRCH-style) | Compress data into many micro-clusters, then cluster those | Streaming or memory-limited settings | | Bisecting k-means | Divisive splits, distributed in Spark | Hierarchical-like results at scale |

Code Example

from pyspark.sql import SparkSession
from pyspark.ml import Pipeline
from pyspark.ml.clustering import KMeans
from pyspark.ml.evaluation import ClusteringEvaluator
from pyspark.ml.feature import StandardScaler, VectorAssembler

spark = SparkSession.builder.appName("distributed-kmeans").getOrCreate()

# In production: spark.read.parquet("s3://bucket/customers/") ; here a synthetic DataFrame
df = spark.range(0, 1_000_000).selectExpr(
    "id", "rand(42) * 100 AS spend", "randn(7) * 5 + 20 AS visits", "rand(3) * 10 AS tenure")

pipeline = Pipeline(stages=[
    VectorAssembler(inputCols=["spend", "visits", "tenure"], outputCol="raw"),
    StandardScaler(inputCol="raw", outputCol="features", withMean=True, withStd=True),
    KMeans(k=5, seed=42, initMode="k-means||", maxIter=30, featuresCol="features"),
])
model = pipeline.fit(df)
clustered = model.transform(df)

silhouette = ClusteringEvaluator(featuresCol="features").evaluate(clustered.sample(0.05, seed=42))
print("silhouette on a 5% sample:", round(silhouette, 3))
clustered.groupBy("prediction").count().show()

Practical Tips

  • Scale features in the distributed pipeline so every node uses the same statistics.

  • Cache the feature DataFrame: k-means passes over the data many times.

  • Evaluate on samples: silhouette is O(n^2) in general (Spark’s evaluator uses a cheaper approximation, but sampling still saves time).

  • Reduce dimensions first (PCA in Spark MLlib) for wide data.

  • Save the fitted pipeline to assign new data consistently.

Common Pitfalls

  • Collecting all data to the driver (toPandas()) to run scikit-learn.

  • Skewed partitions that leave one worker doing most of the work.

  • Running DBSCAN-style algorithms naively at scale; they need spatial partitioning or sampling.

32

Where can unsupervised learning add value to a business process?

Unsupervised learning adds business value wherever a company has lots of data but no labels for the question it cares about. A typical example is customer support: thousands of tickets arrive daily, nobody has tagged them by root cause, and clustering their text reveals the main problem themes, their volume and new emerging issues. The value comes from faster insight and automation without labelling costs; the trade-off is that the discovered groups need human review before they drive decisions.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Scenario: Support Ticket Triage

  1. Data: ticket text, product area, customer tier, timestamps.

  2. Represent each ticket as a vector (TF-IDF or sentence embeddings).

  3. Cluster the vectors (k-means or HDBSCAN) to group similar tickets.

  4. Label clusters by reading top terms and sample tickets (“password reset”, “billing double charge”, “app crash on login”).

  5. Act:

  6. Route clusters to the right team automatically.

  7. Write help-centre articles for the largest clusters.

  8. Alert engineering when a new cluster grows quickly (emerging bug).

  9. Measure: resolution time, deflection rate and repeat contacts before and after.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer

tickets = [
    "cannot reset my password", "password reset email never arrives", "forgot password link broken",
    "charged twice this month", "double charge on my card", "refund for duplicate payment",
    "app crashes when I log in", "login screen crashes on android", "app closes after login",
]
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(tickets)                   # rows are L2-normalized by default

km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
terms = np.array(vec.get_feature_names_out())
for c in range(3):
    top = terms[np.argsort(km.cluster_centers_[c])[::-1][:3]]
    print(f"cluster {c}: {', '.join(top)} -> {np.sum(km.labels_ == c)} tickets")

Other High-Value Scenarios

Area Unsupervised use
Marketing Customer segmentation for targeted campaigns
Finance Anomaly detection on transactions for fraud review
Manufacturing Detecting abnormal sensor patterns before failures
Retail Market-basket rules for bundles and store layout
Operations Grouping similar suppliers, routes or incidents
Data quality Finding duplicate or inconsistent records

Common Pitfalls

  • Starting without a decision the clusters will inform.

  • Presenting clusters as facts without validation by domain experts.

  • Not monitoring: clusters drift as products and customers change, so re-cluster periodically.

33

How is unsupervised learning used for image segmentation?

Unsupervised image segmentation divides an image into regions of similar pixels without any labelled masks, by clustering pixels on colour, position or learned features. Classic methods include k-means on colours, mean shift, graph-based methods, superpixels (SLIC) and watershed; modern approaches cluster features from self-supervised vision models. It is cheap and needs no annotation, but the regions it finds are based on appearance, so they may not match meaningful objects.

Read full answer

Applies toscikit-learn 1.5+ and scikit-image 0.24+ (Python 3.12)

Methods

Method Groups pixels by Notes
k-means on colours Colour (RGB, Lab) Simple colour quantization; ignores spatial layout unless coordinates are added
Gaussian mixture Colour distribution Soft assignments
Mean shift Density peaks in colour + position No fixed k; slower
SLIC superpixels Colour + position, local k-means Compact regions used as building blocks
Felzenszwalb (graph-based) Edge weights between neighbouring pixels Adapts to region detail
Watershed Image gradient “basins” Separates touching objects; needs markers
Self-supervised features + clustering Deep features (e.g. from DINO-style models) Regions closer to semantic objects

Code Example

import numpy as np
from skimage import data
from skimage.color import rgb2lab
from skimage.segmentation import slic
from sklearn.cluster import KMeans

image = data.astronaut()                          # (512, 512, 3) RGB
lab = rgb2lab(image)                              # perceptually more uniform colour space

# 1) k-means on colour + scaled position
h, w, _ = lab.shape
yy, xx = np.mgrid[0:h, 0:w]
features = np.column_stack([lab.reshape(-1, 3), 0.2 * yy.ravel(), 0.2 * xx.ravel()])
labels = KMeans(n_clusters=6, n_init=4, random_state=42).fit_predict(features)
segmentation = labels.reshape(h, w)
print("k-means segments:", np.unique(segmentation).size)

# 2) SLIC superpixels: many small, compact regions
superpixels = slic(image, n_segments=200, compactness=10, start_label=0)
print("superpixels:", superpixels.max() + 1)

The position weight (0.2 here) controls how much spatial closeness matters relative to colour.

Use Cases

  • Pre-segmenting images to speed up manual annotation.

  • Medical imaging: separating tissue types by intensity.

  • Satellite imagery: land-cover regions.

  • Background removal and colour quantization.

Common Pitfalls

  • Clustering only colour, so regions with the same colour far apart get one label.

  • Expecting object-level segments: shadows and textures split objects.

  • High resolution makes pixel-level k-means slow; work on superpixels or downsampled images.

34

How do recommendation systems use unsupervised learning?

Recommendation systems use unsupervised learning to find structure in user behaviour without labelled “good recommendation” examples: matrix factorization discovers hidden taste factors, clustering groups similar users or items, nearest-neighbour search finds similar items, and association rules find items bought together. These methods are the backbone of collaborative filtering and candidate generation. The trade-off is that they struggle with new users and items (cold start) and reflect past behaviour, including its biases.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Techniques

Technique What it learns Recommendation use
Matrix factorization (SVD, ALS, NMF) Latent factors for users and items from the interaction matrix Predict missing ratings; embeddings for retrieval
Clustering users or items Groups with similar behaviour Recommend what is popular in a user’s cluster; cold-start fallbacks
k-nearest neighbours Similar users or items by cosine similarity “Users like you liked…” / “Similar items”
Association rules Items that co-occur in baskets “Frequently bought together”
Embeddings (word2vec on sessions, autoencoders) Dense item vectors from sequences Session-based recommendations
Topic models Themes in item descriptions Content-based similarity

How Matrix Factorization Works

The user-item matrix R (users x items, mostly empty) is approximated as R ≈ U V^T:

  • U holds a k-dimensional vector for each user and V one for each item.

  • The predicted score for user u and item i is the dot product u_u · v_i.

  • Factors are learned by minimizing the error on observed entries plus regularization; they often align with real tastes (genre, price level) without being told.

Code Example

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.metrics.pairwise import cosine_similarity
from scipy.sparse import csr_matrix

rng = np.random.default_rng(42)
n_users, n_items = 300, 50
taste = rng.integers(0, 3, n_users)                    # hidden user groups
item_genre = rng.integers(0, 3, n_items)
prob = np.where(taste[:, None] == item_genre[None, :], 0.4, 0.03)
interactions = csr_matrix((rng.random((n_users, n_items)) < prob).astype(np.float32))

svd = TruncatedSVD(n_components=5, random_state=42)
user_vecs = svd.fit_transform(interactions)            # users in latent space
item_vecs = svd.components_.T                           # items in latent space

user = 0
scores = user_vecs[user] @ item_vecs.T
scores[interactions[user].indices] = -np.inf            # do not recommend what they already have
print("recommended items:", np.argsort(-scores)[:5], "| user's taste group:", taste[user])
print("item genres of recommendations:", item_genre[np.argsort(-scores)[:5]])

similar = cosine_similarity(item_vecs[[0]], item_vecs)[0].argsort()[::-1][1:4]
print("items similar to item 0:", similar)

Common Pitfalls

  • Treating missing interactions as dislikes; usually they are unknown.

  • Popularity bias: factorization favours already popular items.

  • Cold start: new items have no interactions, so combine with content features.

  • Evaluating only offline; confirm with online A/B tests.

35

How is unsupervised learning applied to anomaly detection?

Unsupervised anomaly detection learns what “normal” data looks like and flags points that do not fit, without needing labelled examples of anomalies. Methods measure isolation (Isolation Forest), local density (Local Outlier Factor), distance to clusters, probability under a density model, or reconstruction error from an autoencoder. This is valuable because real anomalies such as fraud or faults are rare and often new; the trade-off is more false positives and the need to choose a threshold without labels.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Methods

Method Anomaly score Strengths Weaknesses
Isolation Forest Short average path length in random trees Fast, scales well, no scaling needed Weaker on local anomalies in dense regions
Local Outlier Factor Density much lower than neighbours’ density Finds local anomalies O(n^2)-like cost; needs scaling
One-Class SVM Outside a learned boundary Flexible with kernels Slow on large data; sensitive to parameters
Gaussian mixture / KDE Low probability density Probabilistic scores Struggles in high dimensions
Clustering distance (k-means, DBSCAN noise) Far from centroids or labelled noise Simple, interpretable Depends on clustering quality
Autoencoder High reconstruction error Images, sequences, complex data Needs tuning and enough normal data

Workflow

  1. Train on data that is mostly normal (clean it if you can).

  2. Compute an anomaly score for each point.

  3. Choose a threshold: an expected contamination rate, a review budget (top 100 per day) or a percentile.

  4. Have experts review flagged cases; their feedback provides labels for later supervised or semi-supervised models.

Code Example

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
normal = rng.normal(0, 1, size=(1000, 4))
anomalies = rng.uniform(-6, 6, size=(15, 4))
X = StandardScaler().fit_transform(np.vstack([normal, anomalies]))
is_anomaly = np.r_[np.zeros(1000), np.ones(15)].astype(bool)

iso = IsolationForest(n_estimators=300, contamination=0.015, random_state=42).fit(X)
iso_flags = iso.predict(X) == -1                      # -1 means anomaly

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.015)
lof_flags = lof.fit_predict(X) == -1

for name, flags in [("IsolationForest", iso_flags), ("LOF", lof_flags)]:
    caught = (flags & is_anomaly).sum()
    print(f"{name}: flagged {flags.sum()}, true anomalies caught {caught}/15")

Common Pitfalls

  • Training on data full of anomalies, which then look normal.

  • Setting contamination arbitrarily; it directly sets how many points are flagged.

  • Ignoring seasonality and context (a large transaction is normal on payday).

  • Evaluating without any labels at all; even a small labelled sample helps estimate precision.

36

What are Generative Adversarial Networks and how do they work?

A Generative Adversarial Network (GAN) is a pair of neural networks trained against each other: a generator that turns random noise into synthetic samples, and a discriminator that tries to tell real samples from generated ones. As the discriminator gets better at spotting fakes, the generator learns to produce more realistic data. GANs can generate sharp, realistic images without labels, but training is unstable and can suffer from mode collapse, where the generator produces only a few kinds of output.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

How Training Works

The two networks play a minimax game:

min_G max_D E_x[log D(x)] + E_z[log(1 - D(G(z)))]

  • x: a real sample; z: a random noise vector; G(z): a generated sample.

  • D(x): the discriminator’s probability that x is real.

  • The discriminator maximizes the objective (classify correctly); the generator minimizes it (fool the discriminator).

Each training step: 1. Discriminator step: train on a batch of real samples (label 1) and generated samples (label 0). 2. Generator step: generate a batch and update G so that D labels it real. In practice the generator maximizes log D(G(z)) (the “non-saturating” loss), which gives stronger gradients early on.

Code Example

import torch
from torch import nn

torch.manual_seed(42)
real_data = torch.randn(2000, 1) * 0.5 + 3.0          # target: N(3, 0.5^2)

G = nn.Sequential(nn.Linear(8, 32), nn.ReLU(), nn.Linear(32, 1))
D = nn.Sequential(nn.Linear(1, 32), nn.LeakyReLU(0.2), nn.Linear(32, 1))   # outputs logits
opt_g = torch.optim.Adam(G.parameters(), lr=1e-3)
opt_d = torch.optim.Adam(D.parameters(), lr=1e-3)
bce = nn.BCEWithLogitsLoss()

for step in range(3000):
    real = real_data[torch.randint(0, 2000, (64,))]
    fake = G(torch.randn(64, 8))

    # Discriminator: real -> 1, fake -> 0 (detach so G is not updated here)
    d_loss = bce(D(real), torch.ones(64, 1)) + bce(D(fake.detach()), torch.zeros(64, 1))
    opt_d.zero_grad(); d_loss.backward(); opt_d.step()

    # Generator: make D output 1 for fakes
    g_loss = bce(D(fake), torch.ones(64, 1))
    opt_g.zero_grad(); g_loss.backward(); opt_g.step()

with torch.no_grad():
    samples = G(torch.randn(5000, 8))
print("generated mean/std:", samples.mean().item(), samples.std().item())  # should approach 3 and 0.5

Variants

Variant Idea
DCGAN Convolutional generator and discriminator for images
Conditional GAN Generate samples of a requested class
WGAN / WGAN-GP Wasserstein loss with gradient penalty for more stable training
CycleGAN Image-to-image translation without paired data
StyleGAN High-quality, controllable face and image synthesis

Common Pitfalls

  • Mode collapse: limited variety; mitigations include WGAN-GP and minibatch discrimination.

  • Imbalance: a discriminator that wins too easily gives the generator no useful gradient.

  • Evaluation: losses do not show quality; use metrics such as FID plus visual inspection.

  • Diffusion models now often beat GANs on image quality, but GANs remain much faster at sampling.

37

What is a Variational Autoencoder (VAE)?

A Variational Autoencoder (VAE) is a generative model built like an autoencoder, but its encoder outputs a probability distribution over a latent space (a mean and a variance) instead of a single point. Training balances reconstructing the input with keeping that distribution close to a standard normal, which makes the latent space smooth, so you can sample new points and decode them into new data. VAEs are stable to train and give meaningful latent spaces; their main trade-off is that generated images tend to be blurrier than those from GANs or diffusion models.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

How It Works

  1. Encoder q(z | x): maps input x to a mean mu and log-variance log sigma^2 for each latent dimension.

  2. Reparameterization trick: sample z = mu + sigma * eps with eps ~ N(0, I). Randomness is moved into eps, so gradients can flow through mu and sigma.

  3. Decoder p(x | z): reconstructs x from z.

  4. Loss (negative ELBO): L = reconstruction_loss(x, x_hat) + KL(q(z | x) || N(0, I))

  5. The reconstruction term (MSE or binary cross-entropy) makes outputs match inputs.

  6. The KL term, -0.5 * sum(1 + log sigma^2 - mu^2 - sigma^2), pulls each encoded distribution toward the standard normal, keeping the latent space continuous.

VAE Versus Autoencoder Versus GAN

Aspect Autoencoder VAE GAN
Latent code A point A distribution Random noise input
Can generate new data Not reliably Yes, sample z ~ N(0, I) Yes
Training stability Stable Stable Often unstable
Sample sharpness n/a Often blurry Sharp
Has an encoder Yes Yes No (standard GAN)

Code Example

import torch
from torch import nn
import torch.nn.functional as F

class VAE(nn.Module):
    def __init__(self, d=784, h=256, k=16):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU())
        self.mu = nn.Linear(h, k)
        self.logvar = nn.Linear(h, k)
        self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d), nn.Sigmoid())

    def forward(self, x):
        hidden = self.enc(x)
        mu, logvar = self.mu(hidden), self.logvar(hidden)
        z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu)   # reparameterization trick
        return self.dec(z), mu, logvar

def vae_loss(x, x_hat, mu, logvar):
    recon = F.binary_cross_entropy(x_hat, x, reduction="sum")
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (recon + kl) / x.size(0)

torch.manual_seed(42)
model = VAE()
x = torch.rand(32, 784)                      # stand-in for a batch of MNIST images in [0, 1]
x_hat, mu, logvar = model(x)
print("loss:", vae_loss(x, x_hat, mu, logvar).item())

with torch.no_grad():
    new_images = model.dec(torch.randn(8, 16))   # generate 8 new samples
print(new_images.shape)                      # torch.Size([8, 784])

Common Pitfalls

  • Posterior collapse: the decoder ignores z when the KL term dominates; use KL warm-up or a beta schedule.

  • Beta-VAE: weighting the KL term more gives more disentangled factors at the cost of reconstruction quality.

  • Using MSE on binary data or BCE on unbounded data; match the loss to the data.

38

How does unsupervised learning contribute to NLP?

Unsupervised learning is central to modern NLP: most language knowledge is learned from huge amounts of raw, unlabelled text. Word embeddings, topic models, text clustering and, above all, self-supervised language-model pre-training (predicting the next or masked word) all learn from text without human labels. This lets small labelled datasets go much further; the trade-off is heavy compute and models that absorb the biases and errors present in their training text.

Read full answer

Applies toNLP concepts (examples in scikit-learn 1.5+, Python 3.12)

Contributions

Technique What it learns Uses
Word embeddings (word2vec, GloVe, fastText) Dense word vectors where similar words are close Features, similarity, analogies
Self-supervised language models (BERT, GPT-style) Contextual representations of tokens and sentences Fine-tuning, zero-shot tasks, generation
Sentence embeddings Vectors for whole sentences or documents Semantic search, clustering, deduplication
Topic models (LDA, NMF) Themes as distributions over words Exploring document collections
Document clustering Groups of similar texts Ticket triage, news grouping
Subword tokenization (BPE, WordPiece) Vocabulary from frequent character sequences Handling rare and unknown words

Why It Works: the Distributional Hypothesis

“Words that occur in similar contexts tend to have similar meanings.” By predicting words from their context (or the reverse), a model has to encode meaning, syntax and some world knowledge into its representations.

Code Example

from sklearn.decomposition import NMF
from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "the striker scored a goal in the final match", "the team won the league after a late goal",
    "the coach praised the defenders after the match", "the central bank raised interest rates",
    "inflation and interest rates worry investors", "stock markets fell as the bank tightened policy",
    "the new phone has a faster chip and better camera", "the laptop battery and chip got upgrades",
]
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(docs)

nmf = NMF(n_components=3, random_state=42, max_iter=500)
doc_topics = nmf.fit_transform(X)                  # document-topic weights
terms = vec.get_feature_names_out()
for t, weights in enumerate(nmf.components_):
    print(f"topic {t}:", ", ".join(terms[i] for i in weights.argsort()[::-1][:4]))
print("topic of each doc:", doc_topics.argmax(axis=1))

Common Pitfalls

  • Bias: embeddings and language models reproduce stereotypes from their training text.

  • Topic models need careful preprocessing (stop words, lemmatization) and a chosen number of topics.

  • Static embeddings give one vector per word, so “bank” (river) and “bank” (money) share it; contextual models fix this.

39

What is the role of unsupervised pre-training in deep learning?

Unsupervised pre-training trains a neural network on large amounts of unlabelled data first, so it learns general-purpose representations, and then fine-tunes it on a smaller labelled dataset for the actual task. Historically it helped initialize deep networks layer by layer; today it lives on as self-supervised pre-training, the basis of large language models and modern vision models. It greatly reduces the labelled data needed, at the cost of expensive pre-training compute and the risk of inheriting biases from the pre-training data.

Read full answer

Applies todeep learning concepts (examples in PyTorch 2.x, Python 3.12)

Historical Role (around 2006-2012)

  • Deep networks were hard to train with random initialization (vanishing gradients).

  • Researchers pre-trained layers one at a time as restricted Boltzmann machines (deep belief networks) or autoencoders, then fine-tuned the whole network with labels.

  • Better activations (ReLU), initialization schemes, batch normalization and large labelled datasets later made this layer-wise approach unnecessary for supervised training.

Modern Role: Self-Supervised Pre-Training

The model creates its own targets from unlabelled data:

Domain Pretext task Example models
Text Predict the next token, or masked tokens GPT-style models, BERT
Images Contrastive learning (match two augmented views), masked patch reconstruction SimCLR, MoCo, DINO, MAE
Audio Predict masked audio segments wav2vec 2.0
Multimodal Match images with their captions CLIP

Why It Helps

  • Data efficiency: fine-tuning needs far fewer labels.

  • Better generalization: representations capture general structure (grammar, edges, shapes).

  • Transfer: one pre-trained model serves many downstream tasks.

  • Regularization effect: starting from good weights steers optimization toward solutions that generalize.

Code Example

import torch
from torch import nn

torch.manual_seed(42)
unlabelled = torch.rand(5000, 64)                   # plenty of unlabelled data
labelled_x, labelled_y = torch.rand(200, 64), torch.randint(0, 3, (200,))  # few labels

encoder = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Linear(32, 16))
decoder = nn.Sequential(nn.Linear(16, 32), nn.ReLU(), nn.Linear(32, 64))

# Stage 1: unsupervised pre-training (reconstruct the input)
opt = torch.optim.Adam([*encoder.parameters(), *decoder.parameters()], lr=1e-3)
for _ in range(200):
    loss = nn.functional.mse_loss(decoder(encoder(unlabelled)), unlabelled)
    opt.zero_grad(); loss.backward(); opt.step()

# Stage 2: fine-tune encoder + a new classification head on the labelled data
classifier = nn.Sequential(encoder, nn.ReLU(), nn.Linear(16, 3))
opt = torch.optim.Adam(classifier.parameters(), lr=1e-4)   # smaller lr keeps pre-trained features
for _ in range(100):
    loss = nn.functional.cross_entropy(classifier(labelled_x), labelled_y)
    opt.zero_grad(); loss.backward(); opt.step()
print("fine-tuning loss:", round(loss.item(), 3))

Common Pitfalls

  • A domain mismatch between pre-training and target data reduces the benefit.

  • Fine-tuning with a high learning rate destroys the pre-trained features (“catastrophic forgetting”).

  • Interview tip: connect the historical answer (layer-wise RBM/autoencoder pre-training) to today’s self-supervised foundation models.

40

What are self-organizing maps in unsupervised learning?

A self-organizing map (SOM), or Kohonen map, is a neural network that projects high-dimensional data onto a low-dimensional grid of neurons (usually 2-D) while preserving topology: inputs that are similar end up on the same or nearby grid cells. It is used for visualization, clustering and exploring structure in data. Its trade-off is that the grid size, learning rate and neighbourhood schedule must be tuned, and it is less common today than PCA, t-SNE or UMAP.

Read full answer

Applies tomachine learning concepts (examples in NumPy 2.x, Python 3.12)

How It Works

  1. Grid: create neurons on a grid (e.g. 10 x 10); each has a weight vector the same size as the input.

  2. Initialize the weights randomly or from the first principal components.

  3. For each training sample x:

  4. Best matching unit (BMU): find the neuron whose weight vector is closest to x.

  5. Update the BMU and its grid neighbours toward x: w_j = w_j + eta(t) * h(j, BMU, t) * (x - w_j) where eta(t) is the learning rate and h is a neighbourhood function (often Gaussian in grid distance) whose radius shrinks over time.

  6. Decay the learning rate and neighbourhood radius as training proceeds: early steps organize the global layout, later steps fine-tune.

Using a Trained SOM

  • U-matrix: shows distances between neighbouring neurons; high values mark cluster boundaries.

  • Component planes: one map per input feature, showing how it varies across the grid.

  • Clustering: cluster the neuron weights (e.g. with k-means), then map each sample to its BMU’s cluster.

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)

rows, cols, epochs = 8, 8, 30
weights = rng.normal(size=(rows, cols, X.shape[1]))
grid = np.stack(np.meshgrid(np.arange(rows), np.arange(cols), indexing="ij"), axis=-1)

n_steps, step = epochs * len(X), 0
for epoch in range(epochs):
    for x in X[rng.permutation(len(X))]:
        frac = step / n_steps
        lr, radius = 0.5 * (1 - frac), max(4.0 * (1 - frac), 0.5)      # both decay over time
        bmu = np.unravel_index(np.argmin(((weights - x) ** 2).sum(axis=2)), (rows, cols))
        grid_dist2 = ((grid - np.array(bmu)) ** 2).sum(axis=2)
        h = np.exp(-grid_dist2 / (2 * radius ** 2))[..., None]          # Gaussian neighbourhood
        weights += lr * h * (x - weights)
        step += 1

bmus = [np.unravel_index(np.argmin(((weights - x) ** 2).sum(axis=2)), (rows, cols)) for x in X]
for species in range(3):
    cells = {bmus[i] for i in np.where(y == species)[0]}
    print(f"species {species} occupies {len(cells)} grid cells")

Libraries such as MiniSom provide tested implementations for real use.

SOM Versus Other Methods

Method Output Notes
SOM Discrete grid with prototypes Topology-preserving, prototypes are interpretable
k-means Unordered centroids No neighbourhood relation between clusters
PCA Linear projection Fast, no grid
t-SNE / UMAP Continuous 2-D embedding Better local structure, no prototypes

Common Pitfalls

  • Not scaling features before training.

  • A grid that is too small merges distinct groups; too large leaves many empty cells.

  • Reading grid distances as exact distances in the original space.

41

How do you implement k-means clustering from scratch in Python?

Implementing k-means from scratch takes three parts: initialize k centroids (k-means++ is best), then alternate between assigning each point to its nearest centroid and moving each centroid to the mean of its points, until the centroids stop moving. Writing it with NumPy shows how simple the algorithm is and where its weaknesses come from. In production, use sklearn.cluster.KMeans, which is faster and handles edge cases.

Read full answer

Applies toNumPy 2.x (Python 3.12)

The Algorithm

  1. k-means++ initialization: pick the first centroid at random; pick each next one with probability proportional to its squared distance from the nearest chosen centroid.

  2. Assignment: label_i = argmin_j ||x_i - c_j||^2.

  3. Update: c_j = mean of points with label j.

  4. Stop when centroids move less than a tolerance or after max_iter iterations.

  5. Restart several times and keep the run with the lowest inertia (sum of squared distances).

Code Example

import numpy as np


def kmeans_plus_plus(X, k, rng):
    centroids = [X[rng.integers(len(X))]]
    for _ in range(1, k):
        d2 = np.min(((X[:, None, :] - np.array(centroids)[None]) ** 2).sum(axis=2), axis=1)
        centroids.append(X[rng.choice(len(X), p=d2 / d2.sum())])
    return np.array(centroids)


def kmeans(X, k, n_init=10, max_iter=300, tol=1e-6, seed=42):
    """Return (labels, centroids, inertia) of the best of n_init runs."""
    rng = np.random.default_rng(seed)
    best = None
    for _ in range(n_init):
        centroids = kmeans_plus_plus(X, k, rng)
        for _ in range(max_iter):
            d2 = ((X[:, None, :] - centroids[None]) ** 2).sum(axis=2)   # (n, k) squared distances
            labels = d2.argmin(axis=1)
            new = np.array([X[labels == j].mean(axis=0) if np.any(labels == j)
                            else X[rng.integers(len(X))]                 # re-seed an empty cluster
                            for j in range(k)])
            shift = np.sum((new - centroids) ** 2)
            centroids = new
            if shift < tol:
                break
        d2 = ((X[:, None, :] - centroids[None]) ** 2).sum(axis=2)
        labels = d2.argmin(axis=1)
        inertia = d2[np.arange(len(X)), labels].sum()
        if best is None or inertia < best[2]:
            best = (labels, centroids, inertia)
    return best


rng = np.random.default_rng(0)
X = np.vstack([rng.normal(loc, 0.6, size=(100, 2)) for loc in ([0, 0], [4, 4], [0, 5])])
labels, centroids, inertia = kmeans(X, k=3)
print("centroids:\n", centroids.round(2))        # close to (0,0), (4,4), (0,5) in some order
print("cluster sizes:", np.bincount(labels), "inertia:", round(inertia, 1))

Complexity

  • O(n * k * d) per iteration for n points, k clusters and d features; this version builds an (n, k, d) array, so for large data compute distances in chunks.

Common Pitfalls

  • Random initialization without restarts, which often gets stuck in poor local minima.

  • Not handling empty clusters (they produce NaN centroids).

  • Forgetting to scale features before clustering.

  • Comparing your result with scikit-learn label by label: cluster numbering is arbitrary, so compare with the Adjusted Rand Index.

42

How do you compute the silhouette coefficient in Python?

The silhouette coefficient of a clustering is the average, over all points, of s(i) = (b(i) - a(i)) / max(a(i), b(i)), where a(i) is the mean distance from point i to the other points in its cluster and b(i) the mean distance to the points of the nearest other cluster. Writing it yourself shows exactly what it measures; in practice you use sklearn.metrics.silhouette_score. The trade-off is O(n^2) distance computations, so large datasets need sampling.

Read full answer

Applies toNumPy 2.x and scikit-learn 1.5+ (Python 3.12)

Algorithm

  1. Compute the pairwise distance matrix D.

  2. For each point i in cluster C:

  3. If C has only one point, s(i) = 0.

  4. a(i) = mean of D[i, j] for the other points j in C.

  5. b(i) = the minimum, over other clusters K, of the mean of D[i, j] for j in K.

  6. s(i) = (b(i) - a(i)) / max(a(i), b(i)).

  7. Return the mean of s(i). It needs at least 2 clusters and fewer clusters than points.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import pairwise_distances, silhouette_score


def silhouette_coefficient(X, labels, metric="euclidean"):
    """Mean silhouette coefficient, matching sklearn.metrics.silhouette_score."""
    X, labels = np.asarray(X), np.asarray(labels)
    unique = np.unique(labels)
    if not 2 <= len(unique) <= len(X) - 1:
        raise ValueError("need 2 <= number of clusters <= n_samples - 1")

    D = pairwise_distances(X, metric=metric)
    scores = np.zeros(len(X))
    for i in range(len(X)):
        same = labels == labels[i]
        n_same = same.sum()
        if n_same == 1:
            continue                                  # singleton cluster: s(i) = 0
        a = D[i, same].sum() / (n_same - 1)           # exclude the point itself (distance 0)
        b = min(D[i, labels == k].mean() for k in unique if k != labels[i])
        scores[i] = (b - a) / max(a, b)
    return scores.mean()


X, _ = make_blobs(n_samples=300, centers=3, random_state=42)
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(X)

print("custom :", round(silhouette_coefficient(X, labels), 6))
print("sklearn:", round(silhouette_score(X, labels), 6))   # the two values match

Notes

  • Complexity: O(n^2) time and memory for the distance matrix; for large n, use silhouette_score(X, labels, sample_size=10000, random_state=42).

  • Per-point scores: sklearn.metrics.silhouette_samples returns s(i) for silhouette plots.

  • Metric: pass metric="cosine" for text or embeddings.

Common Pitfalls

  • Dividing a(i) by the full cluster size instead of size minus one.

  • Including DBSCAN noise points (label -1) as if they were a cluster.

  • Comparing silhouette values computed on differently scaled features.

43

How do you reduce dimensions with PCA in scikit-learn?

PCA in scikit-learn is done with sklearn.decomposition.PCA: you standardize the features, fit PCA on the training data, and transform the data into a smaller number of principal components that keep most of the variance. You can set the number of components directly or as a target share of variance (e.g. n_components=0.95). The trade-off is between fewer dimensions (faster models, less noise) and the information that is discarded.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Steps

  1. Split the data first, so PCA learns only from training data.

  2. Standardize features with StandardScaler; PCA is variance-based, so units matter.

  3. Fit PCA and inspect explained_variance_ratio_ to choose the number of components.

  4. Transform training and test data with the same fitted PCA.

  5. Use a Pipeline to combine scaling, PCA and a model safely.

Code Example

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)          # 30 features
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)

# 1. Inspect variance to choose the number of components
scaler = StandardScaler().fit(X_train)
pca_full = PCA(random_state=42).fit(scaler.transform(X_train))
cumulative = np.cumsum(pca_full.explained_variance_ratio_)
print("variance kept by 2, 5, 10 components:", cumulative[[1, 4, 9]].round(3))

# 2. Keep enough components for 95% of the variance
pca = PCA(n_components=0.95, random_state=42).fit(scaler.transform(X_train))
X_train_pca = pca.transform(scaler.transform(X_train))
X_test_pca = pca.transform(scaler.transform(X_test))
print("components kept:", pca.n_components_, "| shapes:", X_train_pca.shape, X_test_pca.shape)

# 3. The same thing inside a pipeline, evaluated with cross-validation
pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression(max_iter=1000))
print("CV accuracy with PCA:", cross_val_score(pipe, X_train, y_train, cv=5).mean().round(3))

# 4. Which original features drive the first component (loadings)
top = np.argsort(np.abs(pca.components_[0]))[::-1][:3]
print("top features in PC1:", top)

Useful Options

Option Purpose
n_components=int Exact number of components
n_components=0.95 Smallest number keeping 95% of variance
svd_solver="randomized" Faster for large data and few components
whiten=True Scales components to unit variance (some models benefit)
inverse_transform Maps back to the original space (reconstruction)
IncrementalPCA Fits in batches when data does not fit in memory
TruncatedSVD For sparse matrices (no centering)

Common Pitfalls

  • Fitting PCA on all data before splitting (leakage).

  • Skipping standardization when features have different units.

  • Assuming the components are interpretable features; inspect components_ (loadings) instead.

44

How do you cluster spatial data with DBSCAN in Python?

DBSCAN on spatial data groups locations that are densely packed (for example, pickup points, store visits or crime reports) and labels isolated points as noise, without needing the number of clusters. For latitude/longitude data, use the haversine distance so eps is a real distance on the Earth’s surface, and a ball-tree index to keep it fast. The main trade-off is choosing eps and min_samples, which set what “dense” means.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Steps

  1. Prepare coordinates: latitude and longitude in degrees, converted to radians for haversine.

  2. Choose eps: a physical radius, e.g. 300 metres, divided by the Earth’s radius (about 6,371 km) to get radians.

  3. Choose min_samples: the minimum number of points within eps to form a dense area.

  4. Fit DBSCAN(metric="haversine", algorithm="ball_tree").

  5. Inspect: number of clusters, noise share, and a representative point per cluster.

Code Example

import numpy as np
from sklearn.cluster import DBSCAN

rng = np.random.default_rng(42)
# Synthetic GPS points around three hotspots in a city, plus scattered noise
hotspots = np.array([[40.7580, -73.9855], [40.7128, -74.0060], [40.7306, -73.9352]])
points = np.vstack([h + rng.normal(0, 0.0015, size=(150, 2)) for h in hotspots] +
                   [rng.uniform([40.70, -74.02], [40.78, -73.93], size=(60, 2))])

EARTH_RADIUS_M = 6_371_000
eps_m = 300                                           # neighbourhood radius in metres
db = DBSCAN(eps=eps_m / EARTH_RADIUS_M, min_samples=10,
            metric="haversine", algorithm="ball_tree")
labels = db.fit_predict(np.radians(points))           # haversine expects [lat, lon] in radians

n_clusters = len(set(labels) - {-1})
print("clusters:", n_clusters, "| noise points:", (labels == -1).sum())
for c in range(n_clusters):
    centre = points[labels == c].mean(axis=0)
    print(f"cluster {c}: {np.sum(labels == c)} points, centre {centre.round(4)}")

Choosing Parameters

  • k-distance plot: compute each point’s distance to its min_samples-th neighbour, sort, and pick eps at the knee.

  • Domain sense: 50 m for points inside one building, 1 km for neighbourhoods.

  • HDBSCAN (sklearn.cluster.HDBSCAN) handles clusters of varying density, such as dense downtown and sparse suburbs.

Common Pitfalls

  • Using Euclidean distance on raw degrees: one degree of longitude shrinks toward the poles, so distances are distorted.

  • Passing coordinates in [lon, lat] order; haversine in scikit-learn expects [lat, lon].

  • One eps for areas of very different density, which merges dense areas or labels sparse ones as noise.

  • Taking the centroid of a curved or ring-shaped cluster as a meaningful location; consider the medoid instead.

45

How do you implement the Apriori algorithm in Python?

Implementing Apriori means finding all itemsets whose support meets a threshold by growing them one item at a time: start with frequent single items, join frequent k-itemsets into (k + 1)-candidates, prune candidates that have an infrequent subset, count the rest, and repeat. Association rules are then generated from the frequent itemsets using confidence and lift. A from-scratch version is great for understanding; for large data use FP-Growth or a library.

Read full answer

Applies toPython 3.12 (standard library only)

Code Example

from itertools import combinations


def apriori(transactions, min_support):
    """Return {frozenset(itemset): support} for all frequent itemsets."""
    transactions = [frozenset(t) for t in transactions]
    n = len(transactions)

    def frequent_of(candidates):
        counts = {c: sum(c <= t for t in transactions) for c in candidates}
        return {c: cnt / n for c, cnt in counts.items() if cnt / n >= min_support}

    items = {frozenset([i]) for t in transactions for i in t}
    current = frequent_of(items)                       # frequent 1-itemsets
    result, k = dict(current), 1
    while current:
        prev = list(current)
        # Join step: unions of two frequent k-itemsets that have size k + 1
        candidates = {a | b for a, b in combinations(prev, 2) if len(a | b) == k + 1}
        # Prune step: every k-subset must be frequent (the Apriori property)
        candidates = {c for c in candidates
                      if all(frozenset(s) in current for s in combinations(c, k))}
        current = frequent_of(candidates)
        result.update(current)
        k += 1
    return result


def rules(frequent, min_confidence):
    out = []
    for itemset, sup in frequent.items():
        for r in range(1, len(itemset)):
            for antecedent in map(frozenset, combinations(itemset, r)):
                consequent = itemset - antecedent
                conf = sup / frequent[antecedent]
                if conf >= min_confidence:
                    lift = conf / frequent[consequent]
                    out.append((set(antecedent), set(consequent), round(sup, 2), round(conf, 2), round(lift, 2)))
    return sorted(out, key=lambda x: -x[4])


transactions = [
    {"bread", "milk"},
    {"bread", "diapers", "beer", "eggs"},
    {"milk", "diapers", "beer", "cola"},
    {"bread", "milk", "diapers", "beer"},
    {"bread", "milk", "diapers", "cola"},
]
frequent = apriori(transactions, min_support=0.6)
for itemset, sup in sorted(frequent.items(), key=lambda x: (len(x[0]), -x[1])):
    print(sorted(itemset), round(sup, 2))
# 4 single items (bread, milk, diapers 0.8; beer 0.6) and 4 pairs at 0.6; no triple reaches 0.6

for a, c, s, conf, lift in rules(frequent, min_confidence=0.7):
    print(f"{a} -> {c}: support={s}, confidence={conf}, lift={lift}")

Every subset of a frequent itemset is itself frequent, so the lookups frequent[antecedent] and frequent[consequent] always succeed.

Complexity and Improvements

  • Each level scans all transactions for every candidate: roughly O(candidates x transactions x itemset size).

  • Speed-ups: transaction ID lists (Eclat), hash trees for counting, or FP-Growth, which avoids candidates.

  • For real data, use mlxtend.frequent_patterns or Spark’s FPGrowth.

Common Pitfalls

  • Generating candidates without the prune step, which counts many itemsets needlessly.

  • A very low min_support on many items, which explodes the number of candidates.

  • Reporting rules by confidence alone; always check lift.

46

How would you segment customers for targeted marketing with unsupervised learning?

Customer segmentation groups customers with similar behaviour or needs so marketing can target each group with suitable messages, offers and channels. A solid strategy builds behavioural features such as recency, frequency and monetary value (RFM), scales them, clusters with k-means or a Gaussian mixture, and then profiles and validates each segment with the business. The trade-off is between many fine segments that are hard to act on and a few broad ones that may hide important differences.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

Strategy

  1. Define the goal: retention, upselling, reactivation; it decides which features matter.

  2. Build features per customer from a fixed time window:

  3. Recency: days since the last purchase.

  4. Frequency: number of orders.

  5. Monetary: total or average spend.

  6. Category mix, discount usage, channel, tenure, returns.

  7. Transform and scale: log-transform skewed spend and counts, then standardize.

  8. Cluster: k-means for clear, sized segments; Gaussian mixtures for overlapping groups.

  9. Choose k: silhouette, BIC and how many segments the marketing team can actually serve.

  10. Profile and name each segment (“lapsed high-spenders”, “new bargain hunters”).

  11. Validate: stability across months, different responses in A/B tests.

  12. Operationalize: score customers regularly and push segments to the CRM.

Code Example

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

rng = np.random.default_rng(42)
n = 2000
rfm = pd.DataFrame({
    "recency_days": rng.exponential(60, n).round(),
    "frequency": rng.poisson(5, n) + 1,
    "monetary": rng.lognormal(5, 0.8, n).round(2),
})

pipe = make_pipeline(FunctionTransformer(np.log1p), StandardScaler())
X = pipe.fit_transform(rfm)

for k in range(3, 7):
    labels = KMeans(n_clusters=k, n_init=10, random_state=42).fit_predict(X)
    print(f"k={k}: silhouette={silhouette_score(X, labels):.3f}")

rfm["segment"] = KMeans(n_clusters=4, n_init=10, random_state=42).fit_predict(X)
profile = rfm.groupby("segment").agg(customers=("monetary", "size"), recency=("recency_days", "median"),
                                     frequency=("frequency", "median"), spend=("monetary", "median"))
print(profile.round(1))

Turning Segments Into Actions

Segment profile Example action
Recent, frequent, high spend Loyalty perks, early access
High spend but long inactive Win-back campaign
New, low frequency Onboarding sequence, second-purchase incentive
Frequent, low spend Bundles and upselling

Common Pitfalls

  • Including features the business cannot act on, or IDs and dates that create meaningless clusters.

  • Not log-transforming spend, so a few big spenders become their own cluster.

  • Never refreshing segments; behaviour changes over time.

47

How can clustering create features for a supervised model?

Clustering for feature creation uses unsupervised structure as extra input for a supervised model: the cluster label, the distance to each cluster centre, or soft membership probabilities become new features. They can summarize complex interactions (a “type” of customer or location) that a simple model would struggle to learn. The trade-off is that the features can leak information if the clustering is fitted on test data, and they add little for flexible models like gradient boosting that already learn such groupings.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Ways to Use Clusters as Features

Feature How Benefit
Cluster ID One-hot encode the k-means label Captures group effects for linear models
Distances to centroids KMeans.transform(X) gives k distances Smooth, informative, works well with linear models
Soft memberships GaussianMixture.predict_proba(X) Expresses uncertainty between groups
Geographic clusters DBSCAN or k-means on coordinates Neighbourhood effects (prices, demand)
Cluster-level statistics Mean target per cluster, computed out-of-fold Strong signal, but needs cross-fitting to avoid leakage

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import FeatureUnion, make_pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

X, y = make_moons(n_samples=1000, noise=0.25, random_state=42)
cv = StratifiedKFold(5, shuffle=True, random_state=42)

baseline = make_pipeline(StandardScaler(), LogisticRegression())

# Original features + distances to 10 k-means centroids, all fitted inside each CV fold
with_clusters = make_pipeline(
    StandardScaler(),
    FeatureUnion([
        ("original", FunctionTransformer()),                          # identity: keep the inputs
        ("dist", KMeans(n_clusters=10, n_init=10, random_state=42)),  # transform() -> distances
    ]),
    LogisticRegression(max_iter=1000),
)

for name, model in [("logistic regression", baseline), ("+ cluster distances", with_clusters)]:
    print(name, cross_val_score(model, X, y, cv=cv, scoring="roc_auc").mean().round(3))
# The cluster distances let a linear model capture the curved class boundary.

When It Helps Most

  • Linear models or k-NN on data with non-linear group structure.

  • High-cardinality spatial data (coordinates to neighbourhood clusters).

  • Combining many correlated behaviour features into a few profile features.

Common Pitfalls

  • Leakage: fitting the clustering on the full dataset before cross-validation. Put it inside the pipeline as above.

  • Too many clusters: the features start to memorize the training data.

  • Unstable clusters: if labels change between runs, the model sees a different feature each time; distances are more stable than IDs.

  • Expecting gains with tree ensembles; measure with cross-validation before keeping the features.

48

How would you group similar documents with unsupervised learning?

Grouping similar documents means turning each document into a numeric vector that captures its meaning, then clustering those vectors so documents about the same subject end up together. A practical approach is TF-IDF or sentence embeddings, optional dimensionality reduction, then k-means or HDBSCAN, followed by labelling each cluster with its top terms. The trade-off is between simple, interpretable TF-IDF features and embeddings that understand synonyms but are heavier to compute.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Approach

  1. Clean text: lowercase, remove boilerplate, signatures and duplicates.

  2. Represent:

  3. TF-IDF: word weights; fast and interpretable.

  4. Sentence embeddings (e.g. from a sentence-transformer model): capture meaning, so “car” and “automobile” are close.

  5. Reduce dimensions: truncated SVD (LSA) for TF-IDF, or UMAP for embeddings.

  6. Normalize vectors to unit length so Euclidean k-means behaves like cosine similarity.

  7. Cluster: k-means when you want a fixed number of groups, HDBSCAN when you want the number found automatically and outliers left out.

  8. Label and review: top terms per cluster and a few sample documents; merge or split clusters as needed.

  9. Evaluate: silhouette with cosine distance, stability, and human review.

Code Example

import numpy as np
from sklearn.cluster import KMeans
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import silhouette_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer

docs = [
    "team wins championship after penalty shootout", "striker scores twice in derby match",
    "coach signs new contract with football club", "central bank keeps interest rates unchanged",
    "inflation slows as energy prices fall", "stocks rally after strong earnings reports",
    "new smartphone launches with faster processor", "chip maker unveils energy efficient processor",
    "software update improves laptop battery life",
]
tfidf = TfidfVectorizer(stop_words="english", sublinear_tf=True)
X = tfidf.fit_transform(docs)

lsa = make_pipeline(TruncatedSVD(n_components=5, random_state=42), Normalizer(copy=False))
X_lsa = lsa.fit_transform(X)

km = KMeans(n_clusters=3, n_init=20, random_state=42).fit(X_lsa)
print("silhouette (cosine):", round(silhouette_score(X_lsa, km.labels_, metric="cosine"), 3))

# Top terms: map centroids back from LSA space to term space
centroids_terms = lsa[0].inverse_transform(km.cluster_centers_)
terms = tfidf.get_feature_names_out()
for c in range(3):
    top = terms[np.argsort(centroids_terms[c])[::-1][:3]]
    members = [d[:30] for d, l in zip(docs, km.labels_) if l == c]
    print(f"cluster {c} [{', '.join(top)}]: {members}")

Common Pitfalls

  • Clustering raw counts without TF-IDF weighting or normalization, so long documents dominate.

  • Ignoring duplicates and templates, which form artificial clusters.

  • Choosing k only by metrics; check that topics make sense to readers.

49

How do you detect communities in social networks with unsupervised learning?

Community detection finds groups of nodes in a network that are densely connected to each other and sparsely connected to the rest, such as friend circles, interest groups or coordinated accounts. It is unsupervised: the algorithm uses only the graph structure (and optionally node features). A good framework covers graph construction, choosing an algorithm (modularity-based such as Louvain/Leiden, label propagation, spectral or embedding-based), evaluation and interpretation; the trade-off is that different algorithms and resolutions give different, equally plausible partitions.

Read full answer

Applies tomachine learning concepts (examples in NetworkX 3.x, Python 3.12)

Framework

  1. Build the graph: nodes are users; edges are follows, messages or interactions, weighted by frequency. Decide directed or undirected, and filter weak or spam edges.

  2. Choose an algorithm:

Algorithm Idea Notes
Louvain / Leiden Greedily maximize modularity, then aggregate communities Fast, scales to millions of nodes; Leiden avoids badly connected communities
Label propagation Nodes adopt the most common label among neighbours Very fast, less stable
Spectral clustering k-means on eigenvectors of the graph Laplacian Needs the number of communities
Girvan-Newman Remove high-betweenness edges Intuitive but slow; small graphs only
Node embeddings + clustering node2vec or graph neural networks, then k-means/HDBSCAN Can use node features too
Overlapping methods Clique percolation, mixed-membership models People belong to several communities
  1. Evaluate: modularity Q (fraction of edges inside communities minus the expected fraction in a random graph with the same degrees), conductance, community size distribution and stability across runs.

  2. Interpret: describe communities by shared attributes, topics or central members (high degree or PageRank).

  3. Use: recommendations, targeted content, influence analysis, detecting coordinated inauthentic behaviour.

Code Example

import networkx as nx
from networkx.algorithms import community

G = nx.karate_club_graph()                         # classic social network of 34 club members

communities = community.louvain_communities(G, resolution=1.0, seed=42)
print("communities found:", len(communities))
print("modularity:", round(community.modularity(G, communities), 3))
for i, c in enumerate(sorted(communities, key=len, reverse=True)):
    hub = max(c, key=G.degree)                      # most connected member
    print(f"community {i}: {len(c)} members, hub node {hub}")

lp = community.label_propagation_communities(G)
print("label propagation found:", len(list(lp)), "communities")

Changing resolution produces fewer, larger communities (lower values) or more, smaller ones (higher values).

Common Pitfalls

  • Resolution limit: modularity optimization can merge small but real communities.

  • Hubs and bots distort structure; consider removing or down-weighting them.

  • Single run: Louvain is randomized; check stability across seeds.

  • Ignoring time: communities change; use snapshots or temporal methods.

50

How does unsupervised learning help find patterns in genomic data?

Unsupervised learning in genomics discovers structure in very high-dimensional biological data, such as gene expression for thousands of genes across many samples or cells, without predefined labels. Clustering finds cell types, disease subtypes and co-expressed gene modules; dimensionality reduction reveals population structure and developmental trajectories. It drives discovery, but results are sensitive to normalization and batch effects, so they need biological validation.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Typical Applications

Task Method Insight
Cell-type discovery (single-cell RNA-seq) PCA, then graph clustering (Leiden/Louvain), visualized with UMAP New or rare cell types
Cancer subtyping Hierarchical clustering, consensus clustering, NMF Patient groups with different outcomes
Gene co-expression modules Correlation networks, hierarchical clustering Genes acting in the same pathway
Population structure PCA on genotype data Ancestry groups; correcting confounding in association studies
Mutational signatures NMF on mutation counts Processes such as UV damage or smoking
Trajectory inference Pseudotime on reduced data How cells develop or differentiate

Typical Pipeline (Expression Data)

  1. Quality control: remove low-quality samples or cells.

  2. Normalize for sequencing depth and log-transform counts.

  3. Select highly variable genes (often a few thousand).

  4. Correct batch effects if samples came from different runs or labs.

  5. Reduce with PCA (e.g. 50 components).

  6. Cluster on the reduced data.

  7. Interpret: find marker genes per cluster and compare with known biology.

Code Example

import numpy as np
from sklearn.cluster import AgglomerativeClustering
from sklearn.decomposition import PCA
from sklearn.metrics import adjusted_rand_score

rng = np.random.default_rng(42)
n_per_group, n_genes = 100, 2000
groups = np.repeat([0, 1, 2], n_per_group)
counts = rng.poisson(5, size=(len(groups), n_genes)).astype(float)
for g in range(3):                                   # each group over-expresses 50 marker genes
    counts[groups == g, g * 50:(g + 1) * 50] += rng.poisson(15, size=(n_per_group, 50))

lib_size = counts.sum(axis=1, keepdims=True)
log_norm = np.log1p(counts / lib_size * 1e4)          # depth-normalize, then log
variable = np.argsort(log_norm.var(axis=0))[::-1][:500]   # top variable genes

pcs = PCA(n_components=20, random_state=42).fit_transform(log_norm[:, variable])
labels = AgglomerativeClustering(n_clusters=3, linkage="ward").fit_predict(pcs)
print("ARI vs simulated groups:", round(adjusted_rand_score(groups, labels), 3))

for c in range(3):                                   # marker genes: highest mean difference
    diff = log_norm[labels == c].mean(axis=0) - log_norm[labels != c].mean(axis=0)
    print(f"cluster {c} top marker genes:", np.argsort(diff)[::-1][:5])

Dedicated tools such as Scanpy or Seurat implement these workflows for real single-cell data.

Common Pitfalls

  • Batch effects that look like biological clusters.

  • p >> n: thousands of genes and few samples; reduce dimensions and validate stability.

  • Over-clustering: splitting one cell type into many clusters at high resolution.

  • Treating t-SNE or UMAP distances as biological distances.

51

What are recent advances in clustering algorithms?

Recent advances in clustering have focused on handling varying densities without hand-tuned parameters, scaling to huge datasets, clustering learned representations instead of raw features, and graph-based methods for complex data. Notable examples include HDBSCAN, deep clustering on self-supervised embeddings, the Leiden algorithm for graph communities, UMAP-then-cluster pipelines and scalable approximate methods. The trade-off is often more moving parts: new methods depend on embeddings or graphs that themselves need tuning.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Key Developments

Advance Idea Why it matters
HDBSCAN Builds a density hierarchy and keeps the most stable clusters Varying densities, no eps, noise labelling; in scikit-learn since 1.3
Clustering on embeddings Cluster vectors from pretrained models (sentence transformers, vision models) Semantic text, image and audio clusters
Deep clustering Train the representation and cluster assignments together (e.g. DEC, SCAN, DeepCluster) Better clusters on images and text
Graph community detection Leiden improves Louvain with guaranteed well-connected communities Standard in single-cell genomics and networks
UMAP + density clustering Reduce with UMAP, cluster with HDBSCAN Popular for topic modelling (e.g. BERTopic)
Scalable methods Mini-batch and streaming k-means, approximate nearest-neighbour graphs, GPU libraries Millions to billions of points
Spectral clustering at scale Approximations (Nystrom, landmarks) Non-convex shapes on large data
Clustering with LLMs Embeddings for grouping, LLMs to name and summarize clusters Faster interpretation of results
Fair and constrained clustering Balance protected groups or respect must-link/cannot-link constraints Responsible, domain-aware results

Code Example

import numpy as np
from sklearn.cluster import DBSCAN, HDBSCAN
from sklearn.datasets import make_blobs
from sklearn.metrics import adjusted_rand_score

# Clusters of very different densities: hard for one DBSCAN eps
X, y = make_blobs(n_samples=[400, 400, 100], centers=[[0, 0], [6, 6], [12, 0]],
                  cluster_std=[0.3, 1.5, 0.8], random_state=42)

hdb = HDBSCAN(min_cluster_size=20).fit(X)
db = DBSCAN(eps=0.5, min_samples=10).fit(X)

for name, labels in [("HDBSCAN", hdb.labels_), ("DBSCAN eps=0.5", db.labels_)]:
    print(f"{name}: clusters={len(set(labels) - {-1})}, noise={np.sum(labels == -1)}, "
          f"ARI={adjusted_rand_score(y, labels):.3f}")

Practical Takeaways

  • For text, images and audio, the biggest improvement usually comes from better embeddings, not a new clustering algorithm.

  • HDBSCAN is a strong default when the number of clusters is unknown and densities vary.

  • Always validate: newer methods still produce clusters on noise.

Common Pitfalls

  • Chasing state-of-the-art benchmarks that do not match your data.

  • Clustering on t-SNE output, which can create false gaps.

  • Assuming deep clustering works with small datasets; it needs large data or strong pretrained features.

52

How is unsupervised learning used in reinforcement learning?

Unsupervised learning supports reinforcement learning (RL) by learning useful representations, models and exploration signals from experience without reward labels. Examples include learning compact state representations from pixels, world models that predict future observations, curiosity-driven exploration based on prediction error, discovering skills without rewards, and pre-training on unlabelled data. These methods make RL more sample-efficient, especially when rewards are sparse; the trade-off is extra components that can learn features irrelevant to the actual task.

Read full answer

Applies tomachine learning concepts (deep RL research and practice)

Main Uses

Use Idea Examples
State representation learning Compress high-dimensional observations into informative features Autoencoders, contrastive learning (CURL), data augmentation methods
World models Learn to predict the next observation or latent state, then plan or train in imagination World Models (VAE + RNN), Dreamer, MuZero-style learned models
Intrinsic motivation / curiosity Reward the agent for surprising or novel states Intrinsic Curiosity Module (ICM), Random Network Distillation (RND), count-based bonuses
Unsupervised skill discovery Learn diverse behaviours without external reward DIAYN (Diversity Is All You Need)
Pre-training Learn from unlabelled videos or offline logs before fine-tuning with rewards Pretrained visual encoders, offline RL datasets
Auxiliary tasks Extra self-supervised losses alongside the RL loss UNREAL (pixel control, reward prediction)
State abstraction and clustering Group similar states to simplify planning Clustering of latent states, options discovery

How Curiosity Works (Example)

  1. A model predicts the next state’s features from the current state and action.

  2. The prediction error is added to the reward as an intrinsic bonus.

  3. The agent seeks states it cannot yet predict, so it explores; as the model improves, the bonus fades.

  4. RND makes this robust by predicting the output of a fixed random network, which avoids being distracted by inherently unpredictable noise to some extent.

Benefits

  • Sample efficiency: good representations reduce how many environment interactions are needed.

  • Sparse rewards: intrinsic rewards drive exploration when external rewards are rare (e.g. games where the first reward comes after many steps).

  • Transfer: skills and representations learned without rewards can be reused across tasks.

Common Pitfalls

  • The noisy TV problem: naive curiosity is attracted to random, unpredictable signals.

  • Irrelevant features: reconstruction losses spend capacity on background pixels that do not matter for control.

  • Model errors compound when planning with a learned world model over long horizons.

  • Interview tip: give one concrete method per category (e.g. RND for exploration, Dreamer for world models).

53

Why is interpretability challenging for unsupervised learning models?

Interpretability in unsupervised learning is challenging because there is no target to anchor explanations: a cluster, component or anomaly score has no predefined meaning, so humans must infer what it represents. Results also depend on preprocessing, distance measures and random initialization, and methods like t-SNE, UMAP or autoencoders create abstract, non-linear representations. Good practice combines profiling, surrogate models and stability checks, accepting that some uncertainty about meaning always remains.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+, Python 3.12)

Main Challenges

  • No ground truth: you cannot check an explanation against correct labels.

  • Arbitrary choices shape results: features, scaling, distance metric, k and algorithm all change the clusters.

  • Abstract representations: PCA components mix many features; autoencoder codes and embeddings have no direct meaning.

  • Instability: different seeds or samples can give different clusters with different “stories”.

  • Visualization traps: t-SNE and UMAP distort distances and cluster sizes.

  • Anomaly scores: “unusual” in which way? Scores rarely say why.

  • Confirmation bias: analysts can find a plausible narrative for almost any cluster.

Techniques That Help

Technique What it explains
Cluster profiling Mean or median of each feature per cluster versus the overall average
Surrogate model A shallow decision tree trained to predict cluster labels; its rules describe clusters
SHAP on the surrogate Which features separate each cluster
PCA loadings Which original features drive each component
Prototypes / medoids Representative real examples per cluster
Stability analysis Which clusters are robust and worth interpreting
Feature attribution for anomalies Which features deviate most for a flagged point

Code Example

from sklearn.cluster import KMeans
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier, export_text

data = load_wine(as_frame=True)
X = data.data
labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(StandardScaler().fit_transform(X))

# 1. Profile: how each cluster differs from the overall mean (in standard deviations)
profile = (X.groupby(labels).mean() - X.mean()) / X.std()
print(profile[["alcohol", "flavanoids", "color_intensity", "proline"]].round(2))

# 2. Surrogate tree: human-readable rules that reproduce the clusters
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, labels)
print("surrogate fidelity:", round(tree.score(X, labels), 3))   # how well the rules match the clusters
print(export_text(tree, feature_names=list(X.columns)))

Common Pitfalls

  • Naming clusters from one or two features while ignoring the rest of the profile.

  • Explaining unstable clusters that disappear with another seed.

  • Presenting a surrogate’s rules without its fidelity score.

54

How is unsupervised learning used for multilingual text analysis?

Unsupervised learning for multilingual text relies on representations that place texts with the same meaning close together regardless of language. Multilingual language models and cross-lingual embeddings are learned from unlabelled or loosely aligned text, so you can cluster, search, deduplicate and topic-model documents in many languages together without translation or labelled data. The trade-off is weaker quality for low-resource languages and possible language bias, where texts group by language instead of topic.

Read full answer

Applies toNLP concepts (examples in sentence-transformers 3.x and scikit-learn 1.5+, Python 3.12)

Techniques

Technique How it is learned Use
Aligned word embeddings Monolingual embeddings mapped to a shared space (e.g. adversarial alignment as in MUSE, or Procrustes with a small dictionary) Bilingual lexicon induction, word translation
Multilingual language models Self-supervised masked or next-token prediction on many languages at once (mBERT, XLM-R) Shared contextual representations
Multilingual sentence embeddings Trained so translations map to the same vector (e.g. LaBSE, multilingual sentence-transformers) Cross-lingual search, clustering
Cross-lingual topic modelling Cluster multilingual embeddings, then extract topic words per language Topics across a global corpus
Unsupervised machine translation Back-translation and denoising without parallel data Translation for low-resource pairs

Example Workflow: Clustering Customer Feedback in Many Languages

  1. Embed each message with a multilingual sentence-embedding model.

  2. Normalize the vectors and optionally reduce them with UMAP.

  3. Cluster with k-means or HDBSCAN.

  4. Check that clusters mix languages (a sign they group by topic, not language).

  5. Label clusters with top terms or an LLM summary per cluster.

Code Example

import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans

texts = [
    "The delivery was very late", "La entrega llegó muy tarde", "Die Lieferung kam sehr spät",
    "Great product quality", "Excelente calidad del producto", "Sehr gute Produktqualität",
    "I want a refund", "Quiero un reembolso", "Ich möchte eine Rückerstattung",
]
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
emb = model.encode(texts, normalize_embeddings=True)       # same meaning -> nearby vectors

labels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(emb)
for c in range(3):
    print(c, [t for t, l in zip(texts, labels) if l == c])
# Expected: each cluster groups one topic across English, Spanish and German.

Common Pitfalls

  • Clusters forming by language because the embedding model is not truly cross-lingual.

  • Low-resource languages and code-switched text getting poor embeddings.

  • Tokenization and script differences when using TF-IDF; monolingual bag-of-words features do not align across languages.

  • Evaluating only in English; check examples in every language.

55

What role does unsupervised learning play in big data analytics?

Unsupervised learning in big data analytics turns huge volumes of unlabelled data (logs, clicks, transactions, sensor streams, text) into structure people can use: segments, anomalies, topics, compressed features and associations. It matters because labelling at that scale is impractical, and most of the data has no target anyway. The trade-off is that algorithms must be scalable (distributed, streaming or approximate) and results must be validated, since patterns at scale can be spurious or driven by data quality issues.

Read full answer

Applies tomachine learning concepts (examples in Spark 3.5 MLlib, Python 3.12)

Roles

Role Example Typical scalable method
Segmentation Millions of customers grouped by behaviour Distributed k-means, bisecting k-means
Anomaly detection Fraud, intrusions, faulty sensors in streams Isolation Forest on samples, streaming statistics, autoencoders
Dimensionality reduction Compress thousands of features Distributed PCA / SVD
Topic discovery Themes in support tickets, reviews, news LDA in Spark, embeddings + clustering
Association mining Products bought together across billions of baskets FP-Growth (Spark MLlib)
Deduplication and entity resolution Matching records across systems Locality-sensitive hashing (MinHash, random projections)
Representation learning Embeddings for users, items, text Word2vec, self-supervised models
Data exploration Understanding structure before modelling Sampling + clustering, approximate statistics

Scaling Techniques

  • Distributed algorithms: Spark MLlib, Dask-ML, Ray.

  • Sampling: fit on a representative sample, then score all data in parallel.

  • Mini-batch and streaming updates: models that update as data arrives.

  • Approximation: approximate nearest neighbours, sketches (HyperLogLog, count-min), LSH.

  • GPU acceleration: RAPIDS cuML for k-means, DBSCAN, UMAP.

Code Example

from pyspark.sql import SparkSession
from pyspark.ml.fpm import FPGrowth

spark = SparkSession.builder.appName("baskets").getOrCreate()
baskets = spark.createDataFrame(
    [(1, ["bread", "milk"]), (2, ["bread", "diapers", "beer"]), (3, ["milk", "diapers", "beer"]),
     (4, ["bread", "milk", "diapers", "beer"]), (5, ["bread", "milk", "diapers"])],
    ["id", "items"],
)   # in production: billions of rows read from a data lake

fp = FPGrowth(itemsCol="items", minSupport=0.4, minConfidence=0.6).fit(baskets)
fp.freqItemsets.orderBy("freq", ascending=False).show()
fp.associationRules.orderBy("lift", ascending=False).show()

Common Pitfalls

  • Garbage in, structure out: duplicates, bots and logging errors create convincing but fake patterns.

  • Collecting to one machine: use distributed tools or sampling.

  • Statistical significance: with billions of rows, tiny differences look “significant”; focus on effect size and business value.

  • Drift: segments and anomalies change; schedule re-training and monitoring.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 55 Unsupervised Learning answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects