What is cluster analysis in machine learning?
Cluster analysis is an unsupervised learning technique that groups observations with similar features without using labeled training examples. It helps solve problems such as customer segmentation, grouping similar documents, detecting unusual observations, and summarizing data as representative profiles. Its main trade-off is flexibility versus interpretability and computational cost: flexible methods can model complex structures but may be unstable or sensitive to preprocessing and parameter choices.
Read full answerHide full answer
Applies tomachine learning concepts (Python 3.12; scikit-learn 1.5+ for available implementations)
Purpose and Workflow
Cluster analysis can help answer questions such as:
-
Which customers have similar purchasing behavior?
-
Which observations may be unusual or belong to a separate group?
-
Are there meaningful groups among documents represented as embeddings?
-
Can many observations be summarized using a smaller number of representative profiles?
A typical workflow is:
-
Clean the data, handle missing values, and remove invalid records.
-
Scale features when their units or ranges would otherwise dominate distance calculations.
-
Choose an algorithm based on the expected cluster shapes and whether noise matters.
-
Tune parameters and assess the internal quality and stability of the clusters.
-
Profile each cluster using feature distributions, representative observations, and domain knowledge.
-
Validate whether the groups are useful for the intended application.
Clusters are mathematical partitions, not automatically meaningful ground-truth categories. Domain experts must interpret the groups and decide whether they are stable, useful, and actionable.
Algorithm Comparison
| Algorithm | How It Works | Cluster Shapes Found | Choose k? |
Handles Noise? | Scalability and Main Trade-off |
|---|---|---|---|---|---|
| k-means | Assigns each observation to its nearest centroid and repeatedly updates the centroids | Compact, roughly spherical clusters | Yes | No | O(n * k * d) per iteration; fast but sensitive to initialization, scaling, and outliers |
| k-medoids | Partitions observations around representative data points called medoids | Usually compact groups, though non-spherical groups are possible with suitable dissimilarity | Yes | No explicit mechanism | More expensive than k-means, but centers remain directly interpretable observations |
| Agglomerative hierarchical | Repeatedly merges the most similar clusters and can represent the result as a dendrogram | Depends on linkage; nested clusters are available | Can select after the hierarchy is built | No explicit mechanism | Typically quadratic in the number of observations because of pairwise-distance calculations |
| DBSCAN | Expands clusters through density-connected observations | Arbitrary shapes | No | Yes | Exact neighborhood searches can require O(n^2) time or memory, especially for high-dimensional data |
| HDBSCAN | Builds a hierarchy of clusters with varying densities | Flexible shapes and varying densities | No | Yes | Generally more computationally demanding than fixed-density DBSCAN |
| Gaussian mixture | Models the data as a probabilistic mixture of Gaussian components | Elliptical clusters with soft assignments | Yes | No explicit mechanism | Iterative fitting can be costly and less effective in high-dimensional spaces |
| Spectral clustering | Uses graph connectivity and eigenvectors of the graph Laplacian | Complex or non-convex shapes | Usually | No | Common implementations require a dense affinity matrix and scale poorly |
Agglomerative clustering also supports different linkage criteria. Single linkage can create chaining effects, complete linkage favors compact clusters, average linkage balances the two, and Ward linkage minimizes the increase in within-cluster variance.
Distance Metrics and Scaling
The algorithm usually determines how observations are considered similar:
-
Euclidean distance measures straight-line separation and is common for k-means and hierarchical clustering.
-
Manhattan distance sums the absolute differences across coordinates rather than squaring them, as Euclidean distance does.
-
Cosine distance compares vector direction rather than magnitude, making it useful for text embeddings and other directional representations.
-
Connectivity-based distances identify groups through density or graph relationships rather than direct pairwise distance alone.
Feature scaling is important for distance-based methods because variables with large numeric ranges can dominate the result. Standardization is common, while robust scaling can help when outliers affect the scale. Missing values must generally be handled before clustering because many algorithms require complete numeric feature matrices.
Choosing and Evaluating Clusters
There is rarely one universally correct cluster count.
-
Elbow method: Plot within-cluster variation against
kand look for a bend. -
Silhouette score: Measures whether observations are closer to their own cluster than to neighboring clusters. Values range from
-1to1, with higher values generally indicating better separation. -
Davies-Bouldin index: Measures the average ratio of within-cluster spread to separation between clusters; lower is generally better.
-
BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.
-
Domain judgment: Determines whether the clusters are useful, actionable, and stable.
If representative ground-truth labels are available, external measures such as the adjusted Rand index, normalized mutual information, homogeneity, and completeness can compare discovered clusters with those labels. These labels should be used for evaluation rather than fitting when the goal is genuinely unsupervised discovery.
Common Pitfalls
-
Skipping feature scaling: Distance-based clusters may reflect measurement units rather than meaningful similarity.
-
Ignoring algorithm assumptions: k-means favors compact clusters, while DBSCAN assumes density-connectivity and can fail when densities vary sharply.
-
Treating clusters as ground truth: Profile and interpret each cluster before using it.
-
Choosing parameters from one metric only: Combine statistical criteria, stability checks, and domain knowledge.
-
Over-tuning on one validation set: Assignments may change considerably with
k,eps, linkage, or distance metric. -
Leaking preprocessing information: When clustering is followed by held-out evaluation, fit preprocessing only on the training partition.