Skip to content
Machine Learning & Data Science

Cluster Analysis interview questions and answers

50 questions, 34 with code. About 5 h 39 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is cluster analysis in machine learning?

Cluster analysis is an unsupervised learning technique that groups observations with similar features without using labeled training examples. It helps solve problems such as customer segmentation, grouping similar documents, detecting unusual observations, and summarizing data as representative profiles. Its main trade-off is flexibility versus interpretability and computational cost: flexible methods can model complex structures but may be unstable or sensitive to preprocessing and parameter choices.

Read full answer

Applies tomachine learning concepts (Python 3.12; scikit-learn 1.5+ for available implementations)

Purpose and Workflow

Cluster analysis can help answer questions such as:

  • Which customers have similar purchasing behavior?

  • Which observations may be unusual or belong to a separate group?

  • Are there meaningful groups among documents represented as embeddings?

  • Can many observations be summarized using a smaller number of representative profiles?

A typical workflow is:

  1. Clean the data, handle missing values, and remove invalid records.

  2. Scale features when their units or ranges would otherwise dominate distance calculations.

  3. Choose an algorithm based on the expected cluster shapes and whether noise matters.

  4. Tune parameters and assess the internal quality and stability of the clusters.

  5. Profile each cluster using feature distributions, representative observations, and domain knowledge.

  6. Validate whether the groups are useful for the intended application.

Clusters are mathematical partitions, not automatically meaningful ground-truth categories. Domain experts must interpret the groups and decide whether they are stable, useful, and actionable.

Algorithm Comparison

Algorithm How It Works Cluster Shapes Found Choose k? Handles Noise? Scalability and Main Trade-off
k-means Assigns each observation to its nearest centroid and repeatedly updates the centroids Compact, roughly spherical clusters Yes No O(n * k * d) per iteration; fast but sensitive to initialization, scaling, and outliers
k-medoids Partitions observations around representative data points called medoids Usually compact groups, though non-spherical groups are possible with suitable dissimilarity Yes No explicit mechanism More expensive than k-means, but centers remain directly interpretable observations
Agglomerative hierarchical Repeatedly merges the most similar clusters and can represent the result as a dendrogram Depends on linkage; nested clusters are available Can select after the hierarchy is built No explicit mechanism Typically quadratic in the number of observations because of pairwise-distance calculations
DBSCAN Expands clusters through density-connected observations Arbitrary shapes No Yes Exact neighborhood searches can require O(n^2) time or memory, especially for high-dimensional data
HDBSCAN Builds a hierarchy of clusters with varying densities Flexible shapes and varying densities No Yes Generally more computationally demanding than fixed-density DBSCAN
Gaussian mixture Models the data as a probabilistic mixture of Gaussian components Elliptical clusters with soft assignments Yes No explicit mechanism Iterative fitting can be costly and less effective in high-dimensional spaces
Spectral clustering Uses graph connectivity and eigenvectors of the graph Laplacian Complex or non-convex shapes Usually No Common implementations require a dense affinity matrix and scale poorly

Agglomerative clustering also supports different linkage criteria. Single linkage can create chaining effects, complete linkage favors compact clusters, average linkage balances the two, and Ward linkage minimizes the increase in within-cluster variance.

Distance Metrics and Scaling

The algorithm usually determines how observations are considered similar:

  • Euclidean distance measures straight-line separation and is common for k-means and hierarchical clustering.

  • Manhattan distance sums the absolute differences across coordinates rather than squaring them, as Euclidean distance does.

  • Cosine distance compares vector direction rather than magnitude, making it useful for text embeddings and other directional representations.

  • Connectivity-based distances identify groups through density or graph relationships rather than direct pairwise distance alone.

Feature scaling is important for distance-based methods because variables with large numeric ranges can dominate the result. Standardization is common, while robust scaling can help when outliers affect the scale. Missing values must generally be handled before clustering because many algorithms require complete numeric feature matrices.

Choosing and Evaluating Clusters

There is rarely one universally correct cluster count.

  • Elbow method: Plot within-cluster variation against k and look for a bend.

  • Silhouette score: Measures whether observations are closer to their own cluster than to neighboring clusters. Values range from -1 to 1, with higher values generally indicating better separation.

  • Davies-Bouldin index: Measures the average ratio of within-cluster spread to separation between clusters; lower is generally better.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Domain judgment: Determines whether the clusters are useful, actionable, and stable.

If representative ground-truth labels are available, external measures such as the adjusted Rand index, normalized mutual information, homogeneity, and completeness can compare discovered clusters with those labels. These labels should be used for evaluation rather than fitting when the goal is genuinely unsupervised discovery.

Common Pitfalls

  • Skipping feature scaling: Distance-based clusters may reflect measurement units rather than meaningful similarity.

  • Ignoring algorithm assumptions: k-means favors compact clusters, while DBSCAN assumes density-connectivity and can fail when densities vary sharply.

  • Treating clusters as ground truth: Profile and interpret each cluster before using it.

  • Choosing parameters from one metric only: Combine statistical criteria, stability checks, and domain knowledge.

  • Over-tuning on one validation set: Assignments may change considerably with k, eps, linkage, or distance metric.

  • Leaking preprocessing information: When clustering is followed by held-out evaluation, fit preprocessing only on the training partition.

2

How does unsupervised cluster analysis differ from supervised learning?

Cluster analysis is an unsupervised learning task that groups unlabeled observations according to similarity rather than predicting a known target. It solves problems such as customer segmentation, document grouping, and exploratory data analysis, with the main trade-off being that discovered clusters are not automatically ground truth and require interpretation. Supervised learning instead uses labels to optimize a defined prediction objective, while conventional cluster analysis derives its structure without labels.

Read full answer

Applies tomachine learning concepts (all versions)

Supervised Versus Unsupervised Clustering

  • Unsupervised cluster analysis: The algorithm receives only features and seeks internal structure, often by minimizing within-cluster variation or identifying dense neighborhoods.

  • Supervised learning: The algorithm receives features and target labels, then learns a mapping from inputs to a known output, such as features → class.

  • Labels can still support clustering:

  • Use known labels afterward to evaluate clusters with metrics such as adjusted Rand index.

  • Profile clusters against business attributes to interpret them.

  • Convert cluster assignments into targets and train a supervised classifier on new observations.

  • Apply semi-supervised clustering when labels or constraints guide part of the grouping process.

  • Cluster identity is arbitrary: Cluster 0 in one run may correspond to cluster 1 in another. Evaluate groups rather than treating label numbers as meaningful classes.

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Noise handling Typical scalability
k-means Convex, roughly spherical groups Yes No explicit noise label O(n × k × d) per iteration; scales well
k-medoids Can be more flexible than k-means, but often remains compact Yes No explicit noise label Can require quadratic-in-n distance work; expensive
Agglomerative hierarchical Depends on linkage Can cut the tree later No explicit noise label Often O(n²) memory; can struggle on large data
DBSCAN Arbitrary, non-convex shapes; sensitive to varying density No Points with insufficient density become noise Good on medium data; worst case can approach O(n²)
HDBSCAN Variable-density and irregular clusters No Extracts noise and selects stable clusters Can handle varied density, but typically costs more than fixed-parameter DBSCAN
Gaussian mixture Elliptical, probabilistic clusters Usually yes No explicit noise label; components can absorb tails O(n × k × d) per EM iteration; moderate scalability
Spectral Highly non-convex shapes Yes No explicit noise label Dense graphs can require O(n²) memory; poor at large scale

Agglomerative clustering also depends on its linkage:

  • Single linkage: Can connect points through chains and find elongated groups.

  • Complete linkage: Favors compact clusters and reduces chaining.

  • Average linkage: Compromise between single and complete linkage.

  • Ward linkage: Minimizes increases in within-cluster sum of squares and favors compact, approximately spherical groups.

Distance, Scaling, and Cluster Validation

  • Euclidean distance measures straight-line separation and is sensitive to features with large numeric scales.

  • Manhattan distance measures coordinate-wise separation and can be less dominated by a single large deviation than Euclidean distance.

  • Cosine distance measures directional similarity and is often appropriate for text embeddings.

  • Standardize features when distance-based clustering combines measurements with different units. Tree models generally do not require scaling.

  • Choose the number of clusters using the elbow method, silhouette score, Davies-Bouldin index, domain knowledge, and—for Gaussian mixtures—BIC or AIC.

  • Silhouette scores are higher when clusters are compact and separated, but can favor a particular partition even when its real-world meaning is weak.

  • When reference labels exist, adjusted Rand index can externally evaluate cluster quality. Those labels must not be used during an unsupervised fit.

  • Missing values generally require an explicit strategy, such as imputation or excluding incomplete observations. If imputation is learned from data, fit it only on training data to prevent leakage.

  • Fix random seeds where an algorithm is stochastic and assess whether results remain stable across reasonable changes in data, parameters, and resampling splits.

Common Pitfalls

  • Forcing a cluster count: Statistical criteria should support, not replace, domain judgment.

  • Ignoring feature scale: Magnitude differences can make irrelevant features dominate distance calculations.

  • Treating clusters as ground truth: Cluster labels are hypotheses that must be profiled and validated.

  • Comparing DBSCAN’s silhouette incorrectly: Noise labels must be excluded, so its score is calculated on a different subset from k-means.

  • Using labels during fitting: This changes the problem into supervised or semi-supervised learning.

  • Overfitting model selection: Repeatedly choosing the clustering or hyperparameters that look best on one validation set produces optimistic results; use resampling and compare stability across splits.

  • Evaluating only one metric: Combine internal metrics, external labels when available, and real-world usefulness.

3

What is cluster analysis used for?

Cluster analysis groups unlabeled observations into segments based on similarity, helping discover structure in customer behavior, documents, images, sensor readings, and other data. It supports segmentation, compression, anomaly detection, grouping, recommendation, and exploratory analysis without predefined labels. Its main trade-off is that clusters are model-dependent interpretations rather than objective ground truth.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5+ for `sklearn.cluster.HDBSCAN` (Python 3.12)

Common Use Cases

  • Customer segmentation: Group customers by purchasing behavior, demographics, location, or engagement to support targeted marketing.

  • Document organization: Cluster articles, support tickets, or product descriptions to identify topics and related content.

  • Image and color quantization: Reduce the number of colors in images while preserving visually important regions.

  • Anomaly detection: Use density-based methods such as DBSCAN and HDBSCAN to surface observations outside dense groups for further investigation.

  • Bioinformatics and genomics: Group genes, cells, or samples according to expression patterns.

  • Geospatial analysis: Identify neighborhoods, service areas, or locations with similar spatial characteristics.

  • Recommendation systems: Cluster users or items to generate candidates when explicit similarity data are limited.

  • Data exploration: Identify possible subgroups before building a supervised predictive model.

Algorithm Comparison

Algorithm Cluster shapes found Must choose k? Handles noise? Scalability and considerations
k-means Isotropic, compact, roughly spherical clusters Yes No explicit noise handling Fast and scalable; approximately O(n * k * d) per iteration
k-medoids Compact clusters based on actual observations Yes No explicit noise handling More robust to outliers than k-means, but typically slower
Agglomerative hierarchical Nested clusters; shape depends on linkage Not initially; choose a dendrogram cut Not inherently Standard implementations can require O(n²) time and memory
DBSCAN Arbitrary, connected regions of high density No Yes, through noise labels Neighborhood search can become expensive; worst-case neighborhood comparisons are O(n²)
HDBSCAN Clusters with varying density and irregular shapes No Yes More flexible than DBSCAN and available as sklearn.cluster.HDBSCAN since scikit-learn 1.3
Gaussian mixture models Elliptical, probability-based clusters Usually Can identify low-confidence points, but not dedicated noise detection EM repeatedly recomputes component responsibilities
Spectral clustering Connected or non-convex structures Usually Not inherently Often requires a dense similarity matrix, resulting in high memory use

Agglomerative clustering supports several linkage choices:

  • Single linkage can create “chained” clusters.

  • Complete linkage favors compact clusters with smaller diameters.

  • Average linkage balances the previous two approaches.

  • Ward linkage merges clusters to minimize within-cluster variance and uses Euclidean distances.

Distances, Scaling, and Evaluation

Distance metrics determine what an algorithm considers similar:

  • Euclidean distance measures straight-line separation and is natural for k-means and Ward linkage.

  • Manhattan distance measures total coordinate-wise separation and is often useful for sparse, nonnegative representations, although it does not by itself solve high-dimensional distance concentration.

  • Cosine distance measures angular similarity and is useful for text embeddings because it emphasizes direction rather than vector magnitude.

Scaling matters whenever features use different units. A salary measured in thousands can dominate age measured in years during Euclidean-distance calculations. Standardization or min-max scaling should generally be fitted only on the data used to train the clustering pipeline. Trees are less sensitive to scale, while k-means, k-NN, SVM, PCA, and gradient-based methods usually need appropriately scaled inputs.

Common methods for choosing the number of clusters include:

  • Elbow method: Look for a bend in within-cluster sum-of-squares versus k.

  • Silhouette score: Rewards internally cohesive and well-separated clusters; its range is -1 to 1.

  • Davies-Bouldin index: Penalizes clusters with large variance or excessive overlap.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Domain judgment: Determine whether the resulting segments support a meaningful business or scientific decision.

When trusted labels are available, external metrics such as the adjusted Rand index can compare discovered clusters with those labels. Unsupervised metrics should not be treated as proof that the clusters are meaningful.

Common Pitfalls

  • Scaling incorrectly or fitting transformations on all data can distort distances and introduce evaluation leakage.

  • Choosing k only from a metric ignores domain meaning; silhouette scores can also favor compact shapes that do not match the true structure.

  • Ignoring noise labels from DBSCAN or HDBSCAN can distort cluster summaries and evaluation metrics.

  • Treating clusters as ground truth assumes the algorithm discovered the only valid segmentation. Profile each cluster using feature distributions, business measures, and stability across resamples.

  • Overfitting preprocessing or parameter selection to the test set invalidates the reported evaluation. Use cross-validation or a separate final test set when tuning many configurations.

  • Ignoring missing values and outliers can change distance-based results substantially.

  • Ignoring cluster-size imbalance can let large groups dominate centroid-based methods or leave small groups inadequately represented. Inspect cluster sizes and choose algorithms or sampling strategies appropriate to the density structure.

  • Losing reproducibility makes comparisons difficult. Fix random seeds where supported and record library versions, parameters, and preprocessing choices.

Common interview follow-ups include: Why does k-means fail on non-convex clusters? How are DBSCAN’s eps and min_samples selected? Why must scaling occur before clustering? How would you validate whether a customer segment is useful? What changes when labeled data are available?

4

How does cluster analysis segment data?

Cluster analysis is an unsupervised learning technique that groups observations with similar characteristics, producing segments such as customer types, product groups, or behavioral cohorts. It reveals structure without requiring labels, but the resulting segments depend on feature scaling, distance metrics, algorithm assumptions, and cluster-selection criteria, so they require validation and domain interpretation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Clustering algorithms receive a feature matrix and assign observations to groups. The groups are then profiled to understand their distinguishing characteristics and may support personalization, anomaly detection, recommendation, or targeted analysis.

  • Distance metrics: Euclidean distance measures straight-line separation and works well for continuous, scaled features. Manhattan distance sums absolute coordinate differences and can be useful for additive differences or sparse, high-dimensional data. Cosine distance compares vector direction rather than magnitude, making it useful for text embeddings.

  • Scaling: Standardization or normalization is important when features have different units because an unscaled variable can dominate the distance calculation. Scaling is especially important for k-means, k-NN, SVM, PCA, and gradient-based methods; it is generally unnecessary for tree-based models.

  • Cluster profiles: Analyze each cluster’s feature distributions and representative observations. Clusters are analytical constructs, not guaranteed classes or ground truth.

For Euclidean distance, d(x, y) = sqrt(sum((x_i - y_i)^2)), where x and y are observations, x_i and y_i are their feature values, and i indexes the features.

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Noise handling Scalability and typical complexity
k-means Spherical, convex clusters with similar sizes Yes No O(n * k * d) per iteration; scales well to moderate or large data
k-medoids Can capture non-convex groups; uses actual observations as centers Yes More outlier-resistant, but has no explicit noise label Similarity computation can require O(n² * d) per iteration
Agglomerative hierarchical Nested clusters controlled by linkage; single allows chaining, complete favors compact clusters, average balances both, and Ward minimizes within-cluster variance Optional; cut the dendrogram No explicit noise label Common implementations use O(n³) time and O(n²) memory
DBSCAN Arbitrary-shaped, connected density regions No Yes, using label -1 Roughly O(n log n) with indexed neighborhood queries; worst case O(n²)
HDBSCAN Hierarchical, arbitrary-shaped clusters across varying densities No Yes Tree-based approach generally scales better than pairwise DBSCAN; runtime depends on density and dimensionality
Gaussian mixture models Elliptical clusters with soft membership probabilities Yes No Approximately O(n * k * d) per EM iteration, plus covariance-estimation costs
Spectral clustering Complex, connected, or non-convex structures Yes No explicit mechanism Dense graphs require O(n²) memory and expensive eigenvalue computation; sparse approximations help
  • HDBSCAN in scikit-learn: sklearn.cluster.HDBSCAN has been available since scikit-learn 1.3 and supports variable-density clustering and explicit noise labels without requiring a single global eps value.

Choosing the Number of Clusters

  • Elbow method: Inspect the k-means inertia curve and look for a point after which reductions become marginal.

  • Silhouette score: Measures within-cluster similarity relative to separation from neighboring clusters. Higher values are better, but the score may be misleading for irregular shapes.

  • Davies-Bouldin index: Compares within-cluster dispersion with separation between clusters; lower values are better.

  • BIC and AIC: For Gaussian mixture models, balance model fit against complexity and select the candidate with the lower information criterion.

  • Domain judgment: The final choice should make operational sense. A technically optimal score does not guarantee useful segments.

  • External validation: When reliable labels exist, adjusted Rand index or normalized mutual information can measure agreement. Such labels should evaluate clustering, not define it.

Code Example

The example uses two interleaving moons. K-means partitions them by overall position, while DBSCAN follows their curved density structure.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.08, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

kmeans = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=20, random_state=42)),
])
kmeans_labels = kmeans.fit_predict(X_train)

dbscan = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", DBSCAN(eps=0.30, min_samples=5)),
])
dbscan_labels = dbscan.fit_predict(X_train)

# Evaluate each partition in the standardized feature space used for clustering.
kmeans_features = kmeans[:-1].transform(X_train)
dbscan_features = dbscan[:-1].transform(X_train)
non_noise = dbscan_labels != -1

print(
    f"K-Means: silhouette={silhouette_score(kmeans_features, kmeans_labels):.3f}, "
    f"ARI={adjusted_rand_score(y_train, kmeans_labels):.3f}"
)
print(
    f"DBSCAN: clusters={len(np.unique(dbscan_labels[non_noise]))}, "
    f"noise={np.sum(~non_noise)}, "
    f"silhouette={silhouette_score(dbscan_features[non_noise], dbscan_labels[non_noise]):.3f}, "
    f"ARI={adjusted_rand_score(y_train[non_noise], dbscan_labels[non_noise]):.3f}"
)

# K-Means can assign new observations; scikit-learn's DBSCAN has no predict method.
kmeans_test_labels = kmeans.predict(X_test)
print(
    f"Test K-Means assignments: {len(kmeans_test_labels)}, "
    f"ARI={adjusted_rand_score(y_test, kmeans_test_labels):.3f}"
)

Common Pitfalls

  • Data leakage: Fit scalers, imputers, and other learned preprocessing only on training data. The labels here are used only to evaluate agreement.

  • Wrong metric: Silhouette can favor compact clusters even when DBSCAN captures the geometry better. Combine numerical metrics, stability tests, and domain review.

  • Poor assumptions: K-means struggles with irregular clusters, while DBSCAN depends strongly on eps, min_samples, and density assumptions.

  • Ignoring missing values or outliers: Impute missing features using training-fitted logic and investigate extreme observations before clustering.

  • Overinterpreting clusters: Profile every segment, test stability under resampling, and avoid presenting clusters as inherent categories.

  • Interview follow-ups: Expect questions about choosing k, selecting linkage, validating clusters, handling high-dimensional data, and assigning new observations to an existing clustering model.

5

What challenges arise when clustering high-dimensional data?

High-dimensional clustering is the unsupervised grouping of observations when each observation has many features. Its main challenges are weak distance discrimination, the curse of dimensionality, irrelevant features, difficult cluster-shape assumptions, and high computational cost. The central trade-off is that reducing dimensions can improve tractability while potentially removing meaningful structure, and clusters produced by an algorithm are interpretations rather than ground truth.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Challenges

  • Distance concentration: In high dimensions, distances between many pairs of points become increasingly similar. Nearest and farthest neighbors may therefore be difficult to distinguish, weakening distance-based cluster boundaries.

  • Curse of dimensionality: Data becomes sparse, and most of the feature space is empty. Algorithms that assume compact, dense neighborhoods may struggle to find meaningful groups.

  • Irrelevant or redundant features: Noise dimensions dilute similarities, while correlated dimensions effectively receive extra weight. Feature selection or dimensionality reduction may help.

  • Non-convex cluster shapes: Spherical methods such as k-means can split curved or ring-shaped groups, whereas density-based methods such as DBSCAN and HDBSCAN handle such shapes better.

  • Outliers and noise: Sparse regions can distort centroids, medoids, covariance estimates, and hierarchical merges. Density-based methods explicitly identify some observations as noise.

  • Computational cost: Pairwise distances, hierarchical trees, graph construction, and spectral decomposition can become expensive as the number of samples and features grows.

  • Unstable interpretations: Small changes to features, distance metrics, preprocessing, random initialization, or the selected number of clusters can produce substantially different results.

Distance and Feature Quality

Euclidean distance is commonly written as:

d(x, y) = sqrt(sum_i((x_i - y_i)^2))

Here, x and y are two observations, x_i and y_i are their values for feature i, and the sum runs across all features. Manhattan distance, which sums absolute coordinate differences, can be more robust to individual large values. Cosine distance, often defined as 1 - cosine_similarity(x, y), measures directional rather than absolute similarity and is frequently used for text embeddings.

Feature scaling is important because distances otherwise reflect measurement units rather than importance:

  • Standardization using StandardScaler gives each feature zero mean and unit variance.

  • Min-max scaling maps features to a common range but is sensitive to outliers.

  • Robust scaling uses median and interquartile range when features contain heavy-tailed values.

  • Tree-based models generally need less scaling because their split decisions are invariant to monotonic feature transformations.

  • PCA can remove redundancy and reduce dimensions, but retained components may be difficult to interpret.

Algorithm Trade-offs

Algorithm Cluster shapes typically found Choose k? Noise handling Scalability and complexity
k-means Convex, compact, roughly spherical clusters Yes No explicit noise model Excellent scale; about O(n * k * d) per iteration
k-medoids Compact clusters centred on actual observations (medoids) Yes No explicit noise model Moderate; many implementations require O(n²) pair distances
Agglomerative hierarchical Depends on linkage: single permits chaining, complete favors compact groups, average is a compromise, and Ward favors spherical Euclidean clusters Usually by cutting the tree No explicit noise model Poor at large scale; commonly O(n²) memory and time
DBSCAN Arbitrary, density-connected shapes No Yes, using the -1 noise label Good with spatial indexing; worst case O(n²)
HDBSCAN Density hierarchy and irregular clusters No fixed k required Yes, using outlier detection Moderate to good; hierarchy adds overhead
Gaussian mixture Elliptical, potentially overlapping probability distributions Yes No explicit noise label Moderate; about O(n * k * d) per EM iteration
Spectral clustering Non-convex clusters connected through graphs Usually No explicit noise model Poor; typically O(n³) for eigendecomposition and O(n²) memory

HDBSCAN is available in scikit-learn 1.3+, while DBSCAN requires selecting parameters such as eps and min_samples.

Selecting and Validating Clusters

There is rarely one universally correct cluster count:

  • Elbow method: Look for the point where the within-cluster sum of squares stops decreasing rapidly. It primarily suits k-means-like objectives.

  • Silhouette score: Compares within-cluster similarity with separation from the nearest other cluster. It requires at least two clusters and can favor compact shapes.

  • Davies-Bouldin index: Measures the average similarity between a cluster and its most similar neighboring cluster; lower values indicate better separation.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Domain judgment: Operational meaning often matters more than an internal metric.

  • External validation: When trustworthy labels exist, adjusted Rand index, normalized mutual information, and homogeneity scores can evaluate agreement.

Cluster stability under resampling or perturbation can provide additional evidence. After selecting a solution, profile each cluster and verify that its characteristics have a meaningful interpretation.

Code Example

This example uses standardized training data to show why k-means struggles with curved clusters while DBSCAN identifies the two moons. The synthetic labels are used only for external evaluation and are never provided to either clustering algorithm.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

kmeans_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
kmeans_pipeline.fit(X_train)
kmeans_labels = kmeans_pipeline.named_steps["cluster"].predict(
    kmeans_pipeline.named_steps["scale"].transform(X_train)
)

dbscan_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.30, min_samples=5)),
])
dbscan_labels = dbscan_pipeline.fit_predict(X_train)

non_noise = dbscan_labels != -1

print("KMeans silhouette:", silhouette_score(X_train, kmeans_labels))
print("KMeans adjusted Rand index:",
      adjusted_rand_score(y_train, kmeans_labels))

print("DBSCAN clusters:", len(set(dbscan_labels)) - 1)
print("DBSCAN noise points:", np.sum(~non_noise))
print("DBSCAN non-noise silhouette:",
      silhouette_score(X_train[non_noise], dbscan_labels[non_noise]))
print("DBSCAN adjusted Rand index:",
      adjusted_rand_score(y_train[non_noise], dbscan_labels[non_noise]))

Common Pitfalls

  • Skipping scaling: Raw features with different units can make distance calculations meaningless.

  • Forcing a spherical solution: Use k-means only when approximately compact clusters are plausible.

  • Choosing k solely from a metric: Internal scores measure geometric fit, not business validity.

  • Treating clusters as ground truth: Cluster labels describe a partition under specific assumptions and parameters.

  • Ignoring leakage: Do not use test or validation observations when selecting preprocessing, dimensions, or clustering parameters.

  • Over-tuning on one split: Repeatedly changing k, eps, features, or algorithms until validation results look good can overfit the evaluation.

  • Interpreting visualization alone: PCA or UMAP plots are useful summaries, but their projection may distort the original high-dimensional geometry.

Common interview follow-ups include how to select k, why scaling changes clustering, how DBSCAN handles differently dense clusters, how HDBSCAN differs from DBSCAN, and how to prove that discovered clusters are meaningful.

6

Why are scaling and normalization important in cluster analysis?

Feature scaling and normalization transform variables to comparable ranges so that distance or density calculations reflect true similarity rather than unit differences. They solve the problem of high-magnitude features dominating cluster assignments in distance-based methods such as k-means, DBSCAN, and hierarchical clustering. The main trade-off is that scaling can reduce the influence of a feature whose natural magnitude is informative, and the choice of scaler affects cluster shapes and outlier sensitivity.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); machine learning concepts apply across versions.

Why Scaling Changes Clusters

Most clustering algorithms measure similarity with a distance metric. With Euclidean distance, d(x, z) = sqrt(sum((x_i - z_i)^2)), each x_i and z_i is a feature value, and large-range features contribute disproportionately to the sum. Standardization (z = (x - mean) / std) centers features at mean 0 with unit variance and is the default for many algorithms. Min-max normalization (x' = (x - min) / (max - min)) maps values to [0, 1] but is sensitive to outliers. For text embeddings, cosine distance (1 - (x · z) / (||x|| ||z||)) measures angle rather than magnitude, so L2 normalization is often more appropriate than standardization.

Clustering Algorithms Overview

Algorithm Cluster shapes Choose k? Noise handling Scalability Scaling needed?
k-means Spherical, similar size Yes Assigns all points Fast, O(n * k * d) per iteration Yes
k-medoids Spherical, robust center Yes Assigns all points Slower than k-means Yes (distance-based)
Hierarchical (agglomerative: single/complete/average/Ward) Varies by linkage Cut dendrogram Assigns all points O(n^2) memory/time typical Yes
DBSCAN Arbitrary density No (eps, min_samples) Labels noise (-1) Good with spatial index Yes
HDBSCAN Arbitrary density, varying density No (minimum cluster size) Labels noise Moderate to good Yes
Gaussian Mixture Model (GMM) Elliptical, probabilistic Yes (or BIC/AIC) No explicit noise label (soft assignments) Moderate Often helpful
Spectral clustering Complex manifolds Yes Usually assigns all Expensive (eigendecomposition) Yes (affinity-based)

k-means details are kept brief here; see K-Means Clustering for deeper coverage.

Choosing the Number of Clusters and Validation

  • Elbow method: plot inertia versus k and look for diminishing returns.

  • Silhouette score: measures cohesion vs separation in [-1, 1]; higher is better.

  • Davies-Bouldin index: lower indicates better-separated clusters.

  • BIC/AIC (GMMs): balance fit and model complexity.

  • Adjusted Rand Index (ARI): external metric when ground-truth labels exist; do not treat labels as the goal of unsupervised clustering.

  • Combine metrics with domain judgment and profile each cluster (summary statistics, representative examples). Clusters are patterns found by an algorithm, not ground truth.

Common Pitfalls

  • Not scaling distance-based models: k-NN, SVM, PCA, gradient descent, and most clustering algorithms need scaling; tree-based models usually do not.

  • Fitting scalers on all data before a supervised split: causes data leakage; use Pipeline/ColumnTransformer so preprocessing is fit on training folds only. For pure clustering, fit on the same data you cluster, but be consistent.

  • Using the wrong metric: Euclidean distance on sparse text counts can mislead; consider cosine distance or TF-IDF with normalization.

  • Treating inertia or one score as truth: compare several diagnostics and inspect cluster profiles.

  • Ignoring outliers: min-max scaling and k-means are sensitive; consider robust scaling or DBSCAN/HDBSCAN.

  • Interview follow-ups: “When would you avoid scaling?”, “How do you cluster mixed numeric and categorical data?”, and “How would you validate clusters without labels?”

7

How do you determine the number of clusters in a dataset?

Determining the number of clusters means selecting a partition that balances fit, separation, stability, and usefulness. More clusters usually reduce within-cluster error, but can produce small, unstable, or difficult-to-interpret groups. Because clustering is unsupervised, there is rarely one objectively correct value, so combine diagnostics with domain judgment.

Read full answer

Applies toscikit-learn 1.5+ with NumPy 2.x (Python 3.12)

Common Selection Methods

  • Elbow method: For k-means, plot the within-cluster sum of squares (inertia_) against k and look for diminishing returns. A gradual elbow makes the choice subjective.

  • Silhouette score: Compares each point’s within-cluster distance with its average distance to the nearest other cluster. Values range from -1 to 1, and higher is generally better, although the measure favors compact, separated clusters.

  • Davies-Bouldin index: Combines within-cluster scatter and separation between clusters; lower is generally better.

  • BIC and AIC for Gaussian mixtures: Compare candidate numbers of components while penalizing complexity. Lower values are preferable.

  • Stability analysis: Recluster bootstrapped or perturbed samples and check whether assignments remain consistent.

  • Domain judgment: Check whether each cluster represents a meaningful scientific, business, or operational segment.

With ground-truth labels, external measures such as the adjusted Rand index, normalized mutual information, or V-measure quantify agreement with those labels. They do not show whether the clusters are intrinsically useful.

Algorithm Comparison

Method Cluster shapes found Choosing the count Noise handling Scalability and limitations
K-means Compact, roughly spherical clusters Elbow, silhouette, domain judgment None O(n * k * d) per iteration; sensitive to scaling and outliers
K-medoids Clusters represented by actual observations Elbow, silhouette, domain judgment No explicit noise model Typically more expensive than k-means and still sensitive to outliers and distance choice
Hierarchical Single linkage permits chains; complete linkage favors compact groups; average linkage is intermediate; Ward favors minimum within-cluster variance Cut the dendrogram at k or a distance threshold None inherently Usually requires quadratic memory; most practical for small or medium datasets
DBSCAN Density-connected and potentially non-convex clusters No fixed k; tune eps and min_samples Explicitly labels noise Can use nearest-neighbor indexes but becomes expensive for high-dimensional data
HDBSCAN Hierarchical density clusters, including varying densities No fixed k; min_cluster_size affects granularity Identifies outliers More flexible than DBSCAN but generally more expensive; available as sklearn.cluster.HDBSCAN
Gaussian mixture Elliptical, probabilistic components BIC, AIC, likelihood, domain judgment No explicit background component by default Full-covariance EM is roughly O(n * k * d²) per iteration; simpler covariance types cost less
Spectral Non-convex groups connected through a graph or manifold Choose the number of components, sometimes using an eigengap heuristic Limited and parameter-dependent Dense eigendecomposition can be expensive; sparse-graph methods are usually more scalable

Code Example

This example selects k using training-set diagnostics, evaluates the selected k-means model once on held-out data, and compares k-means with DBSCAN on the same scaled training subset from curved “moons” data.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import silhouette_score
from sklearn.mixture import GaussianMixture
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Candidate cluster counts
X_blobs, _ = make_blobs(
    n_samples=300,
    centers=3,
    cluster_std=1.0,
    random_state=42,
)
X_train, X_test = train_test_split(
    X_blobs, test_size=0.25, random_state=42
)

print("k  train inertia  train silhouette")
scores = {}
for k in range(2, 7):
    pipeline = Pipeline([
        ("scale", StandardScaler()),
        ("cluster", KMeans(n_clusters=k, n_init=10, random_state=42)),
    ])
    train_labels = pipeline.fit_predict(X_train)
    scaled_train = pipeline.named_steps["scale"].transform(X_train)
    inertia = pipeline.named_steps["cluster"].inertia_
    silhouette = silhouette_score(scaled_train, train_labels)
    scores[k] = silhouette
    print(f"{k:2d} {inertia:13.3f} {silhouette:18.3f}")

# Select k without using the test set, then evaluate the final model once.
best_k = max(scores, key=scores.get)
final_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=best_k, n_init=10, random_state=42)),
])
final_pipeline.fit(X_train)
test_labels = final_pipeline.predict(X_test)
scaled_test = final_pipeline.named_steps["scale"].transform(X_test)

print(f"Selected k: {best_k}")
if np.unique(test_labels).size > 1:
    print(f"Final test silhouette: {silhouette_score(scaled_test, test_labels):.3f}")

# Compare information criteria for Gaussian mixtures.
print("\nGaussian mixture information criteria")
for k in range(1, 7):
    pipeline = Pipeline([
        ("scale", StandardScaler()),
        ("gmm", GaussianMixture(n_components=k, random_state=42)),
    ])
    pipeline.fit(X_train)
    scaled_train = pipeline.named_steps["scale"].transform(X_train)
    gmm = pipeline.named_steps["gmm"]
    print(f"k={k}, BIC={gmm.bic(scaled_train):.3f}, "
          f"AIC={gmm.aic(scaled_train):.3f}")

# Compare k-means and DBSCAN on the same training subset.
X_moons, _ = make_moons(
    n_samples=300,
    noise=0.08,
    random_state=42,
)
moon_train, _ = train_test_split(
    X_moons, test_size=0.25, random_state=42
)

kmeans_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
kmeans_labels = kmeans_pipeline.fit_predict(moon_train)

dbscan_pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.3, min_samples=5)),
])
dbscan_labels = dbscan_pipeline.fit_predict(moon_train)
moon_scaled = dbscan_pipeline.named_steps["scale"].transform(moon_train)

# DBSCAN labels -1 as noise; exclude those points from silhouette_score.
clustered = dbscan_labels != -1
cluster_count = np.unique(dbscan_labels[clustered]).size

print(f"\nK-means silhouette: "
      f"{silhouette_score(moon_scaled, kmeans_labels):.3f}")
print(f"DBSCAN clusters found: {cluster_count}")
print(f"DBSCAN noise points: {np.sum(~clustered)}")
if cluster_count > 1:
    print("DBSCAN silhouette: "
          f"{silhouette_score(moon_scaled[clustered], dbscan_labels[clustered]):.3f}")

Practical Considerations

  • Scaling: Standardize features for distance-based methods such as k-means, k-medoids, hierarchical clustering, and DBSCAN. Otherwise, large-range variables dominate the distances.

  • Distance metrics: Euclidean and Manhattan distances are scale-sensitive. Manhattan distance is less dominated than squared Euclidean distance by extreme coordinates, but it is not scale-invariant. Cosine distance is useful for text embeddings because it compares direction rather than magnitude.

  • Missing values and outliers: Impute missing values carefully. DBSCAN and HDBSCAN can isolate sparse observations as noise, whereas k-means centers can be pulled toward outliers.

  • Interpretation: Profile each cluster using feature summaries and relevant domain behavior. Clusters are not ground truth and do not establish causation.

  • Reproducibility: Set random seeds and compare results across seeds, resamples, and reasonable parameter variations.

Common Pitfalls

  • Selecting k from only one diagnostic, such as the elbow or silhouette score.

  • Scaling the full dataset before splitting it.

  • Selecting k repeatedly on the test set rather than reserving it for final evaluation.

  • Computing silhouette scores for DBSCAN without excluding noise labels.

  • Ignoring feature scaling, missing values, outliers, or cluster interpretability.

  • Claiming that discovered clusters are natural categories without profiling and validating them.

Interviewers commonly ask how sensitive the result is to scaling, initialization, missing values, outliers, alternative values of k, and the criteria used to validate the resulting segments.

8

How is the silhouette coefficient used to evaluate clustering?

The silhouette coefficient is a metric that measures how well each data point fits within its assigned cluster compared with the nearest neighboring cluster. It helps assess clustering quality and choose the number of clusters when no ground-truth labels are available. Its main trade-off is that it tends to favor compact, convex, well-separated clusters, so it can be misleading for elongated, nested, or density-varying shapes.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

For each sample i, the silhouette value s(i) is computed as:

s(i) = (b(i) - a(i)) / max(a(i), b(i))

  • a(i): the mean distance between sample i and all other samples in the same cluster. A small a(i) means the point is tightly grouped with its cluster.

  • b(i): the mean distance between sample i and all samples in the nearest neighboring cluster (the other cluster with the smallest mean distance to i, not its own cluster). A large b(i) means the point is far from the next-best cluster.

  • max(a(i), b(i)): the larger of the two mean distances, used to scale the result.

  • s(i): the silhouette value for sample i, ranging from -1 to 1. Values near 1 indicate good assignment, values near 0 indicate the point lies between clusters, and negative values suggest the point may belong to a neighboring cluster.

The overall silhouette score is the mean of s(i) across all samples. Per-sample values can be inspected with sklearn.metrics.silhouette_samples.

Interpreting and Using the Score

  • Choosing k: run clustering for several values of k and pick the k with the highest average silhouette score, while also checking cluster balance and domain meaning.

  • Comparing algorithms: compare scores only when the same distance metric, feature scaling, and data subset are used.

  • Diagnosing clusters: a plot of per-sample silhouette values can reveal clusters that are weak or overlapping.

Method Needs labels What it measures Best for Limitation
Silhouette No Cohesion vs separation per point Choosing k, comparing partitions Prefers convex, separated clusters
Davies-Bouldin No Average cluster similarity (lower is better) Fast internal comparison Also favors compact clusters
Elbow (inertia) No Within-cluster sum of squares vs k k-means-style models Elbow is often subjective
Adjusted Rand Index Yes Agreement with ground truth, adjusted for chance Benchmarking with known labels Not available in real unlabeled tasks

Code Example

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

# Reproducible synthetic data: two interleaving half circles
X, _ = make_moons(n_samples=300, noise=0.05, random_state=42)

# Feature scaling matters for distance-based clustering and silhouette scores
X_scaled = StandardScaler().fit_transform(X)

# k-means assumes roughly spherical clusters
kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans_labels = kmeans.fit_predict(X_scaled)
kmeans_sil = silhouette_score(X_scaled, kmeans_labels)

# DBSCAN can find arbitrary shapes and marks outliers as -1
dbscan = DBSCAN(eps=0.3, min_samples=5)
dbscan_labels = dbscan.fit_predict(X_scaled)

# silhouette_score needs at least 2 clusters; exclude noise points (-1)
mask = dbscan_labels != -1
dbscan_sil = silhouette_score(X_scaled[mask], dbscan_labels[mask])

n_clusters = len(set(dbscan_labels)) - (1 if -1 in dbscan_labels else 0)
n_noise = int(np.sum(dbscan_labels == -1))

print(f"k-means labels: {np.unique(kmeans_labels)}")      # [0 1]
print(f"DBSCAN labels found: {np.unique(dbscan_labels)}") # includes -1 for noise
print(f"DBSCAN clusters: {n_clusters}, noise points: {n_noise}")
print(f"k-means silhouette: {kmeans_sil:.3f}")            # value varies slightly by version/platform
print(f"DBSCAN silhouette (non-noise): {dbscan_sil:.3f}")  # value varies slightly by version/platform

Common Pitfalls

  • Ignoring feature scaling: silhouette uses distances, so unscaled features can dominate the score; scale numeric features for k-means, DBSCAN, and distance-based validation.

  • Treating a high score as truth: clustering has no universal ground truth. A high silhouette score can still produce clusters that are not meaningful for the business or scientific question.

  • Comparing incompatible runs: scores change with distance metric, preprocessing, and whether noise points are included. For DBSCAN/HDBSCAN, report how -1 noise points were handled.

  • Using it alone for non-convex shapes: on moons or rings, density-based methods may fit better even when a convex-cluster metric looks acceptable. Validate with plots and domain judgement.

  • Interview follow-ups: expect questions about the formula, why the range is -1 to 1, how silhouette differs from inertia, how to choose k, and when DBSCAN or HDBSCAN is preferred over k-means.

9

What is the difference between hard and soft clustering?

Hard clustering assigns each observation to exactly one cluster, while soft clustering gives an observation a membership degree for each cluster. Hard assignments are simple to communicate, whereas soft assignments preserve uncertainty when groups overlap. The main trade-off is operational simplicity and interpretability versus richer but potentially model-dependent membership information.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Hard and Soft Outputs

For observation (x_i), hard clustering produces a one-hot assignment (z{ik}), where exactly one component equals 1. Soft clustering produces membership values (gamma{ik}); probabilistic and many fuzzy methods constrain these values to sum to 1.

Aspect Hard clustering Soft clustering
Assignment One cluster per observation Membership value for each cluster
Output Cluster label Membership vector; probabilities in probabilistic models
Uncertainty Discarded Represented explicitly
Common example k-means, DBSCAN Gaussian mixture models
Main limitation Loses boundary ambiguity Depends on the validity of the membership model

A fitted GaussianMixture provides predict_proba(), whose rows are posterior responsibilities under its fitted generative model. predict() converts those responsibilities into the largest-probability hard label, discarding the other values.

Algorithm Context

Method Cluster shapes Choose k? Explicit noise? Main trade-off Typical scalability
k-means Convex, roughly spherical Yes No Fast but sensitive to scale, outliers, and initialization O(n * k * d) per iteration
k-medoids Dissimilarity-dependent; often compact Yes No More outlier-resistant than k-means, but slower O(n²) per PAM update
Agglomerative Linkage-dependent; single linkage can chain Yes No Gives a hierarchy but depends strongly on linkage O(n²) memory; naive O(n³) linkage
DBSCAN Irregular, density-connected groups No Yes Finds non-convex clusters but struggles with varying density Roughly O(n log n) to O(n²)
HDBSCAN Irregular groups with varying density No fixed k Yes Extracts stable clusters from a density hierarchy Often higher cost than DBSCAN
Gaussian mixture Elliptical, possibly overlapping clusters Yes No explicit label Gives soft responsibilities under Gaussian assumptions O(n * k * d²) per full-covariance EM iteration
Spectral Complex connected structures Usually No Captures non-convex geometry but depends on the graph O(n³) for a dense eigendecomposition

Ward linkage merges clusters while reducing within-cluster variance; complete linkage favors compact groups; average linkage balances the two; and single linkage can connect clusters through chains. sklearn.cluster.HDBSCAN has existed since scikit-learn 1.3.

Distances, Scaling, and Validation

  • Euclidean distance: Straight-line geometric distance; common for k-means, Ward linkage, and many k-medoids implementations.

  • Manhattan distance: Sum of absolute coordinate differences.

  • Cosine distance: Measures angular difference and is often suitable for text embeddings.

  • Density-based methods: DBSCAN and HDBSCAN use metric-defined neighborhoods and density reachability rather than a category called “connectivity distance.”

  • Scaling: Standardize or min-max-scale distance-based features when features have incompatible units. Scaling can change DBSCAN density structure, so validate the chosen scale and eps.

Choose k using domain judgment plus methods such as the elbow method, silhouette score, Davies-Bouldin index, or BIC/AIC for Gaussian mixtures. With meaningful reference labels, external metrics such as the adjusted Rand index can assess agreement. Profile each cluster’s size, feature distributions, center or medoid, and practical meaning; clusters are descriptive groups, not automatically ground truth.

Code Example

The generated moons labels are known for this controlled demonstration. The split keeps the scaler fitted only on training data; DBSCAN noise labels are excluded from silhouette and ARI calculations.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.mixture import GaussianMixture
from sklearn.neighbors import NearestNeighbors
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y
)

kmeans = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
dbscan = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.20, min_samples=5)),
])
gmm = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", GaussianMixture(
        n_components=2, n_init=10, random_state=42
    )),
])

kmeans.fit(X_train)
dbscan.fit(X_train)
gmm.fit(X_train)

kmeans_test_labels = kmeans.predict(X_test)
# DBSCAN has no predict(): give each test point the label of its nearest core sample within eps.
db = dbscan.named_steps["cluster"]
X_test_scaled = dbscan.named_steps["scale"].transform(X_test)
dist, idx = NearestNeighbors(n_neighbors=1).fit(db.components_).kneighbors(X_test_scaled)
dbscan_test_labels = np.where(
    dist[:, 0] <= db.eps, db.labels_[db.core_sample_indices_][idx[:, 0]], -1
)
gmm_hard_labels = gmm.predict(X_test)
gmm_soft_labels = gmm.predict_proba(X_test)


def silhouette_without_noise(features, labels):
    mask = labels != -1
    if np.unique(labels[mask]).size < 2:
        return float("nan")
    return silhouette_score(features[mask], labels[mask])


kmeans_train_labels = kmeans.named_steps["cluster"].labels_
dbscan_train_labels = dbscan.named_steps["cluster"].labels_
dbscan_mask = dbscan_test_labels != -1

dbscan_ari = (
    adjusted_rand_score(
        y_test[dbscan_mask], dbscan_test_labels[dbscan_mask]
    )
    if dbscan_mask.any()
    else float("nan")
)

print(f"KMeans ARI: {adjusted_rand_score(y_test, kmeans_test_labels):.3f}")
print(f"DBSCAN ARI: {dbscan_ari:.3f}")
print(f"DBSCAN classified fraction: {dbscan_mask.mean():.3f}")
print(
    "KMeans silhouette: "
    f"{silhouette_without_noise(X_train, kmeans_train_labels):.3f}"
)
print(
    "DBSCAN silhouette: "
    f"{silhouette_without_noise(X_train, dbscan_train_labels):.3f}"
)
print("GMM hard labels:", gmm_hard_labels[:10])
print("GMM soft memberships:\n", gmm_soft_labels[:3])
print("Membership rows sum to one:", np.allclose(gmm_soft_labels.sum(axis=1), 1.0))

Common Pitfalls

  • Treating GMM responsibilities as objectively true or perfectly calibrated probabilities.

  • Forgetting that converting soft assignments to hard labels discards uncertainty.

  • Assuming one internal validation metric determines the correct number of clusters.

  • Failing to scale features before distance-sensitive algorithms.

  • Using DBSCAN when clusters have strongly varying densities; HDBSCAN is often more suitable.

  • Comparing results with external metrics only when meaningful reference labels exist.

  • Treating clusters as ground truth instead of profiling and validating their practical meaning.

10

What is K-means clustering and what are its limitations?

K-means clustering is an unsupervised algorithm that divides data into k groups by minimizing the squared distance between each observation and its cluster’s center, or centroid. It is simple and computationally efficient, but its main trade-off is that it works best for compact, roughly spherical clusters and can perform poorly on irregular shapes, outliers, or poorly scaled features.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

K-means minimizes the within-cluster sum of squares:

objective = sum over clusters of sum over points of ||x - centroid||^2

Here, x is an observation, centroid is the mean of the points assigned to that cluster, ||x - centroid||^2 is the squared Euclidean distance, and the sums are calculated across all points and clusters.

The standard Lloyd algorithm works as follows:

  1. Choose the number of clusters, k.

  2. Initialize k centroids, commonly with K-means++, which selects initial centers that are relatively well separated.

  3. Assign each observation to its nearest centroid using Euclidean distance.

  4. Recalculate each centroid as the mean of the points assigned to it.

  5. Repeat the assignment and recalculation steps until assignments stabilize or the maximum number of iterations is reached.

The resulting centroids are means, so they are not necessarily actual data points. The algorithm seeks a local optimum rather than guaranteeing the global optimum. Random initialization and the starting points can therefore affect the solution.

Key Characteristics and Limitations

  • Assumes roughly spherical clusters: K-means tends to favor compact groups with similar sizes and densities. It may split a curved cluster into pieces.

  • Sensitive to feature scaling: A feature with a much larger numerical range can dominate the distance calculation. Scaling is important before K-means when features have different units or magnitudes.

  • Requires choosing k: The algorithm does not determine the number of clusters. The elbow method, silhouette score, Davies-Bouldin index, and domain knowledge are common selection tools.

  • Sensitive to outliers: Outliers can pull centroids toward themselves, distorting assignments.

  • Not suitable for arbitrary cluster shapes: Non-convex, ring-shaped, elongated, or interlocking clusters are difficult to represent with centroids.

  • Can struggle in high dimensions: Distances become less informative as dimensionality increases, and the curse of dimensionality affects both distance calculations and clustering quality.

  • Produces no uncertainty estimate: The result is a hard assignment; it does not directly express how confident the algorithm is in each point’s cluster membership.

  • Does not automatically identify noise: Unlike density-based methods, ordinary K-means forces every point into one of the k clusters.

Comparison with Other Clustering Methods

Method Cluster shapes typically found Need to choose k? Noise handling Scalability
K-means Compact, spherical, similar-sized clusters Yes No explicit noise model Excellent; approximately O(n * k * d) per iteration
K-medoids Compact clusters centred on actual observations; works with any distance metric Yes Not natively; robust variants can help Good for moderate-sized data; commonly uses PAM
Hierarchical Depends on linkage; can reveal nested structure Can cut the tree after fitting No native noise model Agglomerative methods generally require O(n²) memory
DBSCAN Arbitrary and non-convex clusters No Yes; points are labeled noise Good with spatial indexing; worst case can approach O(n²)
HDBSCAN Arbitrary clusters with varying density No Yes Usually costlier than DBSCAN; implementation-dependent memory use
Gaussian mixture Elliptical, probabilistic clusters Yes Not explicit; outliers can affect covariance estimates Good for moderate data sizes; approximately O(n * k * d) per iteration
Spectral Non-convex or manifold-like structures Usually yes No native noise model Limited by large similarity matrices, often O(n²) memory

For text embeddings, cosine distance is often more meaningful than Euclidean distance because it measures angular similarity rather than vector magnitude.

Code Example

The following example uses make_moons to illustrate a common failure case: K-means may divide the curved structure into compact groups, while DBSCAN can identify the two non-convex groups directly.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score

# Fixed synthetic data: two interleaving, curved clusters
X, _ = make_moons(
    n_samples=300,
    noise=0.08,
    random_state=42,
)

# K-means forces every point into one of two clusters
kmeans = KMeans(
    n_clusters=2,
    init="k-means++",
    n_init=10,
    random_state=42,
)
kmeans_labels = kmeans.fit_predict(X)
kmeans_silhouette = silhouette_score(X, kmeans_labels)

# DBSCAN identifies dense groups and marks sparse points as noise
dbscan = DBSCAN(
    eps=0.30,
    min_samples=5,
)
dbscan_labels = DBSCAN(
    eps=0.30,
    min_samples=5,
).fit_predict(X)

# Silhouette score cannot include noise points labeled -1
non_noise = dbscan_labels != -1
if np.unique(dbscan_labels[non_noise]).size > 1:
    dbscan_silhouette = silhouette_score(
        X[non_noise],
        dbscan_labels[non_noise],
    )
else:
    dbscan_silhouette = float("nan")

print(f"K-means clusters: {np.unique(kmeans_labels).size}")
print(f"K-means silhouette: {kmeans_silhouette:.3f}")
print(f"DBSCAN clusters: {np.unique(dbscan_labels[non_noise]).size}")
print(f"DBSCAN noise points: {np.sum(dbscan_labels == -1)}")
print(f"DBSCAN silhouette: {dbscan_silhouette:.3f}")

The silhouette score ranges from -1 to 1; higher values generally indicate better-separated clusters. It is useful for comparing candidates, but it should not be the only selection criterion. If true labels are available, external measures such as the adjusted Rand index can be used instead.

Common Pitfalls and Interview Tips

  • Do not forget to standardize features when their scales differ.

  • Do not interpret a K-means cluster as a ground-truth class; profile each cluster using domain features.

  • Do not select k only because it gives the lowest inertia, since inertia generally decreases as k increases.

  • Avoid using a validation or test set repeatedly for choosing k or tuning cluster parameters; that can overfit the evaluation.

  • Remember that clustering usually uses all available data because labels are not required, but downstream supervised evaluation should use an appropriate train/test split and fit preprocessing only on training data.

  • A useful follow-up question is: How would you choose k and validate that the clusters are meaningful?

11

What distinguishes hierarchical clustering from K-means?

Hierarchical clustering builds a hierarchy of nested clusters, producing a dendrogram that can be cut at different levels to obtain different cluster counts. K-means directly partitions observations into a fixed number of clusters by minimizing distance to cluster centroids. The main trade-off is that hierarchical clustering offers more structural insight but scales poorly, while K-means is fast and scalable but assumes roughly spherical, similarly sized clusters and is sensitive to initialization and outliers.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Core Differences

Aspect Hierarchical clustering K-means
Output Dendrogram containing nested clusters Flat set of k clusters
Cluster count Chosen afterward by cutting the dendrogram Must be specified before fitting
Cluster representation Merged observations and subclusters One centroid per cluster
Distance linkage Single, complete, average or Ward Primarily Euclidean distance
Cluster shapes More flexible, depending on linkage Tends toward compact, spherical clusters
Outliers No native noise label Can pull centroids toward outliers
Scalability Usually limited to small or medium datasets Scales well to large datasets
Initialization Not random; result is deterministic Uses random initial centroids
Interpretability Dendrogram reveals nested relationships Simple cluster centroids and profiles

A dendrogram is a tree visualization in which observations are leaves and each merge is shown at a linkage distance. Cutting the tree at a chosen height produces a flat clustering.

Hierarchical Linkage Choices

Agglomerative hierarchical clustering starts with each observation as its own cluster and repeatedly merges the closest pair.

  • Single linkage: Merges using the smallest distance between any two observations. It can connect clusters through chains and may produce elongated clusters.

  • Complete linkage: Uses the largest distance between observations in two clusters. It tends to produce compact, similar-sized clusters.

  • Average linkage: Uses the mean pairwise distance. It is a compromise between single and complete linkage.

  • Ward linkage: Merges clusters to minimize the increase in within-cluster squared error. It is designed for Euclidean distances and tends to produce spherical clusters.

K-means does not produce a hierarchy. Its objective can be written as:

K-means objective = sum over observations of squared distance to the assigned centroid

Each observation is assigned to its nearest centroid, and each centroid is updated to the mean of its assigned observations.

Broader Algorithm Comparison

Algorithm Cluster shapes found Choose k Explicit noise handling Typical scalability
K-means Compact, roughly spherical Required No O(n * k * d) per iteration
K-medoids Compact, centred on actual observations Required No explicit noise model Often expensive; PAM build is O(n²)
Agglomerative hierarchical Varies by linkage Optional after merging No native noise label Naive implementation can approach O(n³) and use O(n²) memory
DBSCAN Dense, irregular or crescent-shaped Not required; use eps and min_samples Yes Roughly O(n log n) with spatial indexing
HDBSCAN Hierarchical density-based clusters Not required; parameters control stability and minimum size Yes Usually costlier than DBSCAN; implementation-dependent
Gaussian mixture Elliptical clusters with probabilistic membership Required No explicit noise component O(n * k * d) per EM iteration
Spectral Connected or non-convex shapes Required No Dense affinity matrices require O(n²) memory

Choosing and Validating Clusters

  • Elbow method: Plot K-means within-cluster sum of squares against k and look for diminishing returns.

  • Silhouette score: Measures whether observations resemble their own cluster more than neighboring clusters. Higher values generally indicate better separation.

  • Davies-Bouldin index: Compares within-cluster dispersion with distances between cluster centers; lower is generally better.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Dendrogram inspection: For hierarchical clustering, examine merge distances and cut the tree based on domain requirements.

  • Domain judgment: Statistical scores cannot determine whether clusters are operationally meaningful.

  • External metrics: When ground-truth labels exist, adjusted Rand index, normalized mutual information and homogeneity scores can evaluate agreement.

Clusters are not ground truth. Always profile each cluster using feature means, distributions and representative observations.

Code Example

This example uses scaled training data from make_moons. K-means tends to split the curved groups, while DBSCAN identifies the two density-connected regions and marks low-density observations as noise.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Generate reproducible synthetic data.
X, _ = make_moons(n_samples=100, noise=0.08, random_state=42)

# Keep test data separate; fit preprocessing only on the training set.
X_train, X_test = train_test_split(
    X, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

# K-means partitions every training observation into one of two clusters.
kmeans = KMeans(n_clusters=2, random_state=42, n_init=10)
kmeans_labels = kmeans.fit_predict(X_train_scaled)

# DBSCAN identifies dense clusters and labels sparse observations as noise.
dbscan = DBSCAN(eps=0.30, min_samples=5)
dbscan_labels = dbscan.fit_predict(X_train_scaled)

dbscan_core = dbscan_labels != -1
dbscan_cluster_count = len(np.unique(dbscan_labels[dbscan_core]))
noise_count = np.sum(dbscan_labels == -1)

print("K-means cluster counts:", np.bincount(kmeans_labels))
print("DBSCAN cluster count:", dbscan_cluster_count)
print("DBSCAN noise count:", noise_count)
print("K-means silhouette:", silhouette_score(X_train_scaled, kmeans_labels))

if dbscan_cluster_count == 2:
    print(
        "DBSCAN silhouette:",
        silhouette_score(
            X_train_scaled[dbscan_core],
            dbscan_labels[dbscan_core],
        ),
    )

# Scores can vary slightly across scikit-learn versions.

The DBSCAN silhouette excludes noise observations, so it is not a perfectly like-for-like comparison with the K-means score.

Common Pitfalls

  • Not scaling features: K-means and hierarchical methods using Euclidean distance are affected by feature scale. Standardization or robust scaling is commonly appropriate.

  • Ignoring cosine distance: For text embeddings, cosine distance is often more meaningful than Euclidean distance.

  • Choosing k only by a metric: Always combine statistical criteria with domain knowledge.

  • Using DBSCAN poorly on high-dimensional data: Density becomes less meaningful as dimensionality increases; feature engineering or dimensionality reduction may be needed.

  • Assuming hierarchical clustering handles noise: Standard agglomerative clustering does not label outliers, unlike DBSCAN and HDBSCAN.

  • Overfitting the validation set: Repeatedly trying many cluster counts and selecting only the best result can overstate performance.

  • Ignoring class imbalance in validation: Use stratified splits when labels are available, even though clustering itself is unsupervised.

  • Treating clusters as ground truth: Interpret and validate cluster profiles rather than assuming the algorithm discovered natural categories.

12

How does the distance metric choice affect clustering results?

The distance metric determines when two observations are considered similar, so it directly controls which points are grouped together. Euclidean, Manhattan, cosine, and other metrics emphasize different geometric properties, potentially producing substantially different cluster shapes and assignments. The main trade-off is choosing a metric that reflects the data’s structure rather than relying on one that is merely convenient to compute.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Role of the Distance Metric

A clustering algorithm typically converts pairwise distances into neighborhoods, centroids, or graph connections. Changing the metric therefore changes the geometry on which the algorithm operates.

  • Euclidean distance: Measures straight-line separation. It works well for compact, roughly spherical clusters with similarly scaled features.

  • Manhattan distance: Measures coordinate-wise separation and can be more robust when large individual-coordinate differences would dominate Euclidean distance.

  • Cosine distance: Measures angular similarity rather than vector magnitude. It is useful for text embeddings, where two documents can have similar meaning despite different lengths.

  • Precomputed distances: Allow domain-specific or learned similarities, although they require additional memory and computation.

Feature scaling is crucial when features use different units. For example, a feature measured in thousands will dominate standardized features under Euclidean distance. Outliers also have an outsized effect on Euclidean distance. Scaling is generally unnecessary for tree-based models, but it commonly matters for distance-based clustering, k-NN, SVMs, PCA, and gradient-based methods.

Algorithm and Metric Effects

Algorithm How it groups points Choose k? Handles noise? Strengths, weaknesses, and typical use Typical scale
k-means Assigns points to the nearest centroid; favors compact, spherical clusters Yes No Fast and scalable, but struggles with elongated or nested shapes O(n * k * d) per iteration; large datasets
k-medoids Uses actual observations as cluster centers Yes Limited More robust to outliers than k-means, but slower and usually needs a precomputed distance matrix Medium-sized datasets; roughly O(n²) work per iteration
Agglomerative clustering Repeatedly merges the closest clusters Optional cut No Single linkage can chain points, complete linkage gives compact groups, average linkage balances both, and Ward favors variance reduction Small-to-medium data; O(n²) memory
DBSCAN Connects densely reachable neighborhoods No Yes Finds arbitrary shapes and identifies noise, but sensitive to eps, density, and scaling Medium-to-large data with spatial indexes; worst case O(n²)
HDBSCAN Builds a hierarchy of density-based clusters No Yes Handles variable-density clusters and noise; parameter-heavy Medium-to-large data; memory-intensive
Gaussian mixture model Models clusters as probability distributions Yes Limited Provides soft assignments and covariance models, but generally expects elliptical clusters Medium datasets; depends on covariance structure
Spectral clustering Clusters a graph derived from similarities Yes No Captures non-convex shapes, but expensive and sensitive to graph construction Small-to-medium data; dense eigensolver is often O(n³)

Choosing and Validating Clusters

The number of clusters should be evaluated rather than assumed:

  • Elbow method: Looks for a point where increasing k gives diminishing improvement.

  • Silhouette score: Measures whether points are closer to their own cluster than neighboring clusters; higher is generally better.

  • Davies-Bouldin index: Measures the ratio of within-cluster dispersion to separation between clusters; lower is generally better.

  • BIC/AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Domain judgement: Often the final arbiter; clusters should correspond to meaningful populations or behaviors.

  • Adjusted Rand index: Compares cluster assignments with known labels when labels are available.

Code Example

This example shows why K-means struggles with two interleaving moons while DBSCAN follows their nonlinear density. DBSCAN has no predict() for new points, so both models are fitted and scored on the training split; ordinary exploratory clustering often fits all available unlabeled data.

import numpy as np
from sklearn.datasets import make_moons
from sklearn.cluster import DBSCAN, KMeans
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Generate reproducible two-moons data.
X, y_true = make_moons(n_samples=300, noise=0.08, random_state=42)

X_train, X_test, y_true_train, y_true_test = train_test_split(
    X,
    y_true,
    test_size=0.25,
    random_state=42,
    stratify=y_true,
)

models = {
    "K-means": make_pipeline(
        StandardScaler(),
        KMeans(n_clusters=2, n_init=10, random_state=42),
    ),
    "DBSCAN": make_pipeline(
        StandardScaler(),
        DBSCAN(eps=0.3, min_samples=5, metric="euclidean"),
    ),
}

for name, model in models.items():
    # The pipeline fits the scaler only on training data.
    predicted = model.fit_predict(X_train)

    # Exclude DBSCAN noise points from silhouette calculation.
    valid = predicted != -1
    if len(np.unique(predicted[valid])) >= 2:
        silhouette = silhouette_score(X_train[valid], predicted[valid])
    else:
        silhouette = float("nan")

    # Synthetic labels are available only for this demonstration.
    ari = adjusted_rand_score(y_true_train, predicted)

    print(
        f"{name}: silhouette={silhouette:.3f}, "
        f"adjusted_rand_index={ari:.3f}"
    )

Common Pitfalls

  • Changing the metric without re-tuning eps: DBSCAN’s neighborhood radius is expressed in the selected metric’s units.

  • Ignoring feature scale: Large-range features can create misleading neighborhoods.

  • Assuming clusters are ground truth: Cluster labels describe a partition, not necessarily the only valid truth.

  • Selecting k from the test set: This turns validation into another training decision and can overfit the evaluation.

  • Ignoring outliers or missing values: Impute missing values explicitly or use an algorithm with a documented missing-value policy.

  • Overlooking stability: Compare solutions across resamples, parameter settings, or bootstrap samples, then profile each cluster’s feature distributions.

Common follow-up questions include: Why cosine similarity works well for text embeddings, when Ward linkage is inappropriate, how DBSCAN’s eps relates to density, and how cluster stability can be measured.

13

How does DBSCAN cluster data and detect noise?

DBSCAN groups densely connected points into clusters while labeling isolated low-density observations as noise, without requiring the number of clusters in advance. Its main trade-off is that it can recover irregular cluster shapes but is sensitive to eps, min_samples, feature scaling, and the distance metric.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How DBSCAN Works

DBSCAN uses two principal parameters:

  • eps: Maximum distance within which points are considered neighbors.

  • min_samples: Minimum number of points, including the point itself, required for a dense neighborhood.

Each point is classified as follows:

  • Core point: Its eps neighborhood contains at least min_samples points.

  • Border point: It is not core, but it is within eps of at least one core point.

  • Noise point: It is neither core nor connected to a core point through density-reachable points.

DBSCAN forms clusters from maximal groups of density-connected core points and attaches their border points. Remaining observations receive the label -1 in scikit-learn.

Parameters, Metrics, and Distance

Features should generally be scaled so that large-magnitude variables do not dominate the distance calculation. Euclidean distance is common for numerical data, while cosine distance can be appropriate for text embeddings. Missing values must be handled separately because scikit-learn’s DBSCAN does not accept them directly.

DBSCAN does not require choosing k, so the elbow method does not apply directly. Parameters can be assessed using domain knowledge, stability under small parameter changes, the silhouette score, and the Davies-Bouldin index. Silhouette is commonly calculated after excluding noise, although that selection can bias the comparison and should be documented. If reference labels exist, the adjusted Rand index or normalized mutual information can be used.

Algorithm Comparison

Algorithm Cluster shapes found Number of clusters Noise handling Scalability and complexity
K-means Convex, roughly spherical clusters Must choose k No explicit noise model O(n * k * d) per iteration; scales well
K-medoids More flexible than k-means Must choose k No separate noise label in the basic algorithm; threshold support is variant-specific PAM-style implementations are commonly O(n^2) per iteration
Agglomerative clustering Depends on linkage; single linkage can form elongated groups Must choose k or a cutoff No explicit noise model Naive implementations use O(n^2) memory
DBSCAN Irregular and non-convex clusters No k required Explicitly labels noise Often near O(n log n) with spatial indexing; worst case O(n^2)
HDBSCAN Hierarchical density-based clusters No k required Explicitly labels noise Typically costlier than DBSCAN because it builds a hierarchy
Gaussian mixture Ellipsoidal clusters with probabilistic membership Must choose k No explicit noise label; a broad component may absorb background points Full covariance: O(n * k * d^2) per EM iteration
Spectral clustering Complex graph-defined shapes Must choose k No explicit noise model Graph processing can require O(n^2) memory

Agglomerative clustering supports single, complete, average, and Ward linkage. Single linkage can connect observations through chains, complete linkage favors compact groups, average linkage combines both properties, and Ward linkage minimizes within-cluster variance under Euclidean distances.

Code Example

This example fits the scaler and clustering models on the training split. Scikit-learn’s DBSCAN has no predict method, so it cannot assign labels to held-out rows; silhouette is therefore computed descriptively on the fitted training sample.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.05, random_state=42)

# Labels are used only to stratify the split, not to fit either clusterer.
X_train, _, _, _ = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

dbscan = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", DBSCAN(eps=0.25, min_samples=5)),
])

kmeans = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])

dbscan_labels = dbscan.fit_predict(X_train)
kmeans_labels = kmeans.fit_predict(X_train)

cluster_ids = np.unique(dbscan_labels[dbscan_labels != -1])
noise_count = np.count_nonzero(dbscan_labels == -1)

print(f"DBSCAN clusters: {len(cluster_ids)}")
print(f"DBSCAN noise points: {noise_count}")
print(f"K-means clusters: {len(np.unique(kmeans_labels))}")

# Compare both methods on the same non-noise training observations.
non_noise = dbscan_labels != -1
if len(cluster_ids) >= 2:
    dbscan_silhouette = silhouette_score(
        X_train[non_noise], dbscan_labels[non_noise]
    )
    kmeans_silhouette = silhouette_score(
        X_train[non_noise], kmeans_labels[non_noise]
    )
    print(f"DBSCAN silhouette excluding noise: {dbscan_silhouette:.3f}")
    print(f"K-means silhouette on same observations: {kmeans_silhouette:.3f}")
else:
    print("Silhouette requires at least two DBSCAN clusters.")

Common Pitfalls

  • Ignoring feature scale: Large-magnitude features can dominate distance calculations.

  • Choosing eps blindly: Too small a value creates excessive noise; too large a value merges clusters.

  • Setting min_samples too low: Outliers may become core points and form spurious clusters.

  • Excluding or including noise without justification: Silhouette results can change depending on which observations are evaluated.

  • Leaking preprocessing information: Fit scalers and other learned preprocessing only on training data.

  • Treating clusters as ground truth: Profile each cluster and assess interpretability, stability, and domain usefulness.

14

How does Mean Shift find clusters, and when should you use it?

Mean Shift is a non-parametric, density-mode-seeking clustering algorithm that moves points toward dense regions without requiring the number of clusters in advance. It can recover irregularly shaped groups when the bandwidth matches their density structure. Its main trade-off is flexibility versus strong bandwidth sensitivity and high computational cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Mean Shift repeatedly updates a candidate point toward the local density maximum:

x_(t+1) = sum_i K(||x_i - x_t|| / h) * x_i / sum_i K(||x_i - x_t|| / h)

  • x_t is the current candidate and x_i is an observation.

  • ||x_i - x_t|| is their Euclidean distance.

  • h is the bandwidth defining the neighborhood size.

  • K is the kernel weight; scikit-learn’s MeanShift uses a flat kernel, so points within the bandwidth have weight 1 and others have weight 0.

Each update therefore becomes the mean of the points inside the bandwidth neighborhood. Updates continue until convergence or max_iter. By default, every point is a candidate seed; bin_seeding=True uses binned seeds for speed. Nearby final centers within cluster_centers_eps are merged.

When to Use It

Mean Shift is useful when:

  • The number of clusters is unknown.

  • Density peaks represent meaningful groups.

  • Irregular or curved groups should be detected.

  • The dataset is small or medium-sized for exploratory analysis.

It is less suitable for large datasets, high-dimensional data, strongly unequal feature scales, or settings where a faster method such as k-means is adequate. Its brute-force distance work is roughly O(n * s * d) per iteration, where s is the number of seeds, and can become quadratic when s grows with n.

Algorithm Comparison

Algorithm Typical cluster shapes Choose cluster count Noise handling Scalability
Mean Shift Density modes; irregular if bandwidth fits No No explicit labels; outliers may form modes Brute force; roughly O(n * s * d) per iteration
k-means Compact, spherical Yes None Good; O(n * k * d) per iteration
k-medoids Compact or moderately irregular Yes More robust than k-means, but no noise labels Moderate to expensive
Agglomerative hierarchical Linkage-dependent Threshold or k None Often quadratic pairwise work
DBSCAN Arbitrary shapes No Explicit noise labels sklearn implementation has worst-case O(n²) time and memory
HDBSCAN Arbitrary shapes and varying density No Explicit noise and membership strengths Good for medium-sized data; tree-based
Gaussian mixture Elliptical, probabilistic clusters Number of components None Full-covariance E-step is approximately O(n * k * d²)
Spectral clustering Non-convex or graph-based groups Usually yes None Limited; dense eigendecomposition can reach O(n³)

Agglomerative clustering supports single, complete, average, and Ward linkage. Single linkage can connect groups through chains, while Ward linkage requires Euclidean distances and favors compact clusters.

Distance and Parameter Selection

Scale features when their units or ranges differ; otherwise, a numerically large feature can dominate the Euclidean distances used by scikit-learn’s MeanShift. For text embeddings, cosine distance is often preferable because it measures directional similarity. L2-normalizing embeddings makes Euclidean distance proportional to cosine distance, but MeanShift does not directly accept a precomputed cosine-distance matrix.

Bandwidth is the key Mean Shift parameter: too large a value can merge groups, while too small a value can fragment one group into many modes. Select it through stability across resamples and inspection of cluster profiles.

For k-means, use the elbow method and silhouette or Davies-Bouldin scores. For Gaussian mixtures, use BIC or AIC to compare component counts, and always apply domain judgment. If true labels exist, external metrics such as adjusted Rand index can be reported, but labels do not replace interpretation. Scikit-learn 1.3+ also provides sklearn.cluster.HDBSCAN for density-varying data.

Code Example

import numpy as np
from sklearn.cluster import DBSCAN, KMeans, MeanShift
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split


def safe_silhouette(X, labels):
    # Exclude DBSCAN noise before computing its silhouette score.
    mask = labels != -1
    X_valid = X[mask]
    y_valid = labels[mask]
    n_labels = len(np.unique(y_valid))

    if n_labels < 2 or n_labels >= len(y_valid):
        return "not available"

    return round(float(silhouette_score(X_valid, y_valid)), 3)


# Mean Shift on separated blobs.
blobs_X, _ = make_blobs(
    n_samples=300,
    centers=[[-3.0, -3.0], [3.0, -3.0], [0.0, 3.0]],
    cluster_std=0.6,
    random_state=42,
)
blobs_train, blobs_test = train_test_split(
    blobs_X, test_size=0.2, random_state=42
)

mean_shift = MeanShift(
    bandwidth=1.0,
    bin_seeding=True,
    max_iter=300,
    cluster_centers_eps=0.1,
)
mean_shift.fit(blobs_train)

print("Mean Shift clusters:", len(mean_shift.cluster_centers_))
print("Mean Shift silhouette:", safe_silhouette(
    blobs_train, mean_shift.labels_
))
print("Mean Shift test assignments:", mean_shift.predict(blobs_test).shape[0])

# Compare k-means and DBSCAN on curved data.
moons_X, _ = make_moons(
    n_samples=300, noise=0.05, random_state=42
)
moons_train, _ = train_test_split(
    moons_X, test_size=0.2, random_state=42
)

kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans.fit(moons_train)

dbscan = DBSCAN(eps=0.20, min_samples=5)
dbscan.fit(moons_train)

print("K-means silhouette:", safe_silhouette(
    moons_train, kmeans.labels_
))
print("DBSCAN silhouette:", safe_silhouette(
    moons_train, dbscan.labels_
))
print("DBSCAN noise points:", int(np.sum(dbscan.labels_ == -1)))

With these settings, DBSCAN typically recovers the two curved groups, while k-means divides them with a roughly straight boundary. The DBSCAN silhouette excludes noise points, so it is not directly apples-to-apples with the k-means score.

Common Pitfalls

  • Ignoring feature scale: Scale mixed-unit features within training data or folds to prevent leakage.

  • Choosing one bandwidth blindly: Test several bandwidths and check cluster stability.

  • Allowing outliers to dominate: Mean Shift has no explicit noise model.

  • Treating clusters as ground truth: Profile each cluster against domain knowledge and meaningful features.

  • Optimizing only an internal metric: Silhouette measures separation, not usefulness or causality.

  • Tuning on held-out data: Keep final evaluation data untouched during parameter selection.

15

How does the Expectation-Maximization algorithm work in clustering?

Expectation-maximization (EM) alternates between assigning observations probabilistically to latent clusters and updating model parameters to maximize the expected likelihood. In clustering, it is the fitting procedure for a Gaussian mixture model and produces soft memberships; its main trade-off is flexible uncertainty modeling versus sensitivity to initialization, local optima, and additional computation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Let (z_i) be the unknown cluster of observation (x_i), and let (theta) represent the mixture parameters. EM alternates between:

  1. Expectation step: Compute each observation’s responsibility, or probability, for every cluster:

gamma_ik = pi_k * N(x_i | mu_k, Sigma_k) / sum_j(pi_j * N(x_i | mu_j, Sigma_j))

Here, x_i is observation i; pi_k is cluster k’s mixture weight; mu_k is its mean; Sigma_k is its covariance matrix; N is the Gaussian density; and j ranges over all clusters.

  1. Maximization step: Update the mixture weights, means, covariances, or other parameters by maximizing the expected complete-data log-likelihood under the current responsibilities.

For a Gaussian mixture, the observed-data log-likelihood is:

log L(theta) = sum_i log(sum_k(pi_k * N(x_i | mu_k, Sigma_k)))

The symbols have the meanings given above. EM should not decrease this likelihood, apart from numerical error, but it can converge to a local optimum.

In scikit-learn, GaussianMixture implements this process. Its covariance_type options are full, tied, diag, and spherical; they differ in whether component covariances are unrestricted, shared, diagonal, or a single scaled identity matrix.

Algorithm Comparison

Algorithm How it works and cluster shapes Strengths Weaknesses and noise Choose k? When to use Complexity or scale
k-means Assigns points to nearest centroid; favors compact spherical clusters Fast and scalable Voronoi boundaries; no native noise label Yes Large datasets with compact clusters O(n * k * d) per iteration
k-medoids Uses actual observations as centers; shape depends on the metric More robust to outliers than means Expensive; usually no noise label Yes Small or medium robust clustering tasks Often O(n²) distance storage
Agglomerative hierarchical Merges points according to linkage, then cuts the dendrogram Provides nested clusters O(n²) memory; no explicit noise Later Small datasets or nested group analysis Naive merging can be O(n³)
DBSCAN Connects density-reachable points; finds arbitrary shapes Finds irregular clusters and labels noise Sensitive to density parameters No Irregular clusters with noise Neighborhood queries dominate; quadratic worst case
HDBSCAN Builds a density hierarchy; supports variable-density clusters Handles irregular shapes and noise Parameter and memory overhead No fixed k Unknown density structure and outliers Moderate to expensive; available since scikit-learn 1.3
Gaussian mixture Fits probabilistic ellipses with EM Soft memberships and uncertainty Gaussian assumptions, local optima, no native noise label Yes Overlapping elliptical clusters Full-covariance work is roughly O(n * k * d²) per iteration
Spectral Embeds data using graph eigenvectors; finds non-convex shapes Captures manifold structure Expensive; no native noise label Usually Small datasets with complex boundaries Dense eigendecomposition can be O(n³)

Agglomerative clustering supports single, complete, average, and Ward linkage. Single linkage can chain, complete linkage favors compact clusters, average linkage is a compromise, and Ward minimizes within-cluster squared error but requires Euclidean distances.

Practical Guidance

Euclidean and Manhattan distances are common for numeric data, while cosine distance is useful for text embeddings. Standardize features before distance-based clustering so high-magnitude variables do not dominate; handle missing values with training-fitted imputation.

Choose k using the elbow method, silhouette score, Davies-Bouldin index, domain judgment, and—specifically for Gaussian mixtures—BIC or AIC from GaussianMixture.bic() and GaussianMixture.aic(). If reference labels exist, use an external metric such as adjusted Rand index. Profile and interpret every cluster; discovered clusters are not ground truth.

Code Example

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.mixture import GaussianMixture
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=300, noise=0.07, random_state=42)
X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=42
)

models = {
    "Gaussian mixture": Pipeline([
        ("scaler", StandardScaler()),
        ("cluster", GaussianMixture(
            n_components=2,
            covariance_type="full",
            n_init=10,
            random_state=42,
        )),
    ]),
    "K-means": Pipeline([
        ("scaler", StandardScaler()),
        ("cluster", KMeans(
            n_clusters=2,
            n_init=10,
            random_state=42,
        )),
    ]),
    "DBSCAN": Pipeline([
        ("scaler", StandardScaler()),
        ("cluster", DBSCAN(eps=0.30, min_samples=5)),
    ]),
}

for name, model in models.items():
    model.fit(X_train)
    scaler = model.named_steps["scaler"]
    estimator = model.named_steps["cluster"]
    train_scaled = scaler.transform(X_train)

    if isinstance(estimator, GaussianMixture):
        labels = estimator.predict(train_scaled)
    else:
        labels = estimator.labels_

    valid = labels != -1
    clusters = np.unique(labels[valid])
    score = None
    if clusters.size >= 2:
        score = silhouette_score(train_scaled[valid], labels[valid])

    print(name, "clusters:", clusters, "training silhouette:", score)

responsibilities = models["Gaussian mixture"].predict_proba(X_test)
print("Training shape:", X_train.shape)  # (210, 2)
print("Test shape:", X_test.shape)        # (90, 2)
print("First three GMM membership vectors:\n", responsibilities[:3])

On these moons, DBSCAN follows the curved cluster boundaries, while k-means splits them with an approximately linear boundary. Silhouette favours convex clusters, so k-means can still score higher even though DBSCAN recovers the true shapes. The GMM output shows each point’s probability of belonging to each component.

Common Pitfalls

  • Initialization and local optima: Use multiple initializations and inspect convergence.

  • Singular covariances: GaussianMixture.reg_covar adds diagonal regularization to component covariances.

  • Leakage: Fit scaling, imputation, and feature selection only on training data.

  • Mechanical cluster selection: Internal metrics are evidence, not proof of meaningful groups.

  • False certainty: Retain membership probabilities rather than treating hard labels as unquestionable.

  • Wrong assumptions: Gaussian mixtures work best for elliptical clusters and may model non-Gaussian structure poorly.

16

How do Gaussian Mixture Models contribute to cluster analysis?

A Gaussian Mixture Model (GMM) represents a dataset as a weighted mixture of Gaussian probability distributions. It contributes to cluster analysis by assigning observations probabilistically, making it well suited to overlapping, elliptical groups and uncertainty-aware membership decisions. Its main trade-off is flexibility versus stronger distributional assumptions and increased overfitting risk.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12) and general Gaussian mixture model theory

How It Works

A GMM models the overall data distribution as:

p(x) = sum from k=1 to K of π_k * N(x | μ_k, Σ_k)

Here:

  • x is an observation.

  • K is the number of Gaussian components.

  • π_k is the non-negative mixing weight of component k; all weights sum to one.

  • μ_k is the mean vector of component k.

  • Σ_k is the covariance matrix of component k.

  • N(x | μ_k, Σ_k) is the Gaussian probability density at x for that component.

The model is commonly fitted with the Expectation-Maximization (EM) algorithm:

  1. The E-step calculates each observation’s responsibility, or posterior probability, for every component.

  2. The M-step updates the mixing weights, means, and covariances.

  3. These steps repeat until the log-likelihood stops improving.

The responsibilities support soft assignments: an observation might have component probabilities of 0.95 and 0.05. Because EM can converge to different local optima, multiple initializations are normally used.

Contribution to Cluster Analysis

  • Uncertainty-aware membership: Unlike hard clustering, a GMM retains the probability that an observation belongs to each component.

  • Overlapping clusters: Components can overlap, allowing observations near a boundary to retain meaningful uncertainty.

  • Flexible cluster geometry: Full-covariance components can represent differently sized and oriented elliptical groups.

  • Model-based interpretation: A GMM estimates a probability distribution rather than only partitioning observations into groups.

  • Structure selection: Information criteria can compare candidate component counts and covariance structures, although statistical fit does not prove that the components are meaningful groups.

In scikit-learn 1.5+, GMM covariance options include spherical, diag, tied, and full. Full covariance is the most expressive but also estimates the most parameters.

Comparison with Other Methods

Method Cluster shapes found Choose k? Noise handling Typical scalability
k-means Compact, approximately spherical groups Yes No O(n * k * d) per iteration
k-medoids Compact groups whose geometry depends on the dissimilarity metric Yes No native label Often O(n²) distance computation
Agglomerative hierarchical Depends on linkage; Ward favors compact groups Can choose after merging No Typically O(n²) memory; time ranges from about O(n²) to O(n³) by implementation
DBSCAN Irregular, density-connected groups No; eps and min_samples matter Yes Often near O(n log n) with spatial indexing in favorable low-dimensional cases, but O(n²) worst case
HDBSCAN Groups across varying density levels No; min_cluster_size controls minimum size Yes Typically near O(n log n), with implementation-dependent constants
Gaussian Mixture Overlapping, oriented elliptical groups Yes, often through BIC or AIC No native noise label; a broad background component can be modeled Diagonal: about O(n * k * d) per E-step; full: about O(n * k * d²) plus covariance-update costs
Spectral Non-convex groups based on graph connectivity Yes No native label Dense eigendecomposition can require O(n³) time and O(n²) memory

Choosing, Scaling, and Interpreting Clusters

  • Elbow method: Inspects where increasing the number of groups produces diminishing improvement. For GMMs, BIC or AIC is usually more directly aligned with probabilistic model selection.

  • Silhouette score: Compares within-group and nearest-group separation for a partition; higher is better.

  • Davies-Bouldin index: Rewards groups with low internal spread and little overlap.

  • BIC or AIC: Compare fitted GMMs while penalizing model complexity.

  • Adjusted Rand index: Compares discovered groups with representative labels when external labels are available.

  • Domain judgment: A statistically preferred solution may not correspond to useful or actionable groups.

Scaling is essential for many distance-based methods. A full-covariance GMM can itself model variances and correlations, so standardization is not always theoretically required, but it can stabilize estimation and make regularization more consistent. Preprocessing must still be fitted only on training data during validation.

GMMs naturally correspond to Mahalanobis distance, while their usual representation assumes Euclidean coordinates. Cosine distance can be appropriate for text embeddings, although it usually requires transforming or reducing the representation before using a conventional GMM.

Common Pitfalls

  • Choosing too many components: Redundant components or collapsed covariance estimates can overfit; use BIC, AIC, and stability checks.

  • Ignoring covariance assumptions: Diagonal GMMs cannot model within-component correlations, while full GMMs need considerably more data.

  • Using one EM initialization: EM is non-convex and can produce different local solutions.

  • Treating probabilities as ground truth: High-probability assignments do not make the clusters scientifically or commercially meaningful.

  • Ignoring missing values and scale: Impute missing values appropriately and examine feature scaling.

  • Skipping interpretation: Profile each component’s characteristics, inspect uncertain observations, and test whether groups remain stable across resamples or data periods.

17

How should you preprocess data before performing cluster analysis?

Feature preprocessing makes the distance geometry used by a clustering algorithm reflect meaningful similarity instead of artifacts caused by units, missing values, outliers, or mixed data types. It improves cluster recovery, but overly aggressive transformations can suppress valid rare groups. The main trade-off is removing nuisance variation without imposing assumptions that distort the data.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Preprocessing Checklist

  1. Define the analysis unit

  2. Remove identifiers, constant columns, and variables collected after the clustering event.

  3. Remove only accidental duplicates. Aggregate legitimate repeated observations when customer-, session-, or transaction-level behavior matters.

  4. Handle missing values

  5. Most scikit-learn clustering estimators do not accept NaN values.

  6. Use median imputation for skewed numerical features and iterative or model-based imputation for complex relationships.

  7. Consider adding missingness indicators when absence itself has business meaning.

  8. Scale and transform features

  9. Apply StandardScaler when numerical features have different units or variances; use RobustScaler, logarithms, or power transformations when skewness or extreme values are problematic.

  10. Scale for distance-based clustering, k-NN, SVM, PCA, and gradient-based methods. Tree models generally do not require scaling.

  11. Euclidean distance measures straight-line separation, Manhattan distance sums absolute differences, Mahalanobis distance accounts for feature correlation, cosine distance compares direction and suits text embeddings, and Gower distance supports mixed numerical and categorical data.

  12. Encode categorical variables

  13. One-hot encode low-cardinality nominal features, but avoid very wide matrices for high-cardinality fields.

  14. Never assign arbitrary integer codes to unordered categories when those codes would be treated as continuous.

  15. Consider suitable embeddings for high-cardinality or text data.

  16. Control outliers and redundancy

  17. Investigate extreme values rather than automatically deleting them, since rare observations may form genuine clusters.

  18. Remove or combine highly redundant variables when they impair interpretation. Fit PCA or other dimensionality reduction only on training data.

Algorithm and Distance Considerations

Algorithm Cluster shapes found Choose k? Noise handling Typical scalability
k-means Compact, convex clusters Yes Poor; outliers pull centroids O(n * k * d) per iteration
k-medoids Flexible, potentially non-convex clusters Yes More robust than k-means Often O(n²) distance work
Agglomerative Nested clusters determined by linkage Choose or cut tree No explicit noise model Commonly O(n²) work and memory
DBSCAN Connected regions with varying density No Strong; marks noise as -1 O(n²) worst case; spatial indexing improves typical performance
HDBSCAN Hierarchical clusters with varying density No Strong Higher cost than DBSCAN; practical for moderate-to-large datasets
Gaussian mixture Elliptical, probabilistic clusters Yes; often use BIC Limited Spherical covariance is roughly O(n * k * d) per iteration; full covariance adds d² and d³ terms
Spectral Complex manifold or non-convex structures Yes No explicit noise model Dense formulations are often O(n³)

Agglomerative single linkage can chain points, complete linkage favors compact clusters, average balances the two, and Ward minimizes within-cluster variance for Euclidean distances. sklearn.cluster.HDBSCAN has existed since scikit-learn 1.3. Most clustering implementations are CPU-oriented; GPUs are generally more useful for generating embeddings than for the clustering step itself.

Choosing and Validating Clusters

  • Use the elbow method for k-means, the silhouette score, and the Davies–Bouldin index to compare candidate partitions.

  • Use BIC or AIC to select Gaussian components.

  • Apply domain judgement: a statistically separable partition is not automatically a useful segment.

  • With trusted labels, use permutation-invariant external measures such as adjusted Rand index or normalized mutual information.

  • Assess stability across resamples and profile each cluster’s size, feature distributions, and operational meaning. Clusters are latent partitions, not ground truth.

Code Example

import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Generating labels are used only for evaluation and fold balancing.
X, moon_id = make_moons(n_samples=500, noise=0.08, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, moon_id, test_size=0.2, stratify=moon_id, random_state=42
)

models = {
    "KMeans": Pipeline([
        ("scale", StandardScaler()),
        ("cluster", KMeans(n_clusters=2, n_init=20, random_state=42)),
    ]),
    "DBSCAN": Pipeline([
        ("scale", StandardScaler()),
        ("cluster", DBSCAN(eps=0.3, min_samples=5)),
    ]),
}

def clustering_silhouette(estimator, X, y):
    labels = estimator.fit_predict(X)
    keep = labels != -1
    if np.unique(labels[keep]).size < 2:
        return -1.0

    # Score in the same scaled space in which clustering was performed.
    X_scaled = estimator.named_steps["scale"].transform(X)
    return silhouette_score(X_scaled[keep], labels[keep])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for name, model in models.items():
    scores = cross_val_score(
        model, X_train, y_train, cv=cv,
        scoring=clustering_silhouette, n_jobs=-1
    )
    labels = model.fit_predict(X_train)

    print(f"{name} CV silhouette: {scores.mean():.3f}")
    print(f"{name} adjusted Rand index: "
          f"{adjusted_rand_score(y_train, labels):.3f}")

kmeans = models["KMeans"].fit(X_train)
test_labels = kmeans.predict(X_test)
print("KMeans test adjusted Rand index: "
      f"{adjusted_rand_score(y_test, test_labels):.3f}")

With these settings, DBSCAN should recover the two curved moon groups more successfully, while k-means tends to divide the data across the moons. Internal metrics, external measures when labels exist, and domain evidence should be considered together.

Common Pitfalls

  • Skipping scaling: large-unit features dominate distance calculations.

  • Fitting preprocessing outside a pipeline: validation-fold statistics leak into training.

  • Deleting every outlier or duplicate: valid rare groups or repeated behavior may be lost.

  • One-hot encoding high-cardinality fields: the resulting matrix may be impractically wide.

  • Selecting clusters using only one metric: combine elbow, silhouette, Davies–Bouldin, stability, and domain judgement.

  • Overfitting the validation set: repeatedly tuning cluster definitions on the same split biases the result.

  • Treating clusters as ground truth: document assumptions, profile clusters, and reproduce the analysis with fixed seeds.

18

How do you evaluate the stability of clustering results?

Cluster stability measures how consistently a clustering partitions observations across plausible changes such as resampling, feature perturbation, random initialization, or nearby hyperparameters. It reveals whether a discovered structure is reproducible rather than an artifact of one fit, but it does not by itself show that the clusters are meaningful. The main trade-off is realism versus disruption: perturbations must expose fragility without erasing the structure being studied.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Stability Testing Protocol

  1. Define realistic perturbations

  2. Subsample observations without replacement and compare only observations present in both runs. For bootstrap samples with duplicates, track original observation IDs and use a co-association consensus matrix.

  3. Add Gaussian or domain-specific noise in the same scaled feature space used by the clustering algorithm. For text embeddings, use cosine distance when appropriate.

  4. Predefine perturbation magnitudes so results are not chosen after inspecting scores.

  5. Control the modeling procedure

  6. Keep the algorithm, preprocessing, and selected k constant across perturbations. Fit scalers and other learned preprocessing only on training data.

  7. Select k using elbow analysis, silhouette score, Davies–Bouldin index, BIC/AIC for Gaussian mixtures, and domain knowledge.

  8. Treat sensitivity to parameters such as DBSCAN’s eps or min_samples as a separate experiment.

  9. Repeat and compare

  10. Refit the complete pipeline many times, varying random_state for algorithms with random initialization.

  11. Compare each partition with a reference partition using adjusted Rand index (ARI). ARI is invariant to cluster-label numbering and adjusts for chance agreement.

  12. Normalized mutual information is an information-theoretic alternative that also accepts hard cluster labels.

  13. Summarize stability

  14. For many runs, build a consensus matrix whose entries record how often each pair of observations was clustered together.

  15. Report the median and a lower percentile of agreement scores rather than only the mean.

  16. Keep stability, geometric quality such as silhouette, and external agreement with trusted labels as separate evaluations.

Algorithm Context

Algorithm Shapes and operation k / noise Strengths Weaknesses and typical use Typical complexity
K-means Voronoi cells around centroids; favors compact, similar-scale groups Requires k; no native noise Fast and scalable Sensitive to outliers and initialization; use for compact groups O(n * k * d) per iteration
K-medoids Representative medoids; non-convex shapes are possible Requires k; no native noise More robust to outliers Slower; useful when centroids are inappropriate Often O(n²) per iteration
Agglomerative Hierarchical merges using single, complete, average, or Ward linkage Optional k or threshold; no native noise Interpretable hierarchy Expensive for large datasets O(n²) time and memory
DBSCAN Density-connected regions with arbitrary shapes No k; identifies noise Handles non-convex clusters and outliers Sensitive to density parameters; useful for spatial data Can approach O(n log n) with efficient neighbor search; O(n²) worst case
HDBSCAN, available since scikit-learn 1.3 Hierarchical density clustering with varying cluster densities No k; identifies noise Flexible density structure Requires min_cluster_size; more computationally demanding Neighbor-search dependent; O(n²) worst case
Gaussian mixture Probabilistic assignments to Gaussian-shaped clusters Requires k; no explicit noise class Soft memberships and covariance modeling Assumes Gaussian structure; useful for probabilistic grouping About O(n * k * d²) per full-covariance EM iteration, plus O(k * d³) inversion
Spectral clustering Eigenvectors of a similarity graph; supports complex shapes Requires k; no native noise Useful for graph and manifold data Expensive eigendecomposition and matrix storage Typically O(n³) time and O(n²) memory

Code Example

This example measures DBSCAN stability on the training observations and demonstrates why k-means struggles with the two curved classes in make_moons. The generated labels are used only for this controlled geometry demonstration; they are not assumed to be real-world ground truth.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, StandardScaler

X, y = make_moons(n_samples=400, noise=0.08, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y
)

def dbscan_pipeline(eps=0.20):
    return Pipeline([
        ("scale", StandardScaler()),
        ("cluster", DBSCAN(eps=eps, min_samples=8)),
    ])

# Reference clustering and predictions on the held-out observations
reference_model = dbscan_pipeline()
reference_labels = reference_model.fit_predict(X_train)
dbscan_test_labels = reference_model.predict(X_test)

# Add noise after training-only scaling, then refit the clustering
rng = np.random.default_rng(42)

def add_scaled_noise(data):
    return data + rng.normal(loc=0.0, scale=0.06, size=data.shape)

ari_scores = []
for _ in range(20):
    perturbed_model = Pipeline([
        ("scale", StandardScaler()),
        ("noise", FunctionTransformer(add_scaled_noise)),
        ("cluster", DBSCAN(eps=0.20, min_samples=8)),
    ])
    perturbed_labels = perturbed_model.fit_predict(X_train)
    # ARI treats DBSCAN's -1 noise label as a separate category.
    ari_scores.append(
        adjusted_rand_score(reference_labels, perturbed_labels)
    )

print("Median perturbation ARI:", round(float(np.median(ari_scores)), 3))
print("10th percentile ARI:", round(float(np.quantile(ari_scores, 0.10)), 3))

# Compare k-means and DBSCAN on the curved data
kmeans_model = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
kmeans_model.fit(X_train)
kmeans_test_labels = kmeans_model.predict(X_test)

# External agreement for this synthetic demonstration only
print("K-means ARI:", round(float(adjusted_rand_score(y_test, kmeans_test_labels)), 3))
print("DBSCAN ARI:", round(float(adjusted_rand_score(y_test, dbscan_test_labels)), 3))

def report_silhouette(name, features, labels):
    keep = labels != -1
    clean_labels = labels[keep]
    n_clusters = np.unique(clean_labels).size
    if n_clusters >= 2 and clean_labels.size > n_clusters:
        score = silhouette_score(features[keep], clean_labels)
        print(f"{name} silhouette:", round(float(score), 3))

report_silhouette("K-means", X_test, kmeans_test_labels)
report_silhouette("DBSCAN", X_test, dbscan_test_labels)

Common Pitfalls

  • Using silhouette as a stability metric: it measures compactness and separation within one dataset, not persistence across changes.

  • Comparing mismatched samples with ARI: use common observation IDs or a consensus matrix for resampled data.

  • Leakage or repeated tuning: fit preprocessing on training data and define the stability protocol before inspecting results.

  • Excessive disruption: noise that overwhelms the structure makes every method appear unstable.

  • Confusing stability with meaning: stable clusters still require feature profiling and domain interpretation; they are not automatically ground truth.

19

Which feature selection techniques are suitable for cluster analysis?

Feature selection for cluster analysis identifies the variables that best preserve the geometry used to form clusters. It reduces noise, redundancy, and computation but may discard rare yet meaningful dimensions. Because clustering is usually unsupervised, selection should rely on variance, redundancy, cluster stability, and domain knowledge rather than target labels.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Selection Strategies

  • Variance filtering: Remove near-constant features because they contribute little to distances. Apply this before standardization, which would otherwise give every nonconstant feature approximately the same variance.

  • Correlation filtering: Remove highly correlated features to reduce redundancy. A threshold such as |r| > 0.95 is conservative, but the appropriate value depends on the domain; correlated measurements may still represent distinct concepts.

  • Wrapper methods: Search for a feature subset that improves an unsupervised objective such as silhouette score, Davies-Bouldin index, or clustering stability. These methods can be computationally expensive and can overfit the evaluation metric.

  • Embedded methods: Sparse clustering, sparse PCA, or models with regularization can combine dimensionality reduction with feature selection. Their results may be harder to interpret than those from simple filters.

  • Stability selection: Resample the data and retain features that consistently affect cluster assignments. Stability is often more reliable than one optimized clustering result.

  • Supervised selection with labels: When labels exist, methods such as mutual information or model-based importance can be used, but they may select features that predict the labels rather than the desired cluster geometry.

  • PCA: PCA is dimensionality reduction rather than feature selection because each component is a combination of original features. It can still improve conditioning and remove redundancy, at the cost of interpretability.

Distances, Scaling, and Missing Values

  • Euclidean distance measures straight-line separation and is sensitive to features with large scales or extreme values.

  • Manhattan distance reduces the quadratic influence of a large coordinate difference compared with Euclidean distance, although it remains sensitive to outliers.

  • Cosine distance is appropriate for text embeddings when direction matters more than magnitude.

  • Correlation distance can be useful for clustering profiles, depending on whether level or pattern similarity matters.

  • Scaling is generally needed for k-means, k-medoids, Ward hierarchical clustering, DBSCAN, HDBSCAN, and spectral clustering when features have different units. It can also change the probabilistic interpretation of Gaussian mixture models.

  • Missing values should usually be imputed within the training pipeline. Imputation and feature selection must not use validation or test data.

Algorithm and Validation Choices

Algorithm Cluster shapes found Choose k? Noise handling Main trade-off and scale
k-means Convex, roughly equal-variance clusters Yes No Fast, approximately O(n*k*d*t) per run; sensitive to outliers and initialization
k-medoids Roughly compact clusters around real observations Yes No More robust than k-means, but optimization is expensive
Agglomerative Depends on linkage: single can chain, complete stays compact, average balances both, and Ward minimizes within-cluster variance Cut the dendrogram No Useful for small-to-medium data; typical linkage storage is O(n²)
DBSCAN Arbitrary-shaped, density-connected clusters No Yes Captures nonconvex clusters but depends on eps and density
HDBSCAN Flexible, variable-density clusters No Yes Handles varied density without global eps; generally medium-scale
Gaussian mixture Elliptical clusters with probabilistic membership Yes No Gives probabilities but assumes Gaussian components; moderate-scale
Spectral Nonconvex clusters defined by a graph Usually yes No Powerful for complex shapes, but typically requires O(n²) memory

Select the number of clusters using the elbow method for k-means, silhouette score, Davies-Bouldin index, and domain judgment. For Gaussian mixture models, compare BIC or AIC, where lower values are preferred. If trustworthy labels are available, use external measures such as the adjusted Rand index.

Common Pitfalls

  • Leakage: Fit selection, imputation, and scaling on training data only.

  • Metric overfitting: Repeatedly selecting features using the same silhouette score makes that score optimistic; reserve a final holdout set.

  • Ignoring interpretation: Profile each cluster’s feature distributions and distinguish meaningful subgroups from algorithmic artifacts.

  • Treating clusters as truth: Clusters are hypotheses produced by a model, not automatically ground-truth categories.

  • Discarding rare features: A low-variance feature may encode an important minority pattern, so combine statistical filtering with domain review.

Interviewers commonly ask how k is selected, why scaling changes DBSCAN’s neighborhoods, how stability is measured, and how one prevents feature selection from overfitting the clustering metric.

20

What strategies handle missing values before clustering?

Missing-value treatment prepares data for clustering so that distance calculations, centroids, and density estimates are not distorted. Common strategies include deleting incomplete rows, statistical imputation, model-based imputation, and missingness indicators. The main trade-off is preserving sample size versus introducing artificial structure or bias, so the mechanism and meaning of the missing values should guide the choice.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Handling Options

Strategy How it works Advantages Limitations and when to use it
Delete incomplete rows Remove rows containing missing values Simple and avoids imputation artifacts Reduces sample size and can bias results if missingness is not random
Mean or median imputation Replace missing values with a feature-level statistic Fast and preserves all rows May compress clusters or create artificial relationships; median is more robust to outliers
K-nearest-neighbors imputation Predict each missing value from similar observations Captures local multivariate patterns More computationally expensive and sensitive to feature scaling and missing neighbors
Iterative or model-based imputation Estimate missing values using regression or another learned model Can represent complex relationships Risks overfitting and leakage if fitted outside each training fold
Missingness indicator Add binary columns identifying missing features Preserves information about the missingness pattern Increases dimensionality and may or may not help clustering
Native missing-value support Use a clustering implementation that explicitly accepts missing values Avoids artificial imputation Support and interpretation vary; verify the implementation’s documented behavior

For most standard scikit-learn clustering algorithms, missing values must be handled through imputation or deletion before clustering.

Clustering-Specific Considerations

  • Feature scaling: Standardize features before distance-based methods such as k-means, k-medoids, and agglomerative clustering with Euclidean linkage. Ward linkage only supports Euclidean distance, while PCA also requires centered, appropriately scaled features.

  • Distance metrics: Euclidean distance favors compact, roughly spherical clusters. Manhattan distance can reduce the influence of individual large coordinate differences relative to Euclidean distance, while cosine distance is useful for text embeddings where direction matters more than magnitude.

  • Imputation consistency: Reuse the same fitted preprocessing pipeline for training and deployment data. In train/validation or resampling workflows, fit the imputer and scaler only on the training portion.

  • Outliers and unknowns: Investigate extreme values separately. A missingness indicator can distinguish an unknown value from an observed value, but robust imputation only supplies a robust estimate and does not preserve that distinction.

  • Missingness mechanism: Complete-case deletion is most defensible when missingness is plausibly independent of the observed information; otherwise, imputation and sensitivity analysis may be safer.

Clustering algorithms differ in the structures they identify and in their scalability:

Algorithm Cluster shapes found Must choose k? Handles noise Typical scalability
k-means Convex, roughly spherical clusters Yes No Good; approximately O(n * k * d) per iteration
k-medoids Compact regions around medoids; geometry depends on metric Yes No Often O(n²)
Agglomerative clustering Varies with single, complete, average, or Ward linkage Yes Generally no Usually O(n²)
DBSCAN Arbitrary shapes and density variation No Yes Good with spatial indexing; worst case O(n²)
HDBSCAN Flexible, hierarchical density-based clusters No Yes Good for moderately large datasets
Gaussian mixture models Elliptical, Gaussian-shaped clusters Yes No Moderate; per-iteration cost is similar to k-means
Spectral clustering Non-convex and graph-based structures Yes No Limited because it generally requires an n × n graph

Choosing and Validating Clusters

Use the elbow method, silhouette score, and Davies-Bouldin index as complementary diagnostics. For Gaussian mixture models, BIC and AIC can help select the number of components. Domain knowledge remains essential because these metrics do not establish that the clusters are meaningful.

When trusted labels are available, external measures such as the adjusted Rand index can compare the discovered clusters with those labels. Profile each cluster by feature distributions and domain characteristics, and assess stability across resamples or model configurations. Clusters are unsupervised groupings, not ground truth.

Best Practices

  • Investigate missingness by feature and relevant subgroup before choosing an approach.

  • Compare complete-case, median-imputed, and more advanced imputation results.

  • In cross-validation, fit preprocessing within each training fold using a Pipeline.

  • Test sensitivity to outliers, feature scaling, the imputation strategy, and the assumed number of clusters.

  • Use robust scaling or transform highly skewed features when appropriate.

Common Pitfalls

  • Data leakage: Computing imputation statistics on the full dataset exposes validation or test information to training.

  • Unrealistic imputation: Filling missing values with zeros can pull points toward the origin and manufacture clusters.

  • Ignoring the missingness mechanism: Treating every missing value as ignorable can discard meaningful information or bias cluster membership.

  • Overinterpreting silhouette scores: A high score does not prove that clusters are useful or stable.

  • Treating clusters as labels: Cluster identifiers have no inherent meaning until their members are interpreted.

21

How do you analyze and validate clustering results?

Post-clustering analysis evaluates and interprets the groups produced by a clustering algorithm. Common methods include cluster profiling, internal and external validation, stability testing, visualization, and downstream analysis. The main trade-off is that no single metric proves that clusters are meaningful, so quantitative evidence should be combined with domain knowledge.

Read full answer

Applies toscikit-learn 1.5+ and SciPy 1.13+ (Python 3.12)

Post-Clustering Methods

  • Cluster profiling: Compare cluster sizes and summarize numeric features using means, medians, ranges, or standardized deviations; summarize categorical features using category proportions.

  • Internal validation: Measure cohesion and separation.

  • Silhouette score: silhouette = (b - a) / max(a, b), where a is the mean distance to the sample’s own cluster and b is the mean distance to its nearest other cluster. Higher is generally better.

  • Davies-Bouldin index: Lower is better.

  • Calinski-Harabasz index: Higher is generally better.

  • External validation: When reliable labels exist, compare partitions using the adjusted Rand index, normalized mutual information, purity, or matched-label accuracy.

  • Stability analysis: Resample or perturb the data, rerun the algorithm, and compare the resulting partitions. Stable clusters are more likely to represent reproducible structure.

  • Visualization and noise analysis: Use PCA for global structure and t-SNE or UMAP for neighborhood visualization. Separately inspect DBSCAN and HDBSCAN noise points.

  • Downstream evaluation: Determine whether membership supports recommendations, customer segmentation, anomaly detection, or another actionable task.

Algorithm and Validation Context

Algorithm Cluster shapes found Choose k? Noise handling Scalability and notes
k-means Convex, roughly spherical clusters Yes No Fast, typically O(n * k * d) per iteration; sensitive to scale and outliers
k-medoids Irregular shapes are possible but not guaranteed Yes No PAM-style implementations commonly require roughly O(n²) distance work; metric support varies
Agglomerative clustering Depends on linkage: single handles chains, complete favors compact clusters, average balances them, and Ward favors spherical clusters Yes No Many linkages require roughly O(n²) pairwise work
DBSCAN Arbitrary shapes based on density connectivity No; choose eps and min_samples Yes Often efficient with spatial indexes, but worst-case neighborhood queries can be costly
HDBSCAN Density hierarchy and irregular clusters No Yes sklearn.cluster.HDBSCAN exists since 1.3; neighborhood-search cost depends on density and data scale
Gaussian mixture Elliptical, probabilistic clusters Yes Low-probability points can be flagged as outliers Each EM iteration is roughly O(n * k * d) plus covariance work; supports BIC/AIC
Spectral clustering Non-convex graph-based groups Yes No explicit noise Can find complex shapes but commonly requires O(n²) memory

For candidate values of k, compare the elbow method, silhouette score, Davies-Bouldin index, BIC or AIC for Gaussian mixtures, and domain requirements. Clusters are model-generated partitions, not ground truth.

Distance, Scaling, and Interpretation

  • Euclidean distance measures straight-line separation and is sensitive to feature scale and outliers.

  • Manhattan distance sums coordinate-wise differences and can behave differently from Euclidean distance in sparse or high-dimensional data.

  • Cosine distance compares embedding direction and is commonly used for text embeddings.

  • Scaling: Standardize or robustly scale features before k-means, k-medoids, DBSCAN, HDBSCAN, PCA, SVMs, or other distance-based algorithms. Trees generally do not require scaling.

  • Interpretation: Compare each cluster’s profile with the overall population, name meaningful groups, and validate those names with stakeholders.

Code Example

This example fits the algorithms on training data and evaluates them on held-out data. It shows k-means struggling with curved make_moons data while DBSCAN identifies the two shapes.

import numpy as np
from scipy.optimize import linear_sum_assignment
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import contingency_matrix, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=300, noise=0.05, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

# Each pipeline fits its scaler and clustering model on training data only.
kmeans = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init="auto", random_state=42)),
])

dbscan = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", DBSCAN(eps=0.30, min_samples=5)),
])

kmeans.fit(X_train)
dbscan.fit(X_train)

kmeans_labels = kmeans.predict(X_test)
dbscan_labels = dbscan.predict(X_test)


def silhouette_without_noise(features, labels):
    mask = labels != -1
    cluster_count = np.unique(labels[mask]).size
    if cluster_count < 2 or cluster_count >= len(features):
        return float("nan"), mask
    return silhouette_score(features[mask], labels[mask]), mask


def best_match_accuracy(reference, predicted):
    if len(reference) == 0:
        return float("nan")
    matrix = contingency_matrix(reference, predicted)
    rows, columns = linear_sum_assignment(-matrix)
    return matrix[rows, columns].sum() / len(reference)


kmeans_silhouette, _ = silhouette_without_noise(X_test, kmeans_labels)
dbscan_silhouette, non_noise = silhouette_without_noise(X_test, dbscan_labels)

print(f"K-means silhouette: {kmeans_silhouette:.3f}")
print(f"DBSCAN silhouette: {dbscan_silhouette:.3f}")
print(
    "K-means best-match accuracy: "
    f"{best_match_accuracy(y_test, kmeans_labels):.1%}"
)
print(
    "DBSCAN best-match accuracy: "
    f"{best_match_accuracy(y_test[non_noise], dbscan_labels[non_noise]):.1%}"
)
print(f"DBSCAN noise fraction: {(~non_noise).mean():.1%}")

Common Pitfalls

  • Treating clustering assignments as verified ground-truth labels.

  • Fitting a scaler or clustering model on the full dataset before held-out evaluation.

  • Selecting one metric or one value of k without domain validation.

  • Overinterpreting t-SNE or UMAP layouts as evidence of natural boundaries.

  • Ignoring noise points or excluding them when reporting DBSCAN silhouette scores.

  • Producing mathematically distinct clusters without profiling or interpreting them.

22

What are cluster validation techniques in cluster analysis?

Cluster validation evaluates whether a clustering algorithm produces meaningful, stable, and useful groups. Internal measures assess separation without labels, external measures compare the result with reference labels, and stability tests examine whether clusters survive resampling. The main trade-off is that internal validation is practical but depends on geometric assumptions, while external validation requires labels that may not represent the true structure or business meaning.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Types of Cluster Validation

  • Internal validation: Measures separation using only the observations and their cluster assignments.

  • Silhouette score: Ranges from -1 to 1; higher values generally indicate better-defined clusters.

  • Davies–Bouldin index: Lower values indicate better-separated, less dispersed clusters.

  • Calinski–Harabasz index: Higher values generally indicate better-separated clusters, although the measure assumes cluster geometry that may not fit the data.

  • Elbow method: Looks for a point where increasing the number of clusters no longer substantially reduces within-cluster variation.

  • External validation: Compares clusters with known or externally supplied labels.

  • Common measures include adjusted Rand index, normalized mutual information, homogeneity, completeness, and V-measure.

  • External labels should not automatically be treated as ground truth; they may represent only one valid interpretation.

  • Stability validation: Reclusters bootstrapped, perturbed, or differently sampled versions of the data and checks whether assignments remain similar, often by comparing clusterings with an adjusted Rand index.

  • Domain validation: Examines whether each cluster has a coherent profile and supports a useful decision. This is often more important than a high metric score.

Choosing the Number of Clusters

The elbow method and silhouette score are common for methods such as k-means. Davies–Bouldin can supplement them, while BIC and AIC can select the number of components for Gaussian mixture models; lower BIC or AIC values are preferred. No single criterion is universally reliable, so compare several measures and incorporate domain knowledge.

Selecting and tuning a method repeatedly against one validation statistic can overfit the dataset. Prefer multiple metrics, resampling, and validation on data collected in a different period or population.

Algorithm Trade-offs

Algorithm Cluster shapes found Number of clusters Noise handling Scalability and typical complexity
k-means Convex, similarly sized spherical clusters Requires k No explicit noise label Large-scale friendly; approximately O(n * k * d) per iteration
k-medoids Can capture non-convex shapes with a suitable dissimilarity Requires k Outliers can remain in clusters Moderate; often requires O(n²) pairwise distances
Agglomerative hierarchical Depends on linkage; produces nested clusters Select a dendrogram cut No explicit noise label Usually moderate; common implementations require O(n²) memory
DBSCAN Irregular, density-connected groups No k required Explicitly labels noise Medium to large with spatial indexing; sensitive to density and eps
HDBSCAN Irregular groups with varying densities No k required Explicitly labels noise More robust than DBSCAN across density changes; generally more expensive
Gaussian mixture Elliptical, potentially overlapping clusters Requires component count No explicit noise label Moderate; iterative expectation-maximization can be costly
Spectral clustering Non-convex or manifold-like groups Usually requires k No explicit noise label Best for small or medium data; eigendecomposition is expensive

Single, complete, average, and Ward agglomerative linkage produce different cluster geometry. Ward is particularly suited to Euclidean distances and compact, similarly sized clusters.

Distance, Scaling, and Interpretation

Distance metrics determine what “similar” means. Euclidean distance disproportionately emphasizes large coordinate differences, Manhattan distance sums absolute differences, and cosine distance is useful for text embeddings because it compares direction rather than magnitude.

Standardize features before distance-based clustering when variables have different units or ranges. Scaling changes distances, cluster assignments, and parameters such as DBSCAN’s eps. Trees are generally less sensitive to feature scale, although their splits still depend on the represented quantities.

For example, on two interleaving crescent-shaped groups, k-means often cuts across the crescents, while DBSCAN can recover them when its density parameters match the data. A high silhouette score alone does not prove that clusters are meaningful.

Clusters are hypotheses, not inherent classes. Profile each cluster by feature distributions, label it descriptively, and test whether the groups correspond to actionable behavior or domain concepts.

Common Pitfalls

  • Assuming clusters are ground truth: Use external labels only when they are meaningful.

  • Choosing k solely by one metric: Combine internal scores, stability, and domain knowledge.

  • Ignoring feature scaling: It can cause distance-based methods to prioritize high-magnitude features.

  • Ignoring missing values: Impute or remove them consistently, fitting preprocessing within each training or resampling split to prevent leakage.

  • Treating DBSCAN noise as failure: Noise may represent outliers or a genuinely sparse population.

  • Ignoring cluster size and shape: A high silhouette score does not guarantee balanced or actionable clusters.

  • Overfitting cluster parameters: Tune on resamples and confirm the result on independent observations.

  • Ignoring reproducibility: Fix random seeds and record algorithm, preprocessing, and library versions, especially for randomized methods such as k-means, Gaussian mixtures, and spectral clustering.

  • Skipping interpretation: Always profile clusters before assigning business meaning.

Interviewers commonly ask how eps and min_samples affect DBSCAN, why Ward linkage requires Euclidean distance, how silhouette score handles singleton clusters, and how to distinguish meaningful clusters from algorithmic artifacts.

23

How does the elbow method determine the optimal number of clusters?

The elbow method selects the number of clusters by examining how much the within-cluster error decreases as more clusters are added. It chooses the point where the improvement begins to level off, similar to the bend in an elbow-shaped curve. Its main trade-off is that it is simple and fast, but the bend may be ambiguous or absent.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How the Elbow Method Works

For each candidate number of clusters k, fit a clustering algorithm and calculate its inertia, also called the within-cluster sum of squares:

Inertia(k) = sum of squared distances between each observation and its assigned cluster center

Each observation contributes its squared Euclidean distance from its cluster center, and the values are added across all observations.

A typical curve decreases sharply at first because adding the second or third cluster explains substantial structure. At some point, another cluster produces only a small reduction in inertia. That point is the elbow, and its k is a candidate answer.

For example:

  • k = 1: one large, poorly separated group

  • k = 2: a substantial reduction in inertia

  • k = 3 through k = 10: progressively smaller reductions

  • The elbow: the point after which extra clusters add little value

The elbow is not always sharp. If the curve decreases smoothly, there may be no meaningful optimal k based on this method alone.

Choosing the Number of Clusters

Consider the elbow together with other validation approaches:

  • Silhouette score: Measures how well each point fits its assigned cluster compared with other clusters; higher values are generally better.

  • Davies-Bouldin index: Measures within-cluster spread relative to separation between clusters; lower values are generally better.

  • BIC/AIC: For Gaussian mixture models, these criteria balance fit against model complexity.

  • Domain judgment: The most important check. Clusters should represent meaningful groups, decisions, or behaviors.

  • External metrics: With ground-truth labels, adjusted Rand index, normalized mutual information, and Fowlkes-Mallows score can measure agreement.

The elbow measures compactness, not semantic usefulness. A lower-inertia solution may split a meaningful group into meaningless subgroups, so profile and interpret every resulting cluster.

Distance Metrics and Algorithm Comparison

The elbow method is commonly used with k-means, which uses Euclidean distance. Scaling matters because features with larger numeric ranges otherwise dominate the distance calculation. Manhattan distance is common for sparse data, while cosine distance is useful for text embeddings, where directional similarity matters more than magnitude.

Method Typical cluster shapes Choose k? Noise handling Scalability and complexity
k-means Compact, roughly spherical clusters Yes No native noise label Approximately O(n * k * d) per iteration
k-medoids Can handle more irregular groups than k-means Yes No native noise label Usually slower because medoid updates are expensive
Agglomerative hierarchical Depends on linkage; Ward favors compact groups, while single linkage can form chains Yes, by cutting the tree No native noise label Often O(n²) memory and costly for large data
DBSCAN Arbitrary density-based shapes No Yes, marks outliers as -1 Neighborhood searches can approach O(n² * d)
HDBSCAN Density hierarchy, including clusters of varying density No Yes More flexible about density variation but generally costlier than DBSCAN
Gaussian mixture Elliptical, probabilistic clusters Yes No native noise label Iterative cost depends on k, dimensionality, and covariance structure
Spectral clustering Non-convex and graph-based structures Yes No native noise label Eigenvalue computation is expensive, often around O(n³)

Code Example

This example computes the elbow curve on synthetic blobs and compares k-means with DBSCAN on non-convex moons data. Candidate selection uses only the training split, and each scaler is fitted only on that split.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Compute inertia for k = 1 through 10.
X_blobs, _ = make_blobs(
    n_samples=300,
    centers=4,
    cluster_std=1.0,
    random_state=42,
)
X_train, _ = train_test_split(
    X_blobs, test_size=0.2, random_state=42
)

ks = range(1, 11)
inertias = []

for k in ks:
    model = Pipeline([
        ("scale", StandardScaler()),
        ("cluster", KMeans(n_clusters=k, n_init=10, random_state=42)),
    ])
    model.fit(X_train)
    inertias.append(model.named_steps["cluster"].inertia_)

print("Elbow values:")
for k, inertia in zip(ks, inertias):
    print(f"k={k:2d}  inertia={inertia:.3f}")

# Compare k-means and DBSCAN on curved, non-convex data.
X_moons, _ = make_moons(
    n_samples=300,
    noise=0.05,
    random_state=42,
)
X_train, _ = train_test_split(
    X_moons, test_size=0.2, random_state=42
)

kmeans_model = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
dbscan_model = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.30, min_samples=5)),
])

kmeans_model.fit(X_train)
dbscan_model.fit(X_train)

scaled_train = dbscan_model.named_steps["scale"].transform(X_train)
kmeans_labels = kmeans_model.named_steps["cluster"].labels_
dbscan_labels = dbscan_model.named_steps["cluster"].labels_

non_noise = dbscan_labels != -1
dbscan_cluster_count = len(np.unique(dbscan_labels[non_noise]))

if 2 <= dbscan_cluster_count < non_noise.sum():
    # Use the same observations for both silhouette scores.
    print(
        "k-means silhouette:",
        round(
            silhouette_score(
                scaled_train[non_noise],
                kmeans_labels[non_noise],
            ),
            3,
        ),
    )
    print(
        "DBSCAN silhouette:",
        round(
            silhouette_score(
                scaled_train[non_noise],
                dbscan_labels[non_noise],
            ),
            3,
        ),
    )
    print("DBSCAN clusters:", dbscan_cluster_count)
    print("DBSCAN noise points:", int((~non_noise).sum()))
else:
    print("DBSCAN did not form at least two non-noise clusters.")

Common Pitfalls

  • Treating the elbow as absolute truth: Combine it with silhouette, Davies-Bouldin, domain knowledge, and cluster interpretation.

  • Failing to scale features: Large-range features and outliers can dominate k-means inertia.

  • Choosing k from test data: Select k using training data, then assess the final solution separately.

  • Assuming clusters are ground truth: Clusters are mathematical partitions whose meaning requires interpretation.

  • Ignoring missing values and outliers: Handle them explicitly and consider DBSCAN or HDBSCAN when noise is meaningful.

  • Using unsuitable geometry: K-means favors compact, spherical groups; density-based methods better handle curved or density-varying groups.

  • Interpreting an unsharp curve incorrectly: If no clear elbow exists, the data may not have strong cluster structure.

24

How does random initialization affect K-means clustering results?

Random initialization determines the starting centroids used by K-means, so different initializations can converge to different local optima and produce different partitions. The main trade-off is between initialization and restart cost on one side and solution quality, stability, and reproducibility on the other.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12; NumPy 2.x)

Why Initialization Matters

K-means minimizes the within-cluster sum of squared distances, commonly written as:

inertia = sum over clusters and points of (distance from point to centroid)^2

  • Local optima: Different starting centroids can place the algorithm in different basins of attraction. Lower inertia indicates a better solution under the same data, k, and distance objective.

  • Empty clusters: Reassignment can leave a cluster without points. Scikit-learn relocates that cluster’s center to a point far from the existing centers.

  • Convergence speed: Initialization can change the number of iterations required, although a poor start does not necessarily always take longer.

  • Reproducibility: Setting random_state makes the initialization repeatable for the same input and software environment, but different seeds can still yield different final partitions.

  • Label permutation: Cluster numbers are arbitrary. Compare equivalent partitions with the adjusted Rand index rather than matching raw label numbers.

The init="k-means++" strategy selects initial centers probabilistically while favoring points far from previously selected centers. Running multiple initializations with n_init > 1 and retaining the result with the lowest inertia is generally more reliable than relying on one random start.

Algorithm Context

Algorithm How it works and shapes found Choose k? Noise Strengths Weaknesses Typical use Scale
K-means Assigns points to nearest centroid and updates means; favors compact, roughly spherical clusters Yes No Fast and scalable Sensitive to initialization, scaling, outliers, and non-convex shapes Large datasets with compact clusters O(n*k*d*T) time; O(n*d) storage
K-medoids Partitions around actual data points; shape depends on the dissimilarity Yes No Interpretable centers and non-Euclidean options Costly, outlier-sensitive, and not automatically shape-free Small or medium datasets needing representative medoids Usually requires O(n²) distance computation
Agglomerative hierarchical Repeatedly merges clusters and builds a hierarchy No upfront; cut later No Provides a dendrogram and flexible linkage choices Quadratic cost and sensitive to linkage and outliers Small datasets or hierarchical analysis Often O(n²) time and memory
DBSCAN Connects dense neighborhoods; finds irregular density-connected regions No Yes Handles arbitrary shapes and identifies noise Sensitive to eps and density assumptions Spatial data with irregular clusters and outliers Indexed implementations are moderate; worst case O(n²)
HDBSCAN Builds a density hierarchy and extracts stable clusters Usually no Yes Handles variable density without a fixed k More expensive and sensitive to data conditions Data with varying densities and outliers Moderate to expensive
Gaussian mixture Models clusters as overlapping probability distributions, usually elliptical Yes No Soft assignments and covariance modeling Strong assumptions, convergence sensitivity, and outlier sensitivity Elliptical clusters requiring probabilistic membership Approximately O(n*k*d) per EM iteration for fixed d
Spectral clustering Uses graph eigenvectors before clustering; can capture connected or non-convex structures Usually yes No Handles complex shapes Costly and sensitive to graph construction and scaling Small or medium graph-based datasets Dense implementations commonly require O(n³)

For agglomerative clustering, single linkage favors connectivity, complete linkage favors compact clusters, average linkage uses average pairwise dissimilarity, and Ward linkage favors Euclidean, similarly sized, compact clusters.

Practical Guidance

  • Understand distance metrics: Euclidean distance is straight-line distance and is the objective used by standard scikit-learn KMeans. Manhattan distance sums absolute coordinate differences. Cosine distance measures angular dissimilarity and is often useful for text embeddings, but standard scikit-learn KMeans does not accept it as a metric.

  • Scale features: K-means is sensitive to feature magnitude because large-valued coordinates dominate squared Euclidean distances. Standardize features measured in incompatible units, usually fitting the scaler only on training data before transforming held-out data.

  • Choose k carefully: Compare the elbow method, silhouette score, Davies-Bouldin index, and domain judgment. GMMs additionally support BIC and AIC; labeled comparisons can use the adjusted Rand index.

  • Handle missing values and leakage: Impute missing values within the training pipeline when held-out evaluation is used. Avoid repeatedly tuning preprocessing, k, or initialization against the validation or test set.

  • Interpret clusters: Profile feature distributions, inspect representative members, and identify the practical meaning of each cluster. Clusters are mathematical partitions, not ground-truth classes.

Common Pitfalls

  • Using one random start and treating its result as the only possible optimum.

  • Comparing cluster numbers directly when different runs may permute labels.

  • Selecting k only by inertia, which generally decreases as k increases.

  • Applying K-means to strongly non-convex or density-varying data without comparing DBSCAN, HDBSCAN, or another suitable method.

  • Forgetting to standardize features measured in incompatible units or to process missing values consistently.

25

What advantages does hierarchical clustering have over K-means?

Hierarchical clustering builds a tree of nested clusters, allowing groups to be examined at multiple levels instead of committing immediately to one partition. Over K-means, its main advantages are deferred cluster selection, interpretable relationships between groups, and support for different linkage and distance choices. The main trade-off is scalability: a naïve hierarchical implementation can require quadratic memory and cubic worst-case time, and its results may be sensitive to outliers, scaling, and the chosen metric.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); general hierarchical-clustering concepts are version-independent

Advantages over K-Means

  • Deferred cluster count: When the complete merge tree is retained, it can be cut at different levels to produce coarse or fine partitions. K-means requires the number of clusters before fitting.

  • Nested structure: A dendrogram shows how observations and clusters combine, making relationships within the data easier to interpret.

  • Flexible linkage criteria: Agglomerative clustering supports single, complete, average, and Ward linkage. Single linkage can capture irregular groups but may suffer from chaining; complete linkage favors compact groups; average linkage balances the two; and Ward linkage favors Euclidean, compact, roughly spherical groups.

  • Distance flexibility: Distance-based linkages can use metrics such as Euclidean or Manhattan distance, while methods supporting a precomputed distance matrix can use other suitable measures. Ward linkage itself is restricted to Euclidean distance.

  • Deterministic procedure: Agglomerative clustering does not use random centroid initialization, so the same data, linkage, metric, and implementation produce the same result.

  • Potentially irregular clusters: Single or average linkage may better preserve crescent-shaped or other non-convex groups than K-means. This is not guaranteed, however; Ward linkage on crescent-shaped data can split or merge the wrong regions.

  • Interpretability rather than guaranteed quality: A hierarchy can explain the grouping process, but it does not inherently produce more accurate clusters than K-means.

For comparison, K-means typically costs approximately O(nkdT), where n is the number of samples, k the number of clusters, d the number of features, and T the number of iterations. A naïve agglomerative implementation can require O(n²) memory and O(n³) worst-case time.

Clustering Method Comparison

Method Cluster shapes found Need to choose k? Noise handling Scalability
K-means Compact, roughly spherical clusters Yes No explicit mechanism; outliers can pull centroids Good; suitable for large datasets
K-medoids Non-convex groups when the medoid represents them well Yes No explicit labels; medoids reduce outlier influence Distance calculations can be expensive
Agglomerative Varies by linkage; single linkage can find irregular groups, while Ward favors compact groups Not initially when the full hierarchy is retained; a cut is chosen later No explicit noise labels; robustness depends on linkage Usually poor for large datasets; naïve implementations use O(n³) time and O(n²) memory
DBSCAN Arbitrary-shaped clusters No Strong: noise points receive label -1 Moderate; spatial indexing helps, but worst-case time is O(n²)
HDBSCAN Complex shapes and clusters with different densities No; it uses hierarchy and stability Strong More expensive than DBSCAN
Gaussian mixture Elliptical, probabilistic clusters Yes, although BIC or AIC can help No explicit noise model by default Moderate to expensive
Spectral Connected regions under a graph or similarity matrix Yes No explicit mechanism Expensive; dense eigendecomposition can require O(n³) time

Choosing and Interpreting Clusters

  • Distance metrics and scaling: Standardize numeric features before using Euclidean or Manhattan distance so that variables with large numerical scales do not dominate the result. Cosine distance is often appropriate for text embeddings because it compares direction rather than magnitude.

  • Elbow method: Plot within-cluster sum of squares against k and look for diminishing returns. This criterion is most directly associated with K-means.

  • Silhouette score: Measures how well each point fits its assigned cluster relative to neighboring clusters; higher values generally indicate better separation.

  • Davies-Bouldin index: Compares within-cluster dispersion with separation between clusters; lower values are generally better.

  • BIC/AIC: Useful for choosing the number of components in a Gaussian mixture model.

  • Domain judgement: Statistical indices should support, not replace, interpretation and practical usefulness.

  • External evaluation: If trustworthy labels exist, adjusted Rand index or normalized mutual information can measure agreement. Such labels are not required for clustering and should not automatically be treated as ground truth.

Profile each resulting cluster using feature summaries and representative examples to determine whether it is meaningful. Scikit-learn 1.5 also provides sklearn.cluster.HDBSCAN, an implementation available since scikit-learn 1.3.

Common Pitfalls

  • Ignoring feature scale: Unscaled features can cause distance-based methods to merge the wrong observations.

  • Using Ward for strongly non-convex data: Ward favors compact, roughly spherical groups and assumes Euclidean distance.

  • Assuming every linkage handles outliers equally: Outlier sensitivity varies by linkage, metric, and dataset.

  • Choosing k only mechanically: Cluster counts should also be supported by domain interpretation.

  • Ignoring missing values: Handle missing data explicitly; in a train/evaluation workflow, fit the imputer only on training data.

  • Assuming scalability: Hierarchical and spectral methods can become impractical on large datasets; consider K-means, MiniBatch K-means, or HDBSCAN.

  • Treating clusters as ground truth: Cluster labels depend on the features, scaling, algorithm, and cut level, so they require interpretation and stability checks.

26

How does PAM clustering differ from K-means?

Partitioning Around Medoids (PAM) differs from K-means by representing each cluster with an observed data point called a medoid, whereas K-means uses an arithmetic-mean centroid. PAM minimizes the sum of distances to the medoids and is generally more resistant to outliers when using Manhattan distance, but it is substantially more computationally expensive. K-means scales better but is sensitive to outliers and to features measured on very different scales.

Read full answer

Applies toall versions

Core Difference

For PAM, the objective is:

total distance = sum of each point's distance to its assigned medoid

PAM commonly uses a BUILD initialization phase followed by a SWAP phase that tests whether replacing a medoid with another observation lowers the objective. Because a medoid must be an actual observation, it is interpretable and can be less influenced by extreme values than a centroid.

For K-means, the objective is:

total squared distance = sum of each point's squared Euclidean distance to its assigned centroid

K-means assigns points to the nearest centroid and replaces each centroid with the arithmetic mean of its assigned points. Its centroids are synthetic locations and therefore need not correspond to observations in the dataset.

Algorithm Comparison

Method How It Works Cluster Shapes Choose k? Explicit Noise Scalability
K-means Assigns points to centroids and recomputes their means Spherical, similar-sized clusters Yes No High; approximately O(n*k*d) per iteration
PAM / K-medoids Assigns points to medoids and swaps medoids when beneficial Metric-dependent; can capture some non-spherical groups Yes No Lower; an O(n²) distance matrix is commonly stored
Agglomerative hierarchical Repeatedly merges the closest clusters Depends on linkage: single, complete, average or Ward Can select after merging No Usually limited by O(n²) memory
DBSCAN Groups points connected through dense neighborhoods Arbitrary shapes Uses eps and min_samples Yes Neighborhood queries can become costly in high dimensions
HDBSCAN Builds a density hierarchy and extracts stable clusters Irregular shapes and variable density Uses min_cluster_size Yes Moderate; more flexible than fixed-eps DBSCAN
Gaussian mixture Models clusters as probabilistic Gaussian components Elliptical clusters with potentially different sizes and orientations Usually yes No Roughly O(n*k*d) per EM iteration for fixed dimensionality
Spectral clustering Uses graph eigenvectors to identify connected groups Non-convex and manifold-like structures Usually yes No Low to moderate; a dense graph can require O(n²) memory

Choosing and Evaluating k

No single criterion always identifies the correct number of clusters:

  • Elbow method: Looks for diminishing reductions in within-cluster variation as k increases.

  • Silhouette score: Measures separation between clusters, although it can be misleading for strongly non-convex groups.

  • Davies-Bouldin index: Combines within-cluster dispersion with separation between clusters.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity; they are not standard criteria for PAM or K-means.

  • Domain judgement: A useful clustering should produce segments that are distinguishable, actionable and interpretable.

  • External metrics: When reliable labels exist, adjusted Rand index or another external metric can measure agreement with those labels.

Distance Metrics and Scaling

  • Euclidean distance: Measures straight-line separation and is the objective used by standard K-means.

  • Manhattan distance: Measures total coordinate movement and is commonly used by PAM; it is generally less dominated by a few very large coordinate differences.

  • Cosine distance: Compares vector direction rather than magnitude and is useful for text embeddings when relative direction matters more than overall vector length.

Standardize features before applying distance-based clustering when variables use different units or have substantially different variances. Ward linkage is restricted to Euclidean geometry, while other hierarchical linkages can use compatible precomputed distances. Missing values must be handled explicitly, and duplicate observations can disproportionately influence medoids or centroids.

Common Pitfalls

  • Using accuracy for unlabeled data: Clustering has no natural accuracy metric; use internal metrics, external metrics when labels exist, and stability checks.

  • Ignoring feature scale: A large-range feature can dominate the distance calculation and determine the clusters.

  • Selecting k mechanically: Combine statistical criteria with domain knowledge and profile each cluster’s feature distributions.

  • Treating clusters as ground truth: Clusters are hypotheses, not guaranteed natural classes, and require substantive interpretation.

  • Assuming PAM is immune to outliers: Its robust Manhattan objective reduces outlier influence but does not eliminate poor medoid-selection risks.

  • Leaking preprocessing information: When clustering is part of a predictive pipeline, fit scaling and other learned preprocessing steps on training data only.

27

How do agglomerative and divisive hierarchical clustering differ?

Agglomerative hierarchical clustering organizes unlabeled observations from the bottom up, starting with singleton clusters and repeatedly merging the closest pair. Divisive hierarchical clustering starts with one cluster and repeatedly divides it into smaller groups. Their main trade-off is merge flexibility versus split-search cost: agglomerative clustering offers several linkage criteria, while divisive clustering must choose and evaluate splits, and neither method can undo its committed decisions.

Read full answer

Applies tohierarchical clustering concepts (all versions)

Core Differences

Aspect Agglomerative Divisive
Starting point Each observation is a separate cluster All observations form one cluster
Main operation Merges the two closest clusters Splits one cluster into two groups
Hierarchy Bottom-up Top-down
Stopping decision Continue until a target cluster count or distance threshold is reached Continue until a target cluster count or stopping criterion is reached
Reversibility Early merges cannot be undone Early splits cannot be undone
Scalability Naive implementations use O(n³) time and O(n²) memory Cost depends strongly on the split search; examining every bipartition can be exponential in cluster size, so heuristics are common

No universal speed ordering exists: relative cost depends on the linkage, split criterion, implementation, and data size.

Linkage and Split Criteria

Agglomerative clustering uses a linkage criterion to select the clusters to merge:

  • Single linkage: uses the smallest cross-cluster distance and can create elongated, chained clusters.

  • Complete linkage: uses the largest cross-cluster distance and tends to produce compact clusters.

  • Average linkage: uses the mean cross-cluster distance, balancing the previous two approaches.

  • Ward linkage: merges clusters that produce the smallest increase in within-cluster variance; it requires Euclidean geometry.

Divisive clustering needs a criterion for choosing each split. Common choices include Ward-like minimum-variance splits and k-means-style partitions based on two centroids. The split criterion determines which distance geometry is appropriate. Divisive methods can examine a whole parent partition before committing to a split, whereas agglomerative methods commit to each merge immediately.

Broader Clustering Comparison

Method How it works and cluster shapes Choose k? Noise Strengths, weaknesses, use, and scale
k-means Euclidean centroid assignment; favors compact, roughly spherical clusters Yes No Fast for well-separated spherical groups, but sensitive to initialization and outliers; O(nkd) per iteration
k-medoids Uses actual observations as centers; can accommodate less-spherical groups with suitable settings Yes Limited More robust than means to extreme values, but costly and sensitive to k; approximately O(n²d) per iteration
Agglomerative Linkage-dependent; single linkage can chain points while Ward favors compact groups Only when a target count is used; otherwise use a threshold No native label Produces an interpretable dendrogram, but merges are irreversible; standard implementation uses O(n²) memory
DBSCAN Connects density-reachable regions and finds arbitrary shapes No Yes Useful for irregular clusters, but sensitive to eps and min_samples; neighborhood search can become expensive
HDBSCAN Builds a density hierarchy and extracts stable clusters, including at different density levels No Yes Handles varying density without a global eps, but hierarchy construction adds overhead and cost depends on the data
Gaussian mixture Models elliptical probability distributions with soft assignments Yes No Handles overlapping and elliptical clusters, but depends on distributional assumptions and initialization
Spectral clustering Uses a similarity graph and eigenvectors; finds non-convex shapes Yes No Captures graph structure, but is sensitive to graph construction and expensive for dense data; roughly O(n²) memory and O(n³) eigendecomposition

Distance, Scaling, and Choosing Clusters

Euclidean distance is:

d(x, y) = sqrt(Σ(i=1 to n) (x_i - y_i)²)

Here, d(x, y) is the distance between observations x and y; n is the number of features; i indexes those features; x_i and y_i are corresponding feature values; and the summation adds all squared differences before taking the square root.

Features measured on very different scales can dominate this calculation, so standardization is generally advisable when units or ranges differ. Tree-based models usually do not require scaling, whereas distance-based clustering normally does. Cosine distance is often more meaningful for text embeddings because it compares directional similarity rather than magnitude.

No internal criterion should determine the cluster count by itself:

  • Elbow method: looks for a bend in inertia versus cluster count.

  • Silhouette score: rewards cohesion and separation; higher is generally better.

  • Davies-Bouldin index: lower indicates better-separated clusters.

  • BIC/AIC: compares Gaussian mixture models while penalizing model complexity.

  • Domain judgment: determines whether groups are meaningful or actionable.

With trusted reference labels, external measures such as the adjusted Rand index can evaluate agreement with those labels.

Common Pitfalls

  • Ignoring scale or leakage: scale features when appropriate and fit preprocessing only on training folds.

  • Leaving missing values untreated: impute them using a leakage-safe procedure or use a method designed for missing data.

  • Over-tuning k: repeatedly selecting the configuration that best fits one validation set can overfit it; use stability checks or cross-validation.

  • Ignoring stability: compare results across resamples and parameter ranges, and record random seeds for stochastic methods.

  • Treating clusters as ground truth: profile each cluster’s feature distributions and domain meaning; cluster labels are descriptive rather than inherently correct classes.

  • Using classification accuracy carelessly: arbitrary cluster labels and unequal cluster sizes can make accuracy misleading; use suitable external clustering metrics instead.

28

What are spectral clustering’s benefits and suitable problem types?

Spectral clustering converts data into a similarity graph and partitions that graph to identify groups, including non-convex or manifold-like structures that centroid methods may miss. It is well suited to image segmentation, graph partitioning, and carefully chosen embedding vectors. Its main trade-off is greater modeling flexibility versus the need to choose k, sensitivity to graph construction, and comparatively high eigendecomposition cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12) and general clustering concepts

How It Works

Spectral clustering represents each sample as a graph vertex and connects similar samples using an affinity matrix W. A common affinity is the RBF kernel:

W[i, j] = exp(-distance(x_i, x_j)^2 / (2 * sigma^2))

Here, distance measures dissimilarity, W measures similarity, and sigma controls how quickly affinity decreases as distance increases. The algorithm then:

  1. Constructs the similarity graph.

  2. Computes eigenvectors of a graph Laplacian, commonly L = D - W.

  3. Uses the k lowest-frequency eigenvectors to create an embedded representation.

  4. Clusters that embedding, usually with k-means.

This graph-based representation can separate curved or irregularly shaped groups that direct Euclidean-distance clustering may merge.

Benefits and Suitable Problems

  • Non-convex clusters: Separates crescent-shaped or otherwise irregular groups that k-means can combine.

  • Manifold-like data: Identifies groups that lie on curved surfaces in a higher-dimensional space.

  • Image segmentation: Groups pixels according to color, texture, and neighborhood relationships.

  • Graph partitioning: Finds communities in social, citation, and network data.

  • Embedding-based data: Can group language-model or other vector embeddings according to local similarity.

  • Flexible assumptions: Does not require clusters to be spherical or have similar densities.

The principal limitations are the need to specify n_clusters, sensitivity to the affinity matrix and neighborhood settings, and the absence of a predict method in scikit-learn’s SpectralClustering. Assigning new observations generally requires rebuilding the embedding or using a separate approximation.

Algorithm Comparison

Algorithm Cluster Shapes Found Choose k? Handles Noise? Scalability
k-means Roughly spherical, similarly sized clusters Yes No Large datasets; O(n * k * d) per iteration
k-medoids Usually compact clusters; medoids are actual observations and are less sensitive to extreme values Yes No dedicated noise model Moderate; swap-based updates can require O(n^2) per iteration
Agglomerative Single linkage favors chains, complete favors compact groups, average is intermediate, and Ward favors spherical groups Yes No Basic linkage methods can require O(n^3) time
DBSCAN Arbitrary shapes and density-connected regions No Yes Spatial indexes can make average queries efficient; worst case is O(n^2)
HDBSCAN Arbitrary shapes with varying cluster densities No Yes Implementation-dependent; hierarchy construction can be costly on large datasets
Gaussian mixture Elliptical clusters with probabilistic membership Yes No explicit noise component O(n * k * d) per expectation-maximization iteration
Spectral Non-convex, manifold-like, and graph-connected groups Yes Not natively in scikit-learn Dense graphs use O(n^2) storage and can require O(n^3) eigendecomposition; sparse graphs reduce storage and permit iterative solvers

Choosing Clusters and Handling Features

  • Elbow method: Inspect the inertia curve for centroid-based methods such as k-means.

  • Silhouette score: Measures within-cluster cohesion relative to separation; it can be costly on large datasets.

  • Davies-Bouldin index: Rewards clusters that are compact and well separated.

  • BIC and AIC: Compare Gaussian mixture models while penalizing model complexity.

  • Domain judgment: Check whether each cluster has a meaningful scientific or operational profile.

  • External metrics: With reference labels, use adjusted Rand index, normalized mutual information, or V-measure.

Feature scaling is important when features use different units or have substantially different magnitudes. Euclidean distance is commonly used for continuous measurements, while Manhattan distance can be appropriate when coordinate-wise differences matter. Cosine distance is useful for text embeddings when direction matters more than magnitude. Missing values must be handled before graph construction, and outliers can strongly affect affinities.

Common Pitfalls

  • Scaling after clustering: Standardize features before computing distances or affinities when their scales differ.

  • Choosing k only by one metric: Combine internal metrics, stability tests, and domain knowledge.

  • Ignoring disconnected graphs: Weak affinities can create isolated components and poor global partitions.

  • Treating clusters as truth: Interpret cluster profiles and validate stability across seeds, samples, and parameter choices.

  • Assuming scalability: Exact spectral clustering can be expensive; sparse graphs, approximate methods, or specialized implementations may be necessary for large data.

29

How does affinity propagation clustering work?

Affinity propagation clusters observations by selecting representative data points, called exemplars, and assigning every observation to one of them through iterative message passing. It avoids requiring the number of clusters in advance, but its dense quadratic storage makes it expensive for large datasets and its results can be sensitive to preference and damping.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

Affinity propagation treats every point as a possible exemplar rather than starting with predetermined centroids.

  • Similarity: s(i, k) measures how well observation i matches candidate exemplar k. By default, scikit-learn uses negative squared Euclidean distance.

  • Responsibility: r(i, k) estimates how strongly observation i prefers exemplar k over alternatives.

  • Availability: a(i, k) estimates how strongly exemplar k remains able to represent observation i.

  • Preference: p(k) controls how attractive each candidate exemplar is. Higher values generally produce more exemplars; lower values produce fewer.

The conceptual responsibility update is:

r(i, k) ← s(i, k) − max over k' ≠ k of [a(i, k') + s(i, k')]

For i ≠ k, availability is:

a(i, k) ← min(0, r(k, k) + sum over i' ∉ {i, k} of max(0, r(i', k)))

Self-availability has its own update:

a(k, k) ← sum over i' ≠ k of max(0, r(i', k))

Here, i is an observation, k is a candidate exemplar, and k' represents competing exemplars. Responsibilities balance similarity against competition, while availabilities incorporate support from other observations. The algorithm alternates these updates until messages stabilize or the iteration limit is reached, then returns labels and the indices of the selected exemplars.

Parameter Control and Trade-offs

  • preference=None: Uses the median similarity as the default preference. This is convenient, but the resulting cluster count may not suit the application.

  • damping: Blends new messages with previous messages. The scikit-learn default is 0.5; greater damping can improve stability at the cost of slower convergence.

  • max_iter and convergence_iter: Limit total iterations and require that many stable iterations before declaring convergence.

  • Scalability: The implementation stores dense similarity and message matrices, requiring O(n²) memory and O(n²) work per iteration.

  • Distance metric: Euclidean distance is suitable for scaled numeric features. AffinityPropagation has no metric parameter; for text embeddings, pass a negated cosine-distance matrix with affinity="precomputed".

Use domain judgement when choosing a cluster count or preference. Common validation tools include the elbow method for k-means, silhouette score, Davies-Bouldin score, and BIC/AIC for Gaussian mixtures. If trustworthy labels exist, the adjusted Rand index can compare the clustering with those labels.

Algorithm Comparison

Algorithm Cluster shapes found Choose k Native noise handling Typical scalability
k-means Convex, roughly spherical Yes No O(nkd) per iteration
k-medoids Medoid-centered, often compact; supports arbitrary dissimilarity Yes No Often O(n²) per update
Agglomerative Depends on linkage: single, complete, average, or Ward Yes No Often O(n²)
DBSCAN Non-convex density-connected shapes No Yes Often near O(n log n), worst O(n²)
HDBSCAN Hierarchical, variable-density shapes No Yes Suitable for moderate-to-large datasets
Gaussian mixture Elliptical, probabilistic clusters Usually No O(nkd) per iteration
Spectral Non-convex graph-based structure Usually No Dense methods can require O(n³)
Affinity propagation Exemplar-centered partitions; not shape-specific No No O(n²) memory and work per iteration

Scikit-learn provides sklearn.cluster.HDBSCAN; it was introduced in scikit-learn 1.3.

Code Example

This example compares affinity propagation with k-means and DBSCAN on interleaving moons. DBSCAN follows the curved density structure, while k-means tends to make a roughly linear partition.

import numpy as np

from sklearn.cluster import AffinityPropagation, DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score

X, _ = make_moons(n_samples=500, noise=0.04, random_state=42)

affinity = AffinityPropagation(random_state=42)
affinity_labels = affinity.fit_predict(X)

kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans_labels = kmeans.fit_predict(X)

dbscan = DBSCAN(eps=0.15, min_samples=5)
dbscan_labels = dbscan.fit_predict(X)


def valid_silhouette(labels):
    # Exclude DBSCAN noise explicitly instead of treating -1 as a cluster.
    mask = labels != -1
    filtered_labels = labels[mask]
    n_clusters = len(np.unique(filtered_labels))
    n_samples = len(filtered_labels)

    if not 2 <= n_clusters <= n_samples - 1:
        return float("nan")

    return silhouette_score(X[mask], filtered_labels)


print("Affinity propagation clusters:",
      len(affinity.cluster_centers_indices_))
print("Affinity propagation silhouette:",
      valid_silhouette(affinity_labels))
print("K-means silhouette:", valid_silhouette(kmeans_labels))
print("DBSCAN clusters:",
      len(np.unique(dbscan_labels[dbscan_labels != -1])))
print("DBSCAN silhouette:", valid_silhouette(dbscan_labels))

Common Pitfalls

  • Scaling: Euclidean-based affinity propagation is sensitive to feature scale. Standardize features with different units; when using a train/validation split, fit the scaler on the training portion only.

  • Duplicate observations: Identical points can affect the number of exemplars and make interpretation harder.

  • Large datasets: Dense O(n²) storage can become a bottleneck.

  • Silhouette usage: Exclude DBSCAN noise before computing silhouette. A silhouette computed only on clustered points describes those points, not the noise distribution.

  • Interpretation: Clusters are not ground truth or class labels. Profile each cluster and validate its usefulness against domain knowledge.

Interview Tips

Be ready to explain the difference between responsibility and availability, how preference affects cluster count, and why affinity propagation is unsuitable for very large datasets. Also discuss when DBSCAN or HDBSCAN is preferable for irregular shapes and noise, and remember that clustering requires scientific or business interpretation rather than automatic validation alone.

30

How does BIRCH clustering work?

BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) is an unsupervised clustering algorithm that summarizes data in a hierarchical clustering feature (CF) Tree, allowing large datasets to be clustered without storing all pairwise distances. Its main trade-off is speed and compact memory use versus sensitivity to feature scaling, threshold, and cluster geometry.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How BIRCH Works

  1. Create CF summaries: Each subcluster records statistics such as its sample count, centroid, and variance or spread.

  2. Insert observations: A new point is merged with an existing subcluster when the resulting radius remains within threshold.

  3. Split overloaded nodes: If a node exceeds branching_factor, it is split so the CF Tree remains balanced and bounded.

  4. Create final clusters: With n_clusters=None, the leaf subclusters receive the final labels. With an integer n_clusters, scikit-learn applies a final agglomerative clustering step to the resulting subclusters.

A lower threshold generally produces more, smaller subclusters; a higher value produces broader ones. BIRCH supports incremental updates and avoids a full pairwise-distance matrix.

Comparison With Other Algorithms

BIRCH’s radius model assumes Euclidean geometry, so features measured in different units should be scaled. Cosine distance is useful for text embeddings, although scikit-learn’s Birch does not expose an alternative distance metric.

Algorithm Cluster shapes found Choose k? Explicit noise handling Typical scalability
BIRCH Compact, approximately spherical groups Optional No Large datasets; avoids an (n times n) distance matrix
k-means Convex, similarly sized spherical clusters Yes No (O(nkd)) per iteration
k-medoids Depends on the data and medoids Yes No Often (O(n^2k))
Agglomerative Depends strongly on linkage Optional No Typically (O(n^2))
DBSCAN Arbitrary shapes and variable density No Yes Near (O(n log n)) with spatial indexing; may degrade to (O(n^2))
HDBSCAN Arbitrary shapes and density levels No Yes Medium or large datasets with additional overhead
Gaussian mixture Elliptical, probabilistic clusters Yes No explicit noise cluster Around (O(nkd)) per EM iteration
Spectral Connected or non-convex structures Yes No Usually small to medium; dense graphs require (O(n^2)) memory

Agglomerative clustering also varies by linkage: single can connect distant points, complete favors compact clusters, average balances the two, and Ward minimizes within-cluster variance.

Choosing and Evaluating Clusters

  • Elbow method: Looks for an elbow in within-cluster sum of squares, mainly for k-means-like solutions.

  • Silhouette score: Ranges from -1 to 1; higher values generally indicate better separation.

  • Davies-Bouldin index: Favors clusters that are compact and far from other clusters.

  • BIC/AIC: Model-selection criteria for probabilistic methods such as Gaussian mixture models.

  • Adjusted Rand index: Measures agreement with trusted labels when external labels exist.

  • Domain judgment: Determines whether the clusters represent meaningful populations, products, behaviors, or operating conditions.

Cluster IDs have no intrinsic ordering and are not ground truth. Profile each cluster using feature distributions and domain expertise rather than relying only on an internal metric.

Code Example

The BIRCH example uses three well-separated blobs. The moons example contrasts k-means with DBSCAN, which can capture non-convex density-connected groups.

import numpy as np

from sklearn.cluster import Birch, DBSCAN, KMeans
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import silhouette_score

X_blobs, _ = make_blobs(
    n_samples=300,
    centers=3,
    cluster_std=0.7,
    random_state=42,
)

birch = Birch(n_clusters=3, threshold=0.5, branching_factor=50)
birch.fit(X_blobs)
birch_labels = birch.labels_

print("BIRCH cluster count:", np.unique(birch_labels).size)
print("BIRCH silhouette:", round(silhouette_score(X_blobs, birch_labels), 3))

X_moons, _ = make_moons(
    n_samples=300,
    noise=0.05,
    random_state=42,
)

kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans_labels = kmeans.fit_predict(X_moons)

dbscan = DBSCAN(eps=0.08, min_samples=5)
dbscan_labels = dbscan.fit_predict(X_moons)

# Exclude DBSCAN noise points when calculating silhouette.
mask = dbscan_labels != -1
print("DBSCAN noise points:", np.sum(~mask))
print("k-means silhouette:", round(silhouette_score(X_moons, kmeans_labels), 3))
print(
    "DBSCAN silhouette:",
    round(silhouette_score(X_moons[mask], dbscan_labels[mask]), 3),
)

Best Practices

  • Scale features when their units or ranges differ; handle missing values before fitting because BIRCH does not accept NaN values.

  • Tune threshold and branching_factor, and set n_clusters only when a specific partition is required.

  • Use DBSCAN or HDBSCAN when explicit noise handling or strongly non-convex clusters are priorities.

  • Inspect cluster profiles and record preprocessing, parameters, and random seeds for reproducibility.

Common Pitfalls

  • Unscaled features can cause high-magnitude variables to dominate the clustering.

  • Choosing n_clusters does not establish that the resulting groups are meaningful.

  • Silhouette score can favor compact, convex clusters and undervalue irregular but useful groups.

  • Repeatedly tuning parameters against the same evaluation data creates model-selection leakage.

Common follow-ups include how threshold changes granularity, when DBSCAN is preferable to BIRCH, and how cluster profiles can be validated with domain knowledge.

31

Implement K-means clustering from scratch in Python

K-means partitions unlabeled data into k clusters by repeatedly assigning each point to its nearest centroid and recomputing each centroid as the members’ mean. It provides fast, simple unsupervised grouping, with the main trade-off being sensitivity to outliers, irregular cluster shapes, feature scaling, and the selected value of k.

Read full answer

Applies toPython 3.12 with NumPy 2.x and scikit-learn 1.5+ for dataset generation and evaluation

How It Works

The objective is to minimize the within-cluster sum of squares:

WCSS = sum(i=1..n) ||x_i - mu_(z_i)||²

  • n is the number of data points.

  • x_i is the vector representing point i.

  • z_i is the cluster assigned to point i.

  • mu_(z_i) is the centroid of that cluster.

  • ||...||² is the squared Euclidean distance.

The algorithm then:

  1. Selects initial centroids using K-means++, which favors points far from previously selected centroids.

  2. Assigns each point to its closest centroid.

  3. Recalculates each centroid as the mean of its assigned points.

  4. Re-seeds empty clusters from points far from their current centers.

  5. Repeats until centroid movement is below tol.

  6. Runs n_init initializations and returns the solution with the lowest WCSS.

With n points, k clusters, and d features, each iteration costs approximately O(n * k * d).

Algorithm Comparison

Algorithm Cluster shapes found Choose k Handles noise Typical scalability
K-means Compact, spherical clusters Yes No O(n * k * d) per iteration
K-medoids Convex-ish groups; medoids improve robustness to outliers Yes Limited Often O(n²)
Agglomerative hierarchical Depends on linkage; Ward favors spherical groups Usually No Often O(n²) memory
DBSCAN Arbitrary, density-connected shapes No Yes Average near O(n log n); quadratic worst case
HDBSCAN Irregular clusters with varying density No Yes Graph construction dominates; worst case O(n²)
Gaussian mixture Elliptical clusters with soft assignments Yes No explicit noise model O(n * k * d²) per iteration for full covariance
Spectral Non-convex or manifold-like structures Yes No Typically O(n³)

Single, complete, average, and Ward linkage produce different hierarchical structures. scikit-learn 1.5+ includes sklearn.cluster.HDBSCAN, introduced in scikit-learn 1.3.

Code Example

import numpy as np
from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score


def squared_distances(x, centers):
    """Squared Euclidean distances between every row and centroid."""
    distances = (
        np.sum(x * x, axis=1)[:, None]
        - 2.0 * x @ centers.T
        + np.sum(centers * centers, axis=1)[None, :]
    )
    return np.maximum(distances, 0.0)


def kmeans_plus_plus(x, k, rng):
    """Initialize centroids with K-means++ sampling."""
    n, d = x.shape
    centers = np.empty((k, d), dtype=np.float64)
    closest_squared = np.full(n, np.inf)

    for cluster in range(k):
        if cluster == 0:
            index = rng.integers(n)
        else:
            total = closest_squared.sum()
            if total == 0.0:
                index = rng.integers(n)
            else:
                index = rng.choice(n, p=closest_squared / total)

        centers[cluster] = x[index]
        distances = np.sum((x - centers[cluster]) ** 2, axis=1)
        closest_squared = np.minimum(closest_squared, distances)

    return centers


def kmeans_from_scratch(
    x, k, max_iter=300, tol=1e-4, n_init=10, random_state=42
):
    x = np.asarray(x, dtype=np.float64)
    if x.ndim != 2 or not np.isfinite(x).all():
        raise ValueError("x must be a finite 2D array")

    n = x.shape[0]
    if not 1 <= k <= n:
        raise ValueError("k must be between 1 and the number of rows")

    rng = np.random.default_rng(random_state)
    best_labels = best_centers = None
    best_inertia = np.inf

    for _ in range(n_init):
        centers = kmeans_plus_plus(x, k, rng)

        for _ in range(max_iter):
            distances = squared_distances(x, centers)
            labels = np.argmin(distances, axis=1)
            counts = np.bincount(labels, minlength=k)
            assigned_distance = distances[np.arange(n), labels]

            # Re-seed empty clusters using distant points from non-singleton groups.
            empty_clusters = np.flatnonzero(counts == 0)
            for cluster in empty_clusters:
                candidates = np.flatnonzero(counts[labels] > 1)
                if candidates.size == 0:
                    candidates = np.arange(n)

                point = candidates[np.argmax(assigned_distance[candidates])]
                previous_cluster = labels[point]
                labels[point] = cluster
                counts[previous_cluster] -= 1
                counts[cluster] += 1

            new_centers = np.array(
                [
                    x[labels == cluster].mean(axis=0)
                    if np.any(labels == cluster)
                    else centers[cluster]
                    for cluster in range(k)
                ]
            )

            movement = np.linalg.norm(new_centers - centers, axis=1).max()
            centers = new_centers

            if movement <= tol and empty_clusters.size == 0:
                break

        distances = squared_distances(x, centers)
        labels = np.argmin(distances, axis=1)
        inertia = distances[np.arange(n), labels].sum()

        if inertia < best_inertia:
            best_inertia = inertia
            best_labels = labels.copy()
            best_centers = centers.copy()

    return best_labels, best_centers, best_inertia


# Two curved moons expose K-means' difficulty with non-convex clusters.
x, _ = make_moons(n_samples=500, noise=0.08, random_state=42)

kmeans_labels, centers, inertia = kmeans_from_scratch(x, k=2)
dbscan_labels = DBSCAN(eps=0.08, min_samples=5).fit_predict(x)
non_noise = dbscan_labels != -1

print("K-means cluster sizes:", np.bincount(kmeans_labels))
print("K-means inertia:", round(inertia, 3))
print("K-means silhouette:", round(silhouette_score(x, kmeans_labels), 3))
print("DBSCAN cluster sizes:", np.bincount(dbscan_labels[dbscan_labels >= 0]))
print(
    "DBSCAN silhouette:",
    round(silhouette_score(x[non_noise], dbscan_labels[non_noise]), 3),
)

Practical Guidance

  • Distance and scaling: This implementation uses Euclidean distance, squared when calculating WCSS. Features measured on different scales must be standardized so that large-range variables do not dominate. Manhattan distance is another option for distance-based variants; cosine distance is useful for text embeddings because it compares direction rather than magnitude.

  • Missing values: The implementation rejects NaN and infinite values. Impute missing values before clustering, and fit preprocessing only on training data when clustering supports a downstream predictive model.

  • Choose k carefully: Use the elbow method, silhouette score, Davies-Bouldin index, and domain knowledge. BIC or AIC can help for Gaussian mixtures.

  • Evaluate and interpret: Silhouette is higher when better; Davies-Bouldin and BIC/AIC are lower when better. With ground-truth labels, adjusted Rand index measures external agreement. Profile each cluster’s features, size, and business meaning because clusters are not ground truth.

  • Reproducibility: Set a fixed random seed. K-means++ and multiple initializations reduce—but do not eliminate—sensitivity to initialization.

Common Pitfalls

  • Expecting K-means to discover irregular shapes or automatically identify outliers.

  • Running Euclidean-distance algorithms on unscaled features.

  • Selecting k only because a score looks acceptable without checking cluster profiles.

  • Computing silhouette for a singleton cluster or including DBSCAN noise points labeled -1.

  • Fitting preprocessing or clustering before splitting data when clusters will support a supervised model, which prevents training-to-test leakage.

Interview Tips

Common follow-ups include deriving the K-means update equations, explaining K-means++, analyzing convergence, comparing it with GMM and DBSCAN, and discussing how to validate clusters without labels.

32

How can hierarchical clustering produce groups and a dendrogram in Python?

Hierarchical clustering groups observations by repeatedly merging the most similar clusters, producing a tree that can be visualized as a dendrogram. It solves segmentation tasks when relationships between groups and multiple levels of granularity are useful to inspect. Its main trade-off is interpretability versus scalability: dendrograms are intuitive, but typical implementations require quadratic resources and are sensitive to scaling, distance metrics, and linkage choices.

Read full answer

Applies toscikit-learn 1.5+, SciPy 1.13+, and Matplotlib 3.8+ (Python 3.12)

How It Works

Agglomerative hierarchical clustering starts with each observation as its own cluster:

  1. Calculate the distance between every pair of clusters.

  2. Merge the closest pair according to the selected linkage.

  3. Recalculate distances and repeat.

  4. Cut the resulting tree at a chosen height or number of clusters.

A dendrogram displays each merge, with its vertical position representing the distance at which the merge occurred. SciPy’s linkage creates the merge matrix, while scikit-learn’s cut_tree turns that tree into cluster labels for a requested number of clusters.

Common linkage methods include:

Linkage Merge criterion Suitable shapes Main caveat
Single Smallest distance between any members Irregular, separated groups Can cause chaining
Complete Largest distance between members Relatively compact groups May favor spherical clusters
Average Mean distance between members General-purpose clustering Can split elongated groups
Ward Minimizes the increase in within-cluster variance Compact, similarly sized groups Requires Euclidean distance

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Explicit noise handling Scalability
k-means Convex, similarly sized clusters Yes No Good; approximately O(n * k * d) per iteration
k-medoids Depends on distance and medoid placement; often compact Yes No Moderate to poor for large data
Agglomerative hierarchical (single/complete/average/Ward) Linkage-dependent; Ward favors compact groups Cut afterward No Typically O(n²) memory and work
DBSCAN Dense, irregular shapes No Yes Good with spatial indexing; quadratic worst case
HDBSCAN Irregular and variable-density clusters No Yes Moderate; available in scikit-learn since 1.3
Gaussian mixture Elliptical, probabilistic clusters Yes No explicit noise label; broad components may absorb outliers Good for moderate data sizes
Spectral Non-convex or manifold-like groups Usually yes No Often limited by O(n²) memory

Practical Guidance

  • Scale features: Euclidean distance is affected by feature units. Use StandardScaler for measurements with different ranges, such as age in years and income in dollars.

  • Choose a metric deliberately: Euclidean suits continuous numeric data, Manhattan can be less influenced by large coordinate differences, and cosine distance is useful for text embeddings when direction matters more than magnitude.

  • Select the cut carefully: Consider dendrogram gaps, domain knowledge, silhouette score, Davies-Bouldin index, and the elbow method. BIC and AIC are particularly applicable when comparing Gaussian mixture models. With ground-truth labels, adjusted Rand index can measure agreement.

  • Interpret clusters: Summarize feature distributions for each cluster rather than treating cluster IDs as meaningful labels. Clusters are not automatically ground truth.

  • Handle missing values: Impute them before distance calculations, preferably within a pipeline so preprocessing is fitted only on training data during downstream validation.

Code Example

import matplotlib.pyplot as plt
import numpy as np
from scipy.cluster.hierarchy import dendrogram, linkage
from sklearn.cluster import DBSCAN, KMeans, cut_tree
from sklearn.datasets import make_blobs, make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Hierarchical clustering example
X, _ = make_blobs(
    n_samples=300,
    centers=3,
    cluster_std=1.0,
    random_state=42,
)
X_train, X_test = train_test_split(
    X, test_size=0.2, random_state=42
)

# Fit preprocessing only on the training partition.
preprocess = Pipeline([("scaler", StandardScaler())])
X_train_scaled = preprocess.fit_transform(X_train)
X_test_scaled = preprocess.transform(X_test)

# Each Z row contains merged cluster IDs, distance, and cluster size.
Z = linkage(X_train_scaled, method="ward", metric="euclidean")
labels = cut_tree(Z, n_clusters=3).ravel()

print(
    "Hierarchical training silhouette score: "
    f"{silhouette_score(X_train_scaled, labels):.3f}"
)
print(f"Cluster sizes: {np.bincount(labels).tolist()}")
print(f"Held-out samples reserved for downstream validation: {len(X_test)}")

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

axes[0].scatter(X_train_scaled[:, 0], X_train_scaled[:, 1], c=labels, cmap="viridis")
axes[0].set_title("Hierarchical clusters")
axes[0].set_xlabel("Standardized feature 1")
axes[0].set_ylabel("Standardized feature 2")

dendrogram(Z, ax=axes[1])
axes[1].set_title("Ward linkage dendrogram")
axes[1].set_ylabel("Merge distance")

fig.tight_layout()
plt.show()

# Non-convex comparison: k-means versus DBSCAN
X_moons, _ = make_moons(
    n_samples=300,
    noise=0.05,
    random_state=42,
)
X_moons_train, X_moons_test = train_test_split(
    X_moons, test_size=0.2, random_state=42
)

moon_preprocess = Pipeline([("scaler", StandardScaler())])
X_moons_train_scaled = moon_preprocess.fit_transform(X_moons_train)
X_moons_test_scaled = moon_preprocess.transform(X_moons_test)

kmeans_labels = KMeans(
    n_clusters=2,
    n_init=10,
    random_state=42,
).fit_predict(X_moons_train_scaled)

dbscan = DBSCAN(eps=0.3, min_samples=5).fit(X_moons_train_scaled)
non_noise = dbscan.labels_ != -1

print(
    "K-means silhouette score: "
    f"{silhouette_score(X_moons_train_scaled, kmeans_labels):.3f}"
)
print(
    "DBSCAN silhouette score on non-noise points: "
    f"{silhouette_score(X_moons_train_scaled[non_noise], dbscan.labels_[non_noise]):.3f}"
)
print(f"DBSCAN noise points: {(~non_noise).sum()}")

fig, axes = plt.subplots(1, 2, figsize=(11, 5))

axes[0].scatter(
    X_moons_train_scaled[:, 0],
    X_moons_train_scaled[:, 1],
    c=kmeans_labels,
    cmap="viridis",
)
axes[0].set_title("K-means on moons")

axes[1].scatter(
    X_moons_train_scaled[:, 0],
    X_moons_train_scaled[:, 1],
    c=dbscan.labels_,
    cmap="viridis",
)
axes[1].set_title("DBSCAN on moons")

fig.tight_layout()
plt.show()

Common Pitfalls

  • Failing to scale features: Distance-based clusters can be dominated by features with large numerical values.

  • Choosing a linkage without inspecting the data: Single, complete, average, and Ward methods can produce materially different structures.

  • Selecting or tuning the cut arbitrarily: Compare several cuts and metrics, but preserve a final test set; repeatedly choosing the cut on one validation set can overfit it.

  • Evaluating against noisy DBSCAN labels: Exclude points labeled -1 before calculating silhouette score.

  • Treating clusters as classes: Clusters describe similarity under a chosen representation and metric; they do not establish causality or ground truth.

  • Ignoring scalability: Hierarchical clustering is usually unsuitable for very large datasets because of its quadratic resource requirements.

  • Preparing for follow-ups: Be ready to explain the linkage choices, how the cut affects cluster membership, the Euclidean restriction for Ward linkage, and how outliers or non-convex clusters affect validation.

33

How to use DBSCAN clustering in scikit-learn

DBSCAN is a density-based clustering algorithm that connects points through sufficiently dense neighborhoods, allowing it to find irregularly shaped clusters without specifying their number. Its main trade-off is its ability to identify non-convex clusters and noise versus its sensitivity to eps, min_samples, distance metrics, and feature scaling.

Read full answer

Applies toscikit-learn 1.5+ and Matplotlib 3.9+ (Python 3.12)

How DBSCAN Works

  • Core point: A point with at least min_samples points in its eps neighborhood, counting the point itself.

  • Border point: A point within eps of a core point but not itself a core point.

  • Noise point: A point that is neither core nor border; scikit-learn assigns it label -1.

  • Cluster: A density-connected group of core points and their associated border points.

Unlike k-means, DBSCAN does not require choosing the number of clusters beforehand. It is useful for curved clusters, customer segmentation, spatial data, and anomaly detection.

Parameters, Metrics, and Distances

  • eps: Neighborhood radius. Increasing it can merge clusters; decreasing it can turn points into noise.

  • min_samples: Minimum neighborhood density required for a core point. Larger values favor only dense regions.

  • Distance metric: Euclidean suits continuous numerical data; Manhattan can suit grid-like data. Cosine distance, 1 - cosine_similarity, is useful for text embeddings.

  • Scaling: Standardize numerical features when using distance-sensitive metrics such as Euclidean distance, or one large-range feature can dominate the distance calculation.

  • Silhouette score: Ranges from -1 to 1, with higher values indicating better separation. Exclude DBSCAN noise and require at least two non-noise clusters.

For algorithms that require k, use the elbow method, silhouette score, Davies-Bouldin index, and domain knowledge; Gaussian mixture models can also use BIC or AIC. Adjusted Rand index compares results with known labels, but clustering labels are not themselves ground truth.

Algorithm Comparison

Algorithm How it works / shapes Strengths Weaknesses and noise Choose k? Use when Typical scalability
k-means Centroid partitions with compact, convex clusters Fast and scalable Assumes compact clusters; no explicit noise Yes Well-separated numerical clusters O(n*k*d) per iteration
k-medoids Partitions around representative data points Interpretable and more outlier-resistant Costly; no explicit noise label Yes Mixed or outlier-sensitive data Often O(n²) per iteration or worse
Hierarchical Dendrogram using single, complete, average, or Ward linkage Shows nested structure Linkage-sensitive; no explicit noise Optional Small-to-medium datasets needing hierarchy Typically O(n²) time and memory
DBSCAN Density-connected arbitrary shapes Finds non-convex clusters and noise Sensitive to eps, scaling, and dimension No Irregular numerical or spatial clusters Near O(n log n) with spatial indexing; worst case O(n²)
HDBSCAN Hierarchical density-based clusters Handles varied density and noise More expensive; exact mode can use O(n²) memory No Clusters with different densities Exact mode often requires O(n²) work and memory
Gaussian mixture Probabilistic elliptical clusters Provides soft membership probabilities Assumes Gaussian structure; no explicit noise Yes Elliptical or overlapping clusters O(n*k*d) per iteration
Spectral Graph-based non-convex clusters Captures manifold-like shapes Expensive; no explicit noise label Yes Non-linear or graph-structured data O(n²) memory; dense eigendecomposition can take O(n³)

Single linkage can chain distant observations, complete linkage favors compact groups, average linkage balances the two, and Ward linkage minimizes the increase in within-cluster variance.

Code Example

This example compares k-means with DBSCAN on the two-moons dataset. K-means typically cuts across the curved clusters, while DBSCAN follows their boundaries and identifies noise.

import matplotlib.pyplot as plt
import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=300, noise=0.08, random_state=42)
X_train, _ = train_test_split(
    X, test_size=0.30, random_state=42, stratify=y
)

kmeans = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])

dbscan = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.15, min_samples=5, metric="euclidean")),
])

kmeans_labels = kmeans.fit_predict(X_train)
dbscan_labels = dbscan.fit_predict(X_train)
X_scaled = kmeans.named_steps["scale"].transform(X_train)

# Evaluate each result on the same scaled training data
kmeans_score = silhouette_score(X_scaled, kmeans_labels)
non_noise = dbscan_labels != -1
dbscan_clusters = np.unique(dbscan_labels[non_noise])
dbscan_score = (
    silhouette_score(X_scaled[non_noise], dbscan_labels[non_noise])
    if dbscan_clusters.size >= 2
    else float("nan")
)

print(f"K-means clusters: {len(np.unique(kmeans_labels))}")
print(f"DBSCAN clusters: {dbscan_clusters.size}")
print(f"K-means silhouette: {kmeans_score:.3f}")
print(f"DBSCAN silhouette: {dbscan_score:.3f}")

# DBSCAN usually has the higher silhouette score on this dataset
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

for ax, labels, title in zip(
    axes,
    [kmeans_labels, dbscan_labels],
    ["k-means", "DBSCAN"],
):
    for label in np.unique(labels):
        mask = labels == label
        is_noise = label == -1
        ax.scatter(
            X_scaled[mask, 0],
            X_scaled[mask, 1],
            color="gray" if is_noise else plt.cm.tab10(label % 10),
            marker="x" if is_noise else "o",
            label="Noise" if is_noise else f"Cluster {label}",
        )
    ax.set_title(title)
    ax.set_xlabel("Feature 1")
    ax.set_ylabel("Feature 2")
    ax.legend()

plt.tight_layout()
plt.show()

Common Pitfalls

  • Not scaling features: Standardize numerical features for Euclidean DBSCAN unless their scales have a meaningful common interpretation.

  • Missing values: DBSCAN does not accept NaN; impute inside the training-only pipeline.

  • Overfitting parameters: Do not choose eps and min_samples on the same data used to report final quality; use holdout data or stability analysis.

  • Invalid silhouette usage: Exclude noise and check that at least two clusters remain.

  • Ignoring interpretation: Profile cluster sizes, feature distributions, and stability; clusters are statistical groupings, not automatically meaningful categories.

  • Ignoring reproducibility: Set random seeds for generated data and stochastic baselines; DBSCAN itself is deterministic for fixed input.

  • Expecting new-point assignment: Standard scikit-learn DBSCAN is a fitting algorithm and does not provide a predict method for assigning unseen observations.

34

How do I calculate silhouette scores for different cluster counts in Python?

The silhouette score measures how well each point fits its assigned cluster compared with the nearest other cluster. It helps select the number of clusters by balancing within-cluster cohesion and between-cluster separation. Its main limitation is that it tends to favor compact, well-separated clusters and may score irregular shapes poorly.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12), NumPy 2.x

How It Works

For each point i, silhouette score calculates:

s(i) = (b(i) - a(i)) / max(a(i), b(i))

  • a(i): average distance from point i to other points in its own cluster.

  • b(i): smallest average distance from point i to points in another cluster.

  • s(i): silhouette value for point i, ranging from -1 to 1.

A value near 1 indicates a well-assigned point, while a value near -1 suggests it belongs to another cluster. silhouette_score returns the mean score, whereas silhouette_samples returns the score for every individual point.

For numeric features, Euclidean distance is common. Manhattan distance is less sensitive than Euclidean distance to large differences in individual coordinates, although it can still be affected by outliers. Cosine distance is often appropriate for text embeddings. Scaling is important whenever features use different units because large-magnitude features otherwise dominate the distance calculation.

Choosing the Number of Clusters

Several methods can be used together:

  • Silhouette score: Prefer a value that is high and stable, not merely the highest result by a tiny margin.

  • Elbow method: Look for the point where within-cluster sum of squares stops decreasing rapidly.

  • Davies-Bouldin index: Lower values indicate less overlap between clusters.

  • BIC and AIC: Useful for Gaussian mixture models, where they balance fit against model complexity.

  • Domain judgment: The final clusters must be useful and interpretable for the application.

When ground-truth labels are available, use external measures such as the adjusted Rand index (ARI). Clustering labels themselves are not ground truth and should be profiled by examining feature distributions, sizes, and practical meaning.

Algorithm Comparison

Algorithm Cluster shapes found Need to choose k? Noise handling Typical scalability
K-means Compact, roughly spherical clusters Yes No O(n * k * d) per iteration
K-medoids Groups around selected data points; shape depends on the metric and implementation Yes Implementation-dependent Often quadratic due to pairwise distances
Agglomerative Single linkage permits chaining; complete is compact; average balances both; Ward favors spherical clusters Optional, by cutting the tree No Naive implementation can be cubic
DBSCAN Arbitrary, non-convex shapes No; tune eps and min_samples Yes, using label -1 Neighborhood indexing improves large-data scaling
HDBSCAN Arbitrary shapes with hierarchical stability analysis No; tune min_cluster_size Yes The exact algorithm can be costly at scale; the generic algorithm is more efficient
Gaussian mixture Elliptical, Gaussian-shaped clusters Yes No Full covariance models become costly in high dimensions
Spectral clustering Complex graph, manifold, or non-convex structures Yes No Sparse graphs scale better than dense implementations

Code Example

import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split


def silhouette_scores_by_k(
    X,
    cluster_counts,
    test_size=0.25,
    random_state=42,
):
    """Return held-out silhouette scores for several cluster counts."""
    X = np.asarray(X, dtype=float)

    # Fit clustering algorithms only on the training partition.
    X_train, X_test = train_test_split(
        X,
        test_size=test_size,
        random_state=random_state,
    )

    scores = {}

    for k in cluster_counts:
        if not isinstance(k, (int, np.integer)):
            raise TypeError("Each cluster count must be an integer.")
        if k < 2 or k >= len(X_train):
            raise ValueError("k must be at least 2 and smaller than the sample count.")

        model = KMeans(
            n_clusters=k,
            n_init=10,
            random_state=random_state,
        )
        model.fit(X_train)

        # Assign held-out observations to the fitted clusters.
        test_labels = model.predict(X_test)
        represented_clusters = np.unique(test_labels).size

        # A valid silhouette requires 2 to n_samples - 1 represented labels.
        if not 2 <= represented_clusters <= len(X_test) - 1:
            raise ValueError(
                f"k={k} produced {represented_clusters} represented clusters; "
                f"silhouette requires between 2 and {len(X_test) - 1}."
            )

        scores[k] = float(silhouette_score(X_test, test_labels))

    return scores


X, _ = make_moons(
    n_samples=500,
    noise=0.08,
    random_state=42,
)

X_train, X_test = train_test_split(
    X,
    test_size=0.25,
    random_state=42,
)

scores = silhouette_scores_by_k(
    X,
    cluster_counts=[2, 3, 4],
    random_state=42,
)

print("K-means silhouette scores:")
for k, score in scores.items():
    print(f"k={k}: {score:.3f}")

# K-means with two clusters divides the moons incorrectly.
kmeans_labels = KMeans(
    n_clusters=2,
    n_init=10,
    random_state=42,
).fit_predict(X_train)

print(
    "K-means k=2 training silhouette:",
    f"{silhouette_score(X_train, kmeans_labels):.3f}",
)

# DBSCAN follows the non-convex shape and identifies noise.
dbscan_labels = DBSCAN(
    eps=0.3,
    min_samples=5,
).fit_predict(X_train)

non_noise = dbscan_labels != -1
dbscan_clusters = np.unique(dbscan_labels[non_noise])

# Noise points are excluded because label -1 is not a cluster.
if dbscan_clusters.size >= 2:
    dbscan_score = silhouette_score(
        X_train[non_noise],
        dbscan_labels[non_noise],
    )
    print(f"DBSCAN clusters found: {dbscan_clusters.size}")
    print(f"DBSCAN silhouette: {dbscan_score:.3f}")

Common Pitfalls

  • Ignoring feature scale: Standardize features with different units before distance-based clustering. Put the scaler and clusterer in a scikit-learn Pipeline so the scaler learns only from training data.

  • Selecting the maximum blindly: A slightly higher score does not guarantee more useful or interpretable clusters.

  • Including DBSCAN noise in silhouette calculation: Filter out label -1 before calling silhouette_score, while also reporting the amount of excluded noise.

  • Reporting training scores as generalization estimates: Fit on training data and evaluate assignments on held-out data; refit on all data only after selecting the final configuration.

  • Ignoring missing values or sparse data: Impute or otherwise preprocess missing values before clustering, and choose distance metrics appropriate for the data representation.

  • Skipping cluster profiling: Examine feature distributions, cluster sizes, outliers, and domain relevance; clustering produces partitions, not inherently meaningful classes.

  • Resource assumptions: scikit-learn clustering generally runs on the CPU. Specialized GPU implementations may help with very large embedding datasets, but ordinary interview-scale problems should use the standard CPU APIs.

35

How to implement Gaussian Mixture Model clustering with scikit-learn?

A Gaussian Mixture Model (GMM) represents data as a weighted mixture of Gaussian distributions, allowing overlapping clusters and probabilistic membership. It solves unsupervised grouping when clusters may differ in size, shape, and orientation. Its main trade-off is flexibility versus Gaussian assumptions, computational cost, and overfitting risk.

Read full answer

Applies toscikit-learn 1.5+ and Matplotlib 3.9+ (Python 3.12; NumPy 2.x)

How It Works

A GMM models the data distribution as:

p(x) = sum(k=1..K) pi_k * N(x | mu_k, Sigma_k)

  • x is an observation vector.

  • K is the number of Gaussian components.

  • pi_k is component k’s weight; the weights sum to one.

  • mu_k is component k’s mean vector.

  • Sigma_k is its covariance matrix.

  • N denotes a Gaussian probability density.

GaussianMixture uses expectation-maximization: the E-step computes each point’s responsibility for each component, and the M-step updates the weights, means, and covariances. predict() returns the component with the highest responsibility, while predict_proba() returns soft membership probabilities.

The covariance_type option controls covariance flexibility: spherical uses one variance per component, diag uses feature-specific variances, tied shares one covariance matrix, and full uses a separate covariance matrix per component. With full covariance, an iteration is roughly O(n * K * d² + K * d³), where n is the sample count and d is the feature count.

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Handles noise Typical scalability
k-means Convex, similar-size groups Yes No O(n * k * d) per iteration; highly scalable
k-medoids Compact groups centred on actual observations Yes No Often quadratic; medium-scale data
Agglomerative Depends on linkage; Ward favors compact groups Yes No Typically quadratic memory; medium-scale data
DBSCAN Arbitrary density-connected groups No Yes Spatial indexing is possible; quadratic worst case
HDBSCAN Flexible density clusters and hierarchy No Yes Medium-to-large data; in scikit-learn since 1.3
Gaussian mixture Overlapping elliptical groups Yes No Moderate; depends on components and dimensions
Spectral Nonlinear manifold structures Yes No Quadratic memory; small-to-medium data

Single linkage can follow chained, non-convex structures; complete and average linkage favour more compact groups, and Ward linkage specifically targets compact Euclidean clusters.

Choosing and Evaluating Clusters

  • GMM: Compare BIC or AIC across component counts; lower values are preferred.

  • Other methods: Use the elbow method, silhouette score, or Davies-Bouldin index.

  • Known labels: Adjusted Rand Index, normalized mutual information, or purity can compare clusters with reference labels.

  • Domain judgment: The selected count and cluster profiles must be scientifically or operationally meaningful.

  • Silhouette score: It ranges from -1 to 1, with higher values indicating better separation, but can favor convex clusters. Exclude DBSCAN points labeled -1.

Standardize numerical features before clustering because Euclidean-distance methods are scale-sensitive. For text embeddings, L2-normalizing vectors makes Euclidean distance directly related to cosine distance.

Code Example

import numpy as np
import matplotlib.pyplot as plt

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.mixture import GaussianMixture
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Reproducible, non-convex data; learned preprocessing uses training data only.
X, _ = make_moons(n_samples=500, noise=0.08, random_state=42)
X_train, X_test = train_test_split(
    X, test_size=0.25, random_state=42
)


def make_gmm(k):
    return Pipeline([
        ("scaler", StandardScaler()),
        ("gmm", GaussianMixture(
            n_components=k,
            covariance_type="full",
            n_init=10,
            random_state=42,
        )),
    ])


# Select the component count using BIC on training data.
candidates = list(range(1, 6))
bic_values = []

for k in candidates:
    model = make_gmm(k)
    model.fit(X_train)
    scaled_train = model.named_steps["scaler"].transform(X_train)
    bic_values.append(float(model.named_steps["gmm"].bic(scaled_train)))

best_k = candidates[int(np.argmin(bic_values))]
print("BIC by component count:", dict(zip(candidates, bic_values)))
print("Selected GMM components:", best_k)

# Fit the selected GMM.
gmm = make_gmm(best_k).fit(X_train)
gmm_train = gmm.predict(X_train)
print(
    "GMM test cluster counts:",
    np.unique(gmm.predict(X_test), return_counts=True),
)

# Fit comparison models on the same training data.
kmeans = Pipeline([
    ("scaler", StandardScaler()),
    ("kmeans", KMeans(n_clusters=2, n_init=10, random_state=42)),
]).fit(X_train)

dbscan = Pipeline([
    ("scaler", StandardScaler()),
    ("dbscan", DBSCAN(eps=0.10, min_samples=5)),
]).fit(X_train)

kmeans_train = kmeans.predict(X_train)
dbscan_train = dbscan.fit_predict(X_train)
X_train_scaled = gmm.named_steps["scaler"].transform(X_train)


def safe_silhouette(labels):
    valid = labels != -1
    cluster_count = np.unique(labels[valid]).size
    if cluster_count < 2 or valid.sum() <= cluster_count:
        return float("nan")
    return silhouette_score(X_train_scaled[valid], labels[valid])


for name, labels in [
    ("GMM", gmm_train),
    ("k-means", kmeans_train),
    ("DBSCAN", dbscan_train),
]:
    cluster_count = np.unique(labels[labels != -1]).size
    print(
        f"{name}: clusters={cluster_count}, "
        f"noise={(labels == -1).sum()}, "
        f"silhouette={safe_silhouette(labels):.3f}"
    )

# Visualize training assignments; DBSCAN has no out-of-sample predict method.
fig, axes = plt.subplots(1, 3, figsize=(12, 4))
for ax, (name, labels) in zip(
    axes,
    [("Gaussian Mixture", gmm_train),
     ("k-means", kmeans_train),
     ("DBSCAN", dbscan_train)],
):
    for label in np.unique(labels):
        mask = labels == label
        color = "lightgray" if label == -1 else plt.cm.tab10(label % 10)
        ax.scatter(
            X_train[mask, 0], X_train[mask, 1],
            c=[color], marker="o", s=18,
        )
    ax.set_title(name)

plt.tight_layout()
plt.show()

On moon-shaped data, k-means typically divides the curves with a straight boundary, while DBSCAN follows their density-connected shapes. The GMM may discover useful substructure, but its Gaussian assumption can misrepresent strongly curved clusters.

Common Pitfalls

  • Skipping scaling: Standardize features before applying distance-sensitive models.

  • Ignoring missing values: GMM and DBSCAN reject NaN values; impute within a training-fitted pipeline to avoid leakage.

  • Selecting k mechanically: BIC, silhouette, and elbow plots provide evidence rather than domain validation.

  • Treating clusters as ground truth: Labels are arbitrary; profile each cluster using feature distributions and domain behavior.

  • Ignoring soft membership: GMM probabilities contain uncertainty that hard labels discard.

  • Using incorrect DBSCAN metrics: Exclude label -1 from silhouette calculations and do not call DBSCAN for out-of-sample assignment.

36

How do I compare clustering algorithms in Python 3.12?

Clustering comparison evaluates how well different algorithms partition observations and which assumptions suit the data. It helps select an approach by balancing geometric flexibility, computational cost, parameter sensitivity, and interpretability. Because clusters have no inherent ground truth, conclusions should combine internal metrics, stability checks, cluster profiles, and domain judgment.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Noise handling Main trade-offs Typical scalability
K-means Spherical or convex clusters Yes No explicit noise label Fast and simple; assumes spherical, similar-variance clusters O(n * k * d) per iteration
K-medoids Usually compact; shape depends on the dissimilarity measure Yes No standard label Interpretable medoids and less sensitivity to outliers than centroids, but slower; not in core scikit-learn Exact PAM methods are quadratic or worse in n
Agglomerative Varies with single, complete, average, or Ward linkage No when using a distance threshold No explicit label Produces a hierarchy; single can chain, Ward favors spherical clusters, and complete tends to produce compact clusters Often O(n²) pairwise work; extra memory can be O(n) with tree implementations
DBSCAN Irregular clusters formed by dense connected regions No Labels noise as -1 Finds arbitrary shapes and outliers; sensitive to eps and min_samples Indexed searches can be near O(n log n) in low dimensions but can degrade to O(n²)
HDBSCAN Hierarchical density clusters with varying density No Labels noise as -1 More flexible than DBSCAN but more expensive; sensitive to density parameters Nearest-neighbor work can also degrade to O(n²)
Gaussian mixture Elliptical components with soft membership Component count must be supplied; compare candidate counts No standard noise label Probabilistic assignments and covariance estimates, but assumes Gaussian components Roughly O(n * k * d²) per iteration with full covariance
Spectral Non-convex or manifold-like groups Yes No explicit label Captures nonlinear structure but requires a similarity graph Usually O(n²) memory for the similarity matrix

Choosing and Evaluating Clusters

  • Elbow method: Plot within-cluster variation against k and look for a meaningful bend.

  • Silhouette score: Measures cohesion and separation; higher is better.

  • Davies-Bouldin score: Combines within-cluster dispersion with between-cluster separation; lower is better.

  • BIC and AIC: Compare Gaussian mixture models with different component counts; lower is preferred.

  • Domain judgment: Test whether cluster profiles are meaningful and actionable.

  • External metrics: With known labels, use adjusted Rand index, normalized mutual information, or V-measure. Such labels are useful for validation, but real clusters are not automatically ground truth.

  • Stability: Prefer solutions that remain similar under resampling or reasonable hyperparameter changes.

Code Example

This example compares algorithms on the two-moons dataset. HDBSCAN has been available in sklearn.cluster since scikit-learn 1.3. Standardization is fitted only on the training data through each pipeline, and silhouette and Davies-Bouldin scores exclude points labeled as noise.

import numpy as np
from sklearn.cluster import (
    AgglomerativeClustering,
    DBSCAN,
    GaussianMixture,
    HDBSCAN,
    KMeans,
    SpectralClustering,
)
from sklearn.datasets import make_moons
from sklearn.metrics import (
    adjusted_rand_score,
    davies_bouldin_score,
    silhouette_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_moons(n_samples=500, noise=0.06, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

algorithms = {
    "KMeans": KMeans(n_clusters=2, n_init=10, random_state=42),
    "Agglomerative-Ward": AgglomerativeClustering(
        n_clusters=2, linkage="ward"
    ),
    "DBSCAN": DBSCAN(eps=0.30, min_samples=5),
    "HDBSCAN": HDBSCAN(min_cluster_size=15, min_samples=5),
    "GaussianMixture": GaussianMixture(
        n_components=2, n_init=10, random_state=42
    ),
    "Spectral": SpectralClustering(
        n_clusters=2,
        affinity="nearest_neighbors",
        random_state=42,
    ),
}

for name, estimator in algorithms.items():
    pipeline = Pipeline([
        ("scale", StandardScaler()),
        ("cluster", estimator),
    ])

    train_labels = pipeline.fit_predict(X_train)
    valid = train_labels != -1
    labels = train_labels[valid]
    cluster_count = len(np.unique(labels))

    if cluster_count >= 2:
        X_scaled = pipeline.named_steps["scale"].transform(X_train)
        silhouette = silhouette_score(X_scaled[valid], labels)
        davies_bouldin = davies_bouldin_score(X_scaled[valid], labels)
    else:
        silhouette = float("nan")
        davies_bouldin = float("nan")

    train_ari = (
        adjusted_rand_score(y_train[valid], labels)
        if valid.any()
        else float("nan")
    )

    # Only these estimators provide general out-of-sample prediction.
    if isinstance(estimator, (KMeans, GaussianMixture)):
        test_labels = pipeline.predict(X_test)
        test_ari = adjusted_rand_score(y_test, test_labels)
        test_result = f"test_ARI={test_ari:.3f}"
    else:
        test_result = "test_ARI=n/a"

    print(
        f"{name:22} clusters={cluster_count}, "
        f"noise={np.sum(~valid)}, "
        f"silhouette={silhouette:.3f}, "
        f"davies_bouldin={davies_bouldin:.3f}, "
        f"train_ARI={train_ari:.3f}, {test_result}"
    )

# Intended behavior: DBSCAN recovers the crescent-shaped groups better
# than KMeans. Exact scores vary slightly by scikit-learn version.

Practical Guidance

  • Distance metrics: Euclidean distance squares coordinate differences, whereas Manhattan distance sums their absolute values; either can be dominated by large-magnitude features. Cosine distance compares direction and is useful for text embeddings.

  • Feature scaling: Standardize distance-based features before clustering. Impute missing values using parameters fitted only on training data.

  • Interpretation: Profile each cluster with feature summaries. A high silhouette score does not explain why a cluster matters.

  • Reproducibility: Set random_state for randomized estimators and retain fitted preprocessing parameters. Only K-means and Gaussian mixtures in this example support direct prediction for new observations.

Common Pitfalls

  • Assuming the visually obvious cluster count is always optimal.

  • Applying K-means to elongated, ring-shaped, or irregular groups.

  • Treating every DBSCAN or HDBSCAN point as a cluster member instead of identifying noise.

  • Tuning repeatedly against the same holdout data, eventually overfitting it.

  • Comparing arbitrary cluster IDs directly rather than using an external metric.

  • Assuming unsupervised cluster labels represent objective ground truth.

37

How do you normalize and scale features before clustering in Python?

Feature scaling places variables on comparable scales so that large-magnitude features do not dominate distance-based clustering. Standardization is generally a strong default for numeric data, while min-max scaling, robust scaling, and row normalization suit different distributions and use cases. The main trade-off is that scaling changes each feature’s influence, so outliers, sparsity, and domain meaning must be considered.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Normalization and Scaling Choices

  • Standardization (StandardScaler): Subtracts each feature’s training-set mean and divides by its training-set population standard deviation. The training data has mean zero and unit variance; transformed unseen data will not generally have exactly those statistics.

  • Min-max scaling (MinMaxScaler): Maps the training data to [0, 1]. Strong outliers can compress ordinary observations, and values transformed from unseen data can fall outside the training range.

  • Robust scaling (RobustScaler): Uses the median and interquartile range, making it less sensitive to extreme values, although sufficiently extreme outliers can still affect the IQR.

  • Row normalization (Normalizer): Scales each sample to unit L2 norm, emphasizing direction rather than magnitude. It is useful for document or embedding vectors.

  • Cosine distance: Measures angular similarity and is already insensitive to vector magnitude. L2-normalizing embeddings first is optional and often redundant.

  • Euclidean and Manhattan distance: Euclidean distance is common for k-means and hierarchical clustering; Manhattan distance sums absolute coordinate differences and can behave differently. Scaling is essential when features use different units because every coordinate contributes according to its magnitude.

Trees do not require feature scaling because they split on individual feature thresholds.

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Noise handling Typical scalability
k-means Compact, roughly spherical clusters Yes No O(n * k * d) per iteration
k-medoids Non-convex groups are possible; sensitive to medoid selection Yes No native support Often O(n²) distance work; moderate data
Agglomerative hierarchical Depends on linkage; Ward favors spherical clusters No fixed k; choose a dendrogram cut No native support Usually O(n²) pairwise work
DBSCAN Arbitrary shapes and dense groups No Yes, using noise labels O(n²) worst-case distance work
HDBSCAN Irregular clusters with varying density No Yes Hierarchical method; practical for medium-sized data
Gaussian mixture Elliptical or Gaussian-shaped clusters Yes No About O(n * k * d²) per iteration with full covariance
Spectral Connected, non-convex, or manifold-like groups Yes No Usually O(n²) memory

Scikit-learn supports single, complete, average, and Ward linkage; Ward requires Euclidean distance. sklearn.cluster.HDBSCAN has existed since scikit-learn 1.3.

Choose k using the elbow method, silhouette score, Davies-Bouldin score, GMM BIC/AIC, and domain knowledge. With trusted labels, the adjusted Rand index can compare clusters with those labels, although clusters are not automatically ground truth.

Code Example

import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler


def make_preprocessor(method: str = "standard") -> Pipeline:
    scalers = {
        "standard": StandardScaler,
        "minmax": MinMaxScaler,
        "robust": RobustScaler,
    }
    if method not in scalers:
        raise ValueError(f"Unknown method: {method!r}")
    return Pipeline(steps=[("scale", scalers[method]())])


# Two curved clusters; labels are used only to validate this synthetic example.
X, y = make_moons(n_samples=500, noise=0.05, random_state=42)
X_train, X_test, y_train, _ = train_test_split(
    X, y, test_size=0.30, random_state=42
)

# Fit preprocessing only on the training partition.
preprocessor = make_preprocessor("standard")
X_train_scaled = preprocessor.fit_transform(X_train)
X_test_scaled = preprocessor.transform(X_test)

kmeans = KMeans(n_clusters=2, random_state=42, n_init=10)
kmeans_labels = kmeans.fit_predict(X_train_scaled)

dbscan = DBSCAN(eps=0.15, min_samples=5)
dbscan_labels = dbscan.fit_predict(X_train_scaled)


def silhouette_for(labels: np.ndarray) -> float:
    # Silhouette is undefined for noise labels, so exclude DBSCAN noise.
    keep = labels != -1
    if np.unique(labels[keep]).size < 2:
        return float("nan")
    return silhouette_score(X_train_scaled[keep], labels[keep])


def ari_for(labels: np.ndarray) -> float:
    keep = labels != -1
    return adjusted_rand_score(y_train[keep], labels[keep])


print("KMeans silhouette:", round(silhouette_for(kmeans_labels), 3))
print("DBSCAN silhouette:", round(silhouette_for(dbscan_labels), 3))
print("KMeans adjusted Rand index:", round(ari_for(kmeans_labels), 3))
print("DBSCAN adjusted Rand index:", round(ari_for(dbscan_labels), 3))
print(
    "DBSCAN clusters and noise points:",
    np.unique(dbscan_labels[dbscan_labels != -1]).size,
    int((dbscan_labels == -1).sum()),
)
print("Held-out scaled shape:", X_test_scaled.shape)

On these moons, k-means tends to split each curved group, while DBSCAN can recover the two dense regions. DBSCAN noise points are excluded from both validation metrics.

Best Practices

  • Fit scalers, imputers, and feature selectors only on training data during evaluation; after evaluation, a final clustering pipeline may be refitted on all unlabeled data.

  • Put imputation and scaling in a Pipeline so every split receives the same fitted transformation.

  • Check missing values because many clustering estimators cannot process NaN.

  • Use RobustScaler for strong outliers and inspect feature distributions before choosing a range-based scaler.

  • Profile each cluster using feature summaries and test stability across seeds or resampled datasets.

  • Conventional scikit-learn clustering runs on CPU; specialized GPU implementations may help with very large datasets.

Common Pitfalls

  • Data leakage: Fitting a scaler before splitting exposes test statistics to the model.

  • Ignoring geometry: Scaling does not make k-means suitable for curved structures such as moons.

  • Forcing k: Density-based methods can discover variable cluster counts and noise.

  • Blind min-max scaling: Outliers can severely compress the useful range.

  • Misinterpreting clusters: Cluster IDs are arbitrary, not classes, causes, or ground truth; profile and validate them before assigning meaning.

38

How do I use a custom distance metric in scikit-learn clustering?

A custom distance metric is a Python callable that measures dissimilarity between observations using application-specific rules. In scikit-learn, it can replace Euclidean distance in estimators such as DBSCAN and AgglomerativeClustering, solving problems where ordinary geometric distance does not reflect similarity. The main trade-off is flexibility versus greater implementation complexity, tuning sensitivity, and potentially higher computational cost.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12, NumPy 2.x)

How It Works

For DBSCAN and AgglomerativeClustering, pass a callable through metric=. Scikit-learn calls it with two one-dimensional feature vectors and expects a single nonnegative floating-point distance:

distance(x, y) = sum(abs(x_i - y_i))

This is Manhattan distance: it adds the absolute difference between corresponding features. Unlike Euclidean distance, it does not take the square root of the summed squared differences.

The callable must not be a pairwise function returning a matrix. If an estimator accepts only precomputed distances, calculate a complete symmetric matrix with a zero diagonal and pass it using metric="precomputed" instead.

Custom callables are not supported uniformly by every clustering estimator. For example, DBSCAN and AgglomerativeClustering accept them, whereas scikit-learn’s KMeans only supports Euclidean distance for its clustering objective. Callable metrics can also force brute-force pairwise distance calculations, making computation approximately O(n²) instead of using optimized Euclidean backends.

Feature scaling is critical when features use different units because larger-range features dominate the distance. Standardization or min-max scaling is commonly needed for k-NN, SVM, PCA, k-means, and gradient-based methods, but not for trees. Cosine distance is often more appropriate for text embeddings because it compares vector direction rather than magnitude.

Algorithm Choices

Method How It Works and Cluster Shapes Choose k? Handles Noise Strengths, Weaknesses, and Scale
k-means Updates arithmetic centroids; favors compact, convex clusters Yes No Fast and scalable; fails on moons or irregular shapes; O(n*k*d) per iteration
k-medoids Uses actual observations as centers; supports non-Euclidean dissimilarity Yes No More robust to unusual points than centroids; often requires O(n²) distance storage
Agglomerative clustering Merges observations hierarchically; shape depends on metric and linkage Usually cut the tree No Interpretable dendrogram; linkage-sensitive and typically O(n²) memory
DBSCAN Connects density-reachable points; finds arbitrary shapes No Yes Effective for noise and non-convex clusters; sensitive to eps and min_samples
HDBSCAN Builds a density hierarchy; supports clusters with different densities No Yes More adaptive than DBSCAN; scikit-learn’s implementation is available since 1.3
Gaussian mixture Models clusters as probability distributions; finds elliptical clusters Yes Not explicitly Provides soft assignments; covariance estimation can be expensive
Spectral clustering Uses a similarity graph and eigenvectors; captures complex shapes Usually No Handles filaments and non-convex groups; requires large graph matrices and is sensitive to k

Agglomerative single linkage can produce chained clusters, complete linkage favors compact clusters, average linkage balances both, and Ward linkage minimizes within-cluster variance but supports only Euclidean-style distances.

Choosing and Evaluating Clusters

The elbow method looks for a change in within-cluster variation, while the silhouette score measures separation relative to cluster compactness. The Davies-Bouldin index rewards clusters that are compact and well separated. For Gaussian mixtures, BIC and AIC balance fit against model complexity. Domain knowledge should ultimately determine whether the clusters represent meaningful groups.

When reference labels exist, the adjusted Rand index (ARI) measures external agreement while remaining insensitive to cluster-label permutations. Such labels are evaluation references, not automatically ground truth. Each cluster should also be profiled using feature distributions, sizes, and business or scientific meaning.

Code Example

The following implements Manhattan distance, uses it with DBSCAN, and compares the result with k-means on the two-moons dataset.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score


def manhattan_distance(x, y):
    """Return the Manhattan distance between two feature vectors."""
    return float(np.abs(x - y).sum())


X, reference_labels = make_moons(
    n_samples=300,
    noise=0.05,
    random_state=42,
)

# KMeans tends to split the moons rather than follow their curved shapes.
kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans_labels = kmeans.fit_predict(X)

# Scikit-learn calls this callable with two one-dimensional vectors.
dbscan = DBSCAN(
    eps=0.30,
    min_samples=5,
    metric=manhattan_distance,
)
dbscan_labels = dbscan.fit_predict(X)

print(
    f"KMeans: silhouette={silhouette_score(X, kmeans_labels):.3f}, "
    f"ARI={adjusted_rand_score(reference_labels, kmeans_labels):.3f}"
)

# DBSCAN uses -1 for noise, which silhouette_score does not accept.
non_noise = dbscan_labels != -1
cluster_count = len(np.unique(dbscan_labels[non_noise]))

print(
    f"DBSCAN: clusters={cluster_count}, "
    f"noise={np.count_nonzero(~non_noise)}, "
    f"silhouette={silhouette_score(X[non_noise], dbscan_labels[non_noise]):.3f}, "
    f"ARI={adjusted_rand_score(reference_labels[non_noise], dbscan_labels[non_noise]):.3f}"
)

The expected qualitative result is that DBSCAN follows the two curved groups, while k-means produces a less meaningful split.

Common Pitfalls

  • Incorrect callable contract: Return one scalar distance for two one-dimensional vectors, not a distance matrix.

  • Unscaled features: Large-range features can dominate the result; scale before clustering when units differ.

  • Data leakage: Fit scalers, TF-IDF vectorizers, or embeddings on training data only when clustering is part of a larger evaluation pipeline.

  • Ignoring noise: Exclude DBSCAN label -1 before calculating silhouette or external cluster metrics.

  • Treating clusters as ground truth: Cluster validity depends on stability and domain interpretation.

  • Over-tuning on one metric: Combine silhouette, Davies-Bouldin, stability across samples, and domain knowledge.

Interviewers commonly ask which metric is appropriate, how eps or k is selected, whether the data requires scaling, how noise is handled, and how cluster stability can be measured.

39

How do you perform customer segmentation with cluster analysis?

Cluster analysis groups customers according to shared characteristics without predefined labels, such as purchasing behavior, value, and engagement. It supports targeted campaigns and customer strategies, but the main trade-off is that flexible algorithms may find more complex groups that still require business interpretation and validation.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Segmentation Workflow

  1. Define the objective: Decide whether the segments will support retention, loyalty campaigns, product recommendations, budgeting, or churn prevention.

  2. Create customer-level features: Aggregate transactions into features such as RFM—recency, frequency, and monetary value—as well as average order value, margin, tenure, return rate, discounts, preferred categories, channels, and engagement.

  3. Prepare the data: Remove duplicates, correct customer identifiers, handle missing values, review extreme spend values, and transform highly skewed monetary variables with log1p.

  4. Fit candidate algorithms: Try several reasonable choices, select parameters using validation metrics and stability checks, and avoid choosing the algorithm only from visual appearance.

  5. Profile and operationalize: Examine cluster size, revenue, margins, preferences, and behavior; assign meaningful business names; validate profitability; and create a repeatable process for assigning new customers.

Algorithm Comparison

Algorithm Typical cluster shapes Choose k? Noise handling Typical scalability
k-means Compact, roughly spherical groups; can split irregular structures Yes None O(n × k × d) per iteration
k-medoids Usually compact groups, although results depend on the dissimilarity measure Yes No explicit noise cluster; usually less sensitive to extreme values than k-means Often needs an O(n²) distance matrix; optimization can be expensive
Agglomerative Single linkage can chain loose points; complete produces compact groups; average is balanced; Ward favors spherical Euclidean groups Cut a dendrogram None Worst-case O(n³) time and O(n²) memory
DBSCAN Arbitrary density-connected shapes No; tune eps and min_samples Sparse points receive label -1 Indexed implementations are often roughly O(n log n); worst case O(n²)
HDBSCAN Arbitrary shapes with varying density No; tune min_cluster_size and related parameters Identifies outliers sklearn.cluster.HDBSCAN has existed since scikit-learn 1.3; optimized runs are typically near-linear, with possible quadratic worst cases
Gaussian mixture Elliptical groups, including correlated Gaussian shapes Usually use BIC or AIC None Diagonal covariance: about O(n × k × d) per iteration; full covariance: about O(n × k × d²)
Spectral clustering Non-convex, connected structures Yes None Dense implementations commonly use O(n²) memory and up to O(n³) eigendecomposition

Features and Model Selection

  • Distance metrics: Euclidean distance measures straight-line separation; Manhattan distance sums coordinate-wise differences and is less dominated by one extreme difference. Cosine distance, equal to 1 - cosine similarity, compares direction rather than magnitude and is useful for text or product embeddings.

  • Scaling: Standardize features for distance-based methods such as k-means, DBSCAN, agglomerative clustering, and spectral clustering; otherwise a large-range variable can determine the clusters. Tree-based methods generally do not require scaling for their splits.

  • Missing values and outliers: Impute missing values where appropriate, investigate extreme customers, and consider robust methods or explicit noise labels.

  • Choosing the number of clusters: Use the elbow method, silhouette score, Davies-Bouldin index, domain judgment, and stability across resamples or later cohorts. Exclude DBSCAN noise points from silhouette calculation. For Gaussian mixtures, compare BIC or AIC across component counts.

  • External validation: If trustworthy labels exist, use metrics such as the adjusted Rand index, which is invariant to cluster-label permutations. Clusters remain descriptive models rather than verified ground truth.

Code Example

This example uses a train partition to fit the scalers and clustering models, then assigns held-out customers to the fitted clusters. On the curved data, k-means may divide each structure, whereas DBSCAN can recover the density-connected groups and identify noise.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import NearestNeighbors
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
X, _ = make_moons(n_samples=500, noise=0.08, random_state=42)

# Add isolated observations to demonstrate DBSCAN's noise handling.
outliers = rng.uniform(low=8.0, high=12.0, size=(20, 2))
X = np.vstack([X, outliers])

X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=42
)

kmeans_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, random_state=42, n_init=10)),
])

dbscan_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("cluster", DBSCAN(eps=0.25, min_samples=5)),
])

# Fit preprocessing and clustering on training data only.
kmeans_pipeline.fit(X_train)
dbscan_pipeline.fit(X_train)

X_train_scaled = kmeans_pipeline.named_steps["scaler"].transform(X_train)
X_test_scaled = kmeans_pipeline.named_steps["scaler"].transform(X_test)

kmeans_train_labels = kmeans_pipeline.named_steps["cluster"].predict(
    X_train_scaled
)
dbscan_train_labels = dbscan_pipeline.named_steps["cluster"].labels_
kmeans_test_labels = kmeans_pipeline.named_steps["cluster"].predict(
    X_test_scaled
)
# DBSCAN has no predict(): assign each new customer to its nearest core sample within eps.
db = dbscan_pipeline.named_steps["cluster"]
dist, idx = NearestNeighbors(n_neighbors=1).fit(db.components_).kneighbors(X_test_scaled)
dbscan_test_labels = np.where(
    dist[:, 0] <= db.eps, db.labels_[db.core_sample_indices_][idx[:, 0]], -1
)

def valid_silhouette(features, labels):
    keep = labels != -1
    if len(np.unique(labels[keep])) < 2:
        return float("nan")
    return silhouette_score(features[keep], labels[keep])

print("K-means silhouette:",
      valid_silhouette(X_train_scaled, kmeans_train_labels))
print("DBSCAN silhouette:",
      valid_silhouette(X_train_scaled, dbscan_train_labels))
print("DBSCAN test noise points:",
      np.sum(dbscan_test_labels == -1))

Common Pitfalls

  • Data leakage: Do not use purchases or features generated after the segmentation date.

  • Not scaling: Distance-based clusters may otherwise reflect units rather than behavior.

  • Including noise in silhouette scoring: DBSCAN label -1 is not a cluster.

  • Over-tuning on one holdout: Repeated selection against the same data produces optimistic results.

  • Ignoring outliers: Extreme customers can strongly influence k-means centroids and medoids.

  • Ignoring business value: A mathematically separable cluster is useful only when it supports an actionable decision.

Common follow-up questions concern choosing k, testing temporal stability, assigning new customers, identifying the features that drive membership, and comparing clustering with rule-based segments.

40

How can cluster analysis support image segmentation?

Cluster analysis supports image segmentation by grouping pixels or extracted feature vectors into regions that are similar under a chosen representation. It can separate objects, tissues, textures, or background regions without requiring pixel-level training labels. The main trade-off is that clusters are not automatically meaningful: results depend heavily on the features, distance metric, algorithm parameters, and domain interpretation.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

How It Supports Segmentation

A typical workflow converts an image into a set of feature vectors, clusters them, and reshapes the cluster assignments back into a segmentation mask.

  • Pixel features: RGB, HSV, or Lab color values support simple color-based segmentation.

  • Texture features: Local binary patterns, Gabor filters, or filter-bank responses distinguish surfaces with similar colors.

  • Spatial features: Combining color with coordinates encourages nearby pixels to receive similar labels.

  • Deep embeddings: Pretrained CNN or vision-transformer features often capture semantic similarity better than raw color, although they require more memory.

  • Superpixels: Algorithms such as SLIC reduce the number of items to cluster while preserving image boundaries.

For example, a road-scene image might yield clusters for vegetation, buildings, roadway, and sky. Each cluster should then be profiled and interpreted; cluster numbers are arbitrary labels, not ground truth.

Algorithm Comparison

Algorithm Cluster shapes found Choose k? Handles noise? Typical scale and complexity
k-means Convex, roughly spherical clusters Yes No High scalability; approximately O(n × k × d) per iteration
k-medoids Shapes supported by the chosen distance metric Yes No Medium to low scalability; often O(n² × d) similarity work
Agglomerative clustering Depends on linkage: single bridges, complete favors compact clusters, average balances both, and Ward favors spherical clusters Yes, by cutting the tree No Memory-intensive; commonly O(n²)
DBSCAN Arbitrary-shaped, density-connected clusters No Yes Medium scalability; indexed searches are often near O(n log n)
HDBSCAN Arbitrary shapes and clusters with different densities No Yes Medium to high scalability; hierarchical density extraction
Gaussian mixture model Elliptical clusters with soft membership probabilities Usually; BIC/AIC can select it Not explicitly Approximately O(n × k × d) per iteration
Spectral clustering Connected or non-convex structures represented by a graph Yes Not explicitly Often expensive; dense formulations may require O(n³)

scikit-learn.cluster.HDBSCAN, available since scikit-learn 1.3, exposes controls such as min_cluster_size and cluster_selection_epsilon.

Features, Distances, and Cluster Selection

  • Distance metrics: Euclidean distance suits continuous color and texture features; Manhattan distance can be more robust to outliers; Mahalanobis distance accounts for correlated features; cosine distance is useful for normalized image or text embeddings.

  • Scaling: Standardize features when their units or ranges differ significantly. Otherwise, a feature with a large numeric range can dominate Euclidean or k-means distances.

  • Cluster count: Use the elbow method, silhouette score, Davies-Bouldin index, BIC/AIC for Gaussian mixtures, and domain judgment. No single metric is definitive.

  • External evaluation: When segmentation labels are available, use adjusted Rand index, normalized mutual information, or IoU-based region measures.

  • Large images: Consider superpixels, MiniBatchKMeans, approximate nearest-neighbor methods, or GPU-accelerated embedding extraction.

Code Example

The first part segments a synthetic image by clustering its RGB pixels. The second demonstrates why k-means can struggle with curved structures while DBSCAN follows non-convex clusters, then computes silhouette scores.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.model_selection import (
    StratifiedShuffleSplit,
    train_test_split,
)

# Segment an image by clustering pixel colors.
yy, xx = np.mgrid[:64, :64]
image = np.zeros((64, 64, 3), dtype=np.float64)
image[:] = [0.05, 0.05, 0.05]
image[(xx - 18) ** 2 + (yy - 18) ** 2 < 100] = [1.0, 0.0, 0.0]
image[(xx >= 38) & (xx < 52) & (yy >= 38) & (yy < 52)] = [0.0, 1.0, 0.0]

features = image.reshape(-1, 3)
indices = np.arange(len(features))
train_idx, test_idx = train_test_split(
    indices, test_size=0.3, random_state=42
)

pixel_model = KMeans(n_clusters=3, n_init=10, random_state=42)
pixel_model.fit(features[train_idx])

pixel_labels = np.empty(len(features), dtype=np.int32)
pixel_labels[train_idx] = pixel_model.predict(features[train_idx])
pixel_labels[test_idx] = pixel_model.predict(features[test_idx])

segmentation = pixel_labels.reshape(64, 64)
print("Pixel segment count:", np.unique(pixel_labels).size)

# Compare k-means and DBSCAN on curved, non-convex clusters.
X, truth = make_moons(n_samples=600, noise=0.07, random_state=42)
splitter = StratifiedShuffleSplit(
    n_splits=1, test_size=0.3, random_state=42
)
train_idx, test_idx = next(splitter.split(X, truth))
X_train, X_test = X[train_idx], X[test_idx]

kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
km_train = kmeans.fit_predict(X_train)
km_test = kmeans.predict(X_test)

dbscan = DBSCAN(eps=0.30, min_samples=12)
db_train = dbscan.fit_predict(X_train)
db_test = dbscan.predict(X_test)

# Restore predictions to the original row order before evaluation.
km_labels = np.empty(len(X), dtype=np.int32)
km_labels[train_idx] = km_train
km_labels[test_idx] = km_test

db_labels = np.empty(len(X), dtype=np.int32)
db_labels[train_idx] = db_train
db_labels[test_idx] = db_test

def report(name, labels, expected):
    keep = labels != -1  # Exclude DBSCAN noise from cluster metrics.
    classes = np.unique(labels[keep])

    if classes.size < 2 or np.sum(keep) <= 2:
        print(f"{name}: silhouette unavailable")
        return

    silhouette = silhouette_score(X[keep], labels[keep])
    ari = adjusted_rand_score(expected[keep], labels[keep])
    print(
        f"{name}: clusters={classes.size}, "
        f"silhouette={silhouette:.3f}, ARI={ari:.3f}"
    )

report("k-means", km_labels, truth)
report("DBSCAN", db_labels, truth)

Common Pitfalls

  • Ignoring feature scaling: Large-range features can dominate distance calculations.

  • Selecting k only by silhouette: The score may favor compact clusters even when they merge real objects or split one object.

  • Treating clusters as ground truth: Always inspect and characterize each cluster before assigning it a semantic meaning.

  • Ignoring spatial connectivity: Pixel-level clustering can produce speckled masks; connected-component filtering or superpixels can improve coherence.

  • Using unsuitable evaluation metrics: Accuracy is not meaningful for unlabeled segmentation; use region metrics, external indices, or domain-specific boundary criteria.

  • Frequent interview follow-ups: How would you select features, choose a distance metric, handle noise, improve scalability, or validate that each cluster represents a meaningful region?

41

How can clustering organize large document collections into topics?

Topic clustering groups documents by the similarity of their text representations so related documents can be summarized under interpreted topic labels. It makes large collections easier to navigate and retrieve, but the main trade-off is scalable computation versus semantic interpretability because clusters are not ground truth and require validation and human profiling.

Read full answer

Applies toscikit-learn 1.5+ and NumPy 2.x (Python 3.12)

Recommended Workflow

  1. Represent the documents

  2. Use sparse TF-IDF vectors for lexical similarity.

  3. Use pretrained sentence embeddings for semantic similarity.

  4. Select a distance and scale appropriately

  5. Euclidean distance suits dense, similarly scaled numerical vectors.

  6. Cosine distance is common for text embeddings because it compares direction rather than magnitude.

  7. L2-normalize embeddings before Euclidean algorithms intended to model spherical or cosine geometry; direct cosine similarity is already scale-invariant.

  8. Standardize raw feature dimensions when required, but do not blindly standardize sparse TF-IDF matrices or pretrained embeddings.

  9. Cluster at scale

  10. For large collections, train MiniBatchKMeans on a representative subset, then use predict to assign remaining documents in batches.

  11. Profile cluster sizes and quality because density-based methods can become expensive on large datasets.

  12. Interpret the clusters

  13. Inspect representative documents, top TF-IDF terms, named entities, and other relevant metadata.

  14. Assign an interpretable topic label and summary, then merge nearly identical clusters or split overly broad ones when necessary.

Algorithm Comparison

Algorithm Cluster shapes and strengths Noise handling Choose k? Scalability and weaknesses
k-means Fast for compact, convex clusters None Yes O(n*k*d) per iteration; sensitive to scale, initialization, and outliers
k-medoids Uses real data points as centers and is less outlier-sensitive than k-means None Yes Often O(n²) distance work; less natural for irregular shapes than density methods
Agglomerative clustering Hierarchical results can be cut at different levels None Tree cut Approximately O(n²) memory; expensive at large scale
DBSCAN Finds irregular, dense regions Assigns sparse points to -1 No Sensitive to eps and min_samples; scikit-learn can take O(n²) worst-case time
HDBSCAN Finds irregular clusters across density levels Supports noise No Costly on large datasets; min_cluster_size requires tuning
Gaussian mixture Soft assignments for overlapping elliptical clusters None Yes Cost grows with n, k, and d; covariance estimation adds substantial overhead
Spectral clustering Captures nonlinear manifold structures None Usually Typically O(n²) memory plus eigendecomposition cost

For agglomerative clustering, single linkage favors connected shapes, complete linkage favors compact clusters, average linkage balances both, and Ward linkage minimizes within-cluster variance and generally favors spherical clusters.

Choosing the Number of Topics

  • Domain judgment: The most important criterion because topics must be meaningful and actionable.

  • Elbow method: Inspect the k-means inertia curve for a clear reduction in improvement as k increases.

  • Silhouette score: Higher values generally indicate better separation, but unusual shapes can score poorly.

  • Davies-Bouldin index: Lower values indicate better separation relative to cluster spread.

  • BIC or AIC: Compare Gaussian mixture models; lower values favor a more parsimonious probabilistic model.

  • Stability: Rerun clustering on resampled data and check whether topic profiles remain consistent.

  • External metrics: With reliable labels, use adjusted Rand index, normalized mutual information, homogeneity, or completeness.

For DBSCAN and HDBSCAN, evaluate the noise proportion separately and calculate silhouette only on non-noise points. Geometry metrics alone cannot establish semantic usefulness.

Code Example

This example uses two-dimensional points as stand-in document embeddings. K-means partitions the curved moons with straight boundaries, while DBSCAN follows their nonlinear density structure.

import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import adjusted_rand_score, silhouette_score

# Deterministic stand-in for two-dimensional document embeddings.
X, true_topics = make_moons(
    n_samples=500,
    noise=0.05,
    random_state=42,
)

# K-means partitions the curved structure into Voronoi regions.
kmeans = KMeans(n_clusters=2, n_init=10, random_state=42)
kmeans_labels = kmeans.fit_predict(X)
kmeans_silhouette = silhouette_score(X, kmeans_labels)

# DBSCAN follows dense connected regions and identifies sparse noise.
dbscan = DBSCAN(eps=0.3, min_samples=5)
dbscan_labels = dbscan.fit_predict(X)

non_noise = dbscan_labels != -1
non_noise_labels = dbscan_labels[non_noise]
dbscan_clusters = len(np.unique(non_noise_labels))

if dbscan_clusters >= 2:
    dbscan_silhouette = silhouette_score(X[non_noise], non_noise_labels)
else:
    dbscan_silhouette = float("nan")

print(f"K-means silhouette: {kmeans_silhouette:.3f}")
print(f"DBSCAN silhouette: {dbscan_silhouette:.3f}")
print(f"DBSCAN clusters: {dbscan_clusters}")
print(f"DBSCAN noise points: {np.sum(~non_noise)}")
print(f"K-means adjusted Rand index: {adjusted_rand_score(true_topics, kmeans_labels):.3f}")

# Exclude noise rather than treating label -1 as a third topic.
print(
    "DBSCAN adjusted Rand index (non-noise): "
    f"{adjusted_rand_score(true_topics[non_noise], non_noise_labels):.3f}"
)

Common Pitfalls

  • Failing to scale or normalize features: This can make distance-based algorithms prioritize high-magnitude dimensions.

  • Ignoring document context: Metadata, section, author, or time may be essential for useful topics.

  • Over-trusting metrics: Silhouette measures geometry, not semantic usefulness or factual coherence.

  • Misusing noise labels: Do not silently count DBSCAN noise as an ordinary cluster in external metrics.

  • Confusing clusters with classes: A document may belong to several topics, and cluster labels are not ground truth.

  • Skipping profile review and drift monitoring: Inspect cluster profiles before publication and monitor their stability over time.

42

What clustering strategy identifies similar geographical regions?

A spatial clustering strategy groups geographic units according to location, attribute similarity, or both. Density-based methods such as DBSCAN and HDBSCAN are useful for irregularly shaped regions and noise detection, while graph-based regionalization is preferable when every region must be geographically contiguous. The main trade-off is flexibility versus predictable boundaries, computational cost, and explicit control over the number of regions.

Read full answer

Applies toclustering concepts; scikit-learn 1.5+ (Python 3.12) for DBSCAN, HDBSCAN, and the named metrics

Recommended Strategy

  1. Define the unit of analysis: Use consistent geographic units, such as store locations, grid cells, census tracts, or polygons.

  2. Represent polygons appropriately: Point-based methods can use centroids or representative points, but this may discard shape and contiguity information. Use an adjacency graph when region boundaries or neighboring units matter.

  3. Prepare the features: Project latitude and longitude into an appropriate local or regional coordinate system. Keep geographic coordinates separate from demographic, environmental, or economic attributes so their relative influence can be controlled.

  4. Handle distance-sensitive features: Impute missing values, standardize continuous attributes when their units differ, and consider robust scaling for heavy-tailed measurements. Do not automatically scale coordinates, because doing so changes the relative geometry.

  5. Select and validate the method: Check cluster profiles, map boundaries, inspect small or mixed groups, and test stability under parameter changes, resampling, or alternative distance weights.

Algorithm Comparison

Method Cluster shapes found Choose k? Noise handling Strengths, weaknesses, and use Typical scale or complexity
k-means Compact, roughly spherical Yes None Fast and interpretable; fails on curved or unequal-density shapes O(n*k*d) per iteration
k-medoids Depends on the distance and medoid placement Yes No explicit label Less sensitive to extreme values than means; not included in scikit-learn core PAM implementations are commonly O(n^2) or worse
Agglomerative: single, complete, average, Ward Depends on linkage; Ward favors compact clusters Usually, when cutting the dendrogram None Supports nested regional structures and domain-defined merge rules Usually O(n^2) time and memory
DBSCAN Arbitrary shapes defined by density No Yes, label -1 Good for separated regions and outliers; sensitive to eps, scaling, and varying density Can approach O(n log n) with tree indexing but degrade toward O(n^2)
HDBSCAN Flexible, variable-density clusters No Yes Handles varying densities and cluster stability; parameter tuning remains important Often O(n log n) to O(n^2), depending on neighbor search and density
Gaussian mixture Elliptical, probabilistic clusters Yes No explicit noise label Provides soft assignments and BIC/AIC selection; assumes Gaussian structure Full covariance is approximately O(n*k*d^2) per iteration
Spectral Non-convex or graph-based regions Usually No explicit label Effective for connected structures; expensive and sensitive to graph construction Often O(n^2) memory

Agglomerative linkage controls how regions merge: single linkage can produce chaining, complete linkage favors compact groups, average linkage is a compromise, and Ward linkage favors variance-minimizing merges under Euclidean distance.

Distance, Scaling, and Parameter Selection

  • Euclidean distance works well for projected coordinates and standardized numeric attributes.

  • Haversine or geodesic distance is more appropriate for raw latitude and longitude, although combining it directly with attributes still requires explicit weighting.

  • Manhattan distance can fit grid-based movement, while cosine distance is useful for text embeddings when direction matters more than magnitude.

  • For mixed spatial and attribute data, tune the relative weight of each distance component rather than assuming that one coordinate unit equals one standardized attribute unit.

For methods requiring k, compare the elbow method, silhouette score, and Davies-Bouldin index. Gaussian mixture models can also be compared with BIC and AIC. DBSCAN uses eps and min_samples; HDBSCAN uses min_cluster_size and can use cluster_selection_epsilon. sklearn.cluster.HDBSCAN has existed since scikit-learn 1.3 and is available in scikit-learn 1.5+.

If trusted labels exist, external measures such as the adjusted Rand index or normalized mutual information can be used. Always combine statistical measures with domain judgment.

Common Pitfalls

  • Ignoring spatial contiguity: DBSCAN and HDBSCAN find dense groups, not necessarily connected administrative regions.

  • Poor scaling: Unscaled attributes can dominate the distance calculation, while indiscriminately scaling coordinates changes geographical geometry.

  • Hiding noise: Excluding DBSCAN noise points can inflate silhouette scores; report the noise proportion and inspect those locations.

  • Overfitting parameters: Repeatedly tuning against the same map or validation set produces unstable regions. Use stability checks and domain review.

  • Treating clusters as ground truth: Clusters are statistical summaries, not inherent classes. Profile each cluster and confirm that its attributes and geography support a real operational decision.

43

How can cluster analysis improve a recommendation system?

Cluster analysis groups users, items, or interaction contexts into behaviorally similar segments, allowing a recommender to generate candidates more efficiently and reveal preference patterns. Its main trade-off is that clusters are coarse, potentially unstable proxies, so cluster-based recommendations can reduce personalization and diversity.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

A common recommendation workflow is:

  1. Build representations: Represent users and items with interactions, ratings, content features, or learned embeddings.

  2. Prepare features: Handle missing values, encode categorical variables, and scale numerical features when the clustering algorithm uses distance-based geometry.

  3. Create clusters: Cluster users to model shared preferences or items to identify substitutes and complementary products.

  4. Profile clusters: Summarize dominant categories, activity levels, item popularity, and average ratings.

  5. Generate candidates: Retrieve highly rated or popular items from the user’s cluster or from related item clusters.

  6. Rerank and evaluate: Apply a personalized model within the candidate set, then measure ranking quality, coverage, diversity, novelty, latency, and segment-level fairness.

For example, an e-commerce system might identify “budget electronics,” “premium audio,” and “outdoor enthusiasts.” A personalized reranker can then distinguish users within the budget-electronics cluster while still benefiting from cluster-level candidate generation.

Algorithm Trade-offs

Algorithm Cluster shapes found Choose k? Handles noise Scalability and trade-offs
k-means Convex, similarly sized clusters Yes No Approximately O(n * k * d) per iteration; fast but sensitive to scaling, initialization, and outliers
k-medoids Convex clusters around representative points Yes No More robust than k-means but generally slower; often requires approximately O(n² * d) distance work
Agglomerative clustering Linkage-dependent: single is flexible, while complete and Ward favor compact groups Yes No Provides an interpretable hierarchy but commonly needs O(n²) memory
DBSCAN Arbitrary, density-based groups No Yes Handles irregular clusters and outliers but is sensitive to eps, min_samples, and density variation
HDBSCAN Irregular clusters with varying density No Yes More robust than DBSCAN across density changes, but has greater memory cost and still requires parameter choices
Gaussian mixture model Probabilistic, elliptical clusters Yes No Supports soft assignments and BIC/AIC selection but assumes component distributions such as Gaussian
Spectral clustering Non-convex groups based on graph connectivity Yes No Handles complex shapes but may require an O(n²) affinity matrix and costly eigendecomposition

Euclidean, Manhattan, and cosine distances respond differently to feature scale and sparsity. Cosine distance is often useful for text or product embeddings because it compares direction rather than magnitude.

Choosing and Validating Clusters

  • Elbow method: Look for diminishing returns in within-cluster sum of squares; it is a heuristic.

  • Silhouette score: Ranges from -1 to 1, with higher values generally indicating better separation. Exclude noise samples from DBSCAN or HDBSCAN.

  • Davies–Bouldin index: Lower values generally indicate less overlap.

  • BIC and AIC: Compare Gaussian mixture models with different numbers of components.

  • Domain judgment: Verify that cluster members share meaningful characteristics.

  • External metrics: With reliable labels, adjusted Rand index or normalized mutual information can compare the partition with those labels.

  • Interpretation: Profile every cluster. Clusters are not ground truth and do not establish causality.

Code Example

This example uses make_moons to show that DBSCAN handles the non-convex structure better than k-means, using silhouette score as a clustering diagnostic.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=300, noise=0.05, random_state=42)

# Keep a partition separate before fitting learned preprocessing or clustering.
X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=42
)

models = {
    "KMeans": Pipeline([
        ("scaler", StandardScaler()),
        ("clusterer", KMeans(
            n_clusters=2, n_init="auto", random_state=42
        )),
    ]),
    "DBSCAN": Pipeline([
        ("scaler", StandardScaler()),
        ("clusterer", DBSCAN(eps=0.2, min_samples=5)),
    ]),
}

for name, model in models.items():
    labels = model.fit_predict(X_train)

    # Noise labels are -1 and cannot be used as silhouette clusters.
    mask = labels != -1
    cluster_ids = np.unique(labels[mask])
    score = (
        silhouette_score(X_train[mask], labels[mask])
        if len(cluster_ids) >= 2
        else float("nan")
    )

    print(
        f"{name}: clusters={len(cluster_ids)}, "
        f"noise={int(np.sum(labels == -1))}, silhouette={score:.3f}"
    )

Common Pitfalls

  • Failing to scale features: Distance-based algorithms are sensitive to feature magnitude.

  • Ignoring missing values or sparsity: Impute appropriately or choose methods and metrics suitable for sparse data.

  • Treating clusters as ground truth: Clusters may reflect geography, popularity, or collection artifacts.

  • Choosing k mechanically: Combine metrics with domain knowledge and cluster profiles.

  • Optimizing only accuracy-like metrics: Also evaluate Recall@K, NDCG@K, coverage, diversity, novelty, and popularity bias.

  • Ignoring leakage: Do not use future interactions or full-data-derived statistics when constructing earlier-period features.

  • Overfitting the validation set: Separate training, model selection, and final evaluation, then validate online when possible.

Interviewers commonly ask where clustering belongs relative to collaborative filtering, how k is selected, how cold-start users are handled, how noise is treated, and why cluster-based recommendation can reduce diversity.

44

How can clustering analyze social network connectivity?

A connectivity-clustering framework groups users or accounts according to their relationships, interaction patterns, or structural roles. It can reveal communities, bridges, isolated or unusual users, and similar network positions; the resulting clusters are interpretive patterns rather than ground-truth classes. The main trade-off is computational scalability versus the ability to model complex or noisy network structures.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12) and general network-analysis concepts

Framework

  1. Define the objective

  2. Community detection: Find dense internal connections separated by weaker external links.

  3. Structural equivalence or role discovery: Group nodes with similar adjacency patterns or positions, such as broadcasters and bridges.

  4. Anomaly detection: Identify sparse, isolated, or unusually connected accounts.

  5. Represent the network

  6. Treat users as nodes and relationships as weighted, directed, or binary edges.

  7. Preserve direction when interactions are directional.

  8. Create feature vectors from adjacency rows, degree, PageRank, clustering coefficient, betweenness, network embeddings, or graph-neural-network representations.

  9. Choose and validate

  10. Compare feature-based clustering with graph-native methods such as Louvain, Leiden, and spectral clustering.

  11. Examine cluster sizes, density, degree profiles, edge flows, and representative users.

  12. Test stability across time windows, resamples, and reasonable parameter changes. Save preprocessing, model parameters, random seeds, and the label permutation for reproducibility.

Algorithm Comparison

Algorithm Shapes or structure found Choose k? Noise Scalability and complexity
k-means Roughly spherical, compact clusters Yes No Excellent; approximately O(n*k*d) per iteration
k-medoids Compact clusters centred on representative medoids Yes No explicit label Often needs pairwise distances and roughly O(n²) memory
Agglomerative hierarchical Linkage-dependent: single permits chaining, complete favors compact groups, average is intermediate, and Ward favors similar-size groups k or cutoff No Standard implementation has roughly O(n³) worst-case time and O(n²) memory
DBSCAN Arbitrary-shaped, density-connected clusters No Yes, label -1 Worst case O(n²); spatial indexing can help
HDBSCAN Flexible shapes and clusters with different densities No Yes Generally costlier than fixed-eps DBSCAN
Gaussian mixture Probabilistic elliptical clusters with different sizes and orientations Yes No Fixed-covariance updates cost roughly O(n*k*d) per EM iteration; covariance fitting adds cost
Spectral clustering Connected communities or nonlinear structures Usually No Dense implementations may require O(n³) eigendecomposition; sparse methods scale better

sklearn.cluster.HDBSCAN has been available since scikit-learn 1.3 and is available in the stated version.

Distances and Scaling

  • Euclidean distance suits continuous features and embeddings; Manhattan distance measures coordinate-wise differences.

  • Cosine distance compares direction rather than magnitude and is useful for text embeddings.

  • Jaccard distance compares neighbor or interaction sets, while shortest-path distance captures graph connectivity but becomes problematic in disconnected graphs.

  • Scale features with incompatible units before distance-based clustering. A raw follower count can dominate Euclidean distance, unweighted objectives and cosine similarity alike; cosine is unchanged only when a whole vector is multiplied by a positive scalar, not when one feature is rescaled.

  • Handle missing attribute values appropriately, but do not impute an absent edge as though it were ordinary missing tabular data; absence may itself be meaningful.

Choosing and Interpreting Clusters

  • Elbow method: Inspect the bend in within-cluster sum of squares versus k.

  • Silhouette score: Measures cohesion relative to separation; for DBSCAN, calculate it on non-noise points and require at least two resulting clusters.

  • Davies-Bouldin index: Favors compact, well-separated clusters.

  • GMM selection: Compare BIC and AIC, for which lower values indicate a better fit-complexity balance.

  • Domain judgment: Check whether memberships, sizes, and network roles make operational sense.

  • Labeled evaluation: With trustworthy labels, use adjusted Rand index, normalized mutual information, or related external metrics.

Profile each cluster with degree distributions, activity, influential users, edge patterns, and temporal behavior. Clusters are hypotheses to interpret, not automatically discovered ground truth.

Code Example

This example uses two-dimensional embeddings as a proxy for network positions. K-means splits the curved groups, while DBSCAN follows their nonlinear shape and can identify outliers.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score

X, _ = make_moons(n_samples=300, noise=0.08, random_state=42)

models = {
    "k-means": KMeans(n_clusters=2, random_state=42, n_init=10),
    "DBSCAN": DBSCAN(eps=0.30, min_samples=5),
}

for name, model in models.items():
    labels = model.fit_predict(X)
    non_noise = labels != -1
    cluster_count = len(np.unique(labels[non_noise]))
    noise_count = int((~non_noise).sum())

    if 1 < cluster_count < non_noise.sum():
        score = silhouette_score(X[non_noise], labels[non_noise])
    else:
        score = float("nan")

    print(
        f"{name}: clusters={cluster_count}, "
        f"noise={noise_count}, silhouette={score:.3f}"
    )

Common Pitfalls

  • Selecting k mechanically: Combine statistical criteria, stability checks, and domain judgment.

  • Using inappropriate distances or skipping scaling: Results can change substantially when features have incompatible units.

  • Treating clusters as truth or tuning against test labels: Interpret clusters and keep evaluation data isolated.

  • Ignoring graph-native objectives: Feature clustering does not necessarily maximize within-community connectivity.

  • Ignoring scale: All-pairs methods can require quadratic memory; use sparse representations or appropriate approximations for large graphs.

45

How do you cluster time-series data such as stocks or weather?

Time-series clustering groups similar series by their shape, dynamics, or behavior. For stocks and weather, it helps discover recurring regimes or operating patterns rather than compare isolated timestamps. The main trade-off is interpretability and scalability against sensitivity to alignment, noise, and assumptions about cluster shape.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Representation and Distance

  • Unit of analysis: Each observation should normally represent one entity’s trajectory, such as one stock or one weather station, rather than one timestamp.

  • Stocks and weather: Raw stock prices are often non-stationary, so log returns, volatility, drawdown, and trend are usually more meaningful than price levels. For weather, consider temperature or precipitation anomalies, seasonality, pressure, and lagged relationships.

  • Feature vectors: Trend, seasonality, autocorrelation, rolling statistics, lag correlations, and Fourier terms make many series directly comparable.

  • Missing data: Use a documented interpolation or imputation policy. When forecasting is involved, interpolation must not use future observations.

  • Scaling: Standardize features for distance-based methods so large-range variables do not dominate. Per-series normalization can compare shape instead of level. Fit learned preprocessing on training data only.

  • Distances: Euclidean distance is simple but alignment-sensitive; correlation distance compares shape; cosine distance suits text embeddings; and dynamic time warping (DTW) handles shifted events at roughly O(T²) work per pair.

Algorithm Comparison

Algorithm Shapes and behavior Choose k? Noise Typical scale
k-means Spherical groups with a fast baseline Yes No O(n × k × d) per iteration
k-medoids Representative observations; irregular structure depends on the dissimilarity Yes No explicit model Typically O(n²) distance work
Agglomerative Single can capture non-convex groups; complete favors compact groups, average compromises, and Ward favors spherical Euclidean groups Usually by cutting the tree No O(n²) time and memory
DBSCAN Arbitrary-shaped density-connected groups No fixed k Yes O(n²) worst case in scikit-learn
HDBSCAN Arbitrary groups with varying densities No fixed k Yes O(n log n) best case, O(n²) worst case
Gaussian mixture Elliptical groups with soft assignments Component count No explicit model About O(n × k × d) per iteration for diagonal covariance; full covariance adds O(n × k × d²)
Spectral Manifold-based non-convex groups Yes No Dense form: O(n³) time and O(n²) memory

sklearn.cluster.HDBSCAN has been available since scikit-learn 1.3.

Choosing and Validating Clusters

  • Elbow method: Look for diminishing improvement as k grows.

  • Silhouette score: Measures cohesion and separation; it requires at least two clusters and at most n - 1 labels. The Davies-Bouldin index is another internal measure, where lower is better.

  • BIC/AIC: Select the number of Gaussian components while penalizing complexity.

  • Domain judgment and stability: Check whether clusters remain meaningful under resampling, time windows, and reasonable parameter changes.

  • External labels: When representative labels exist, use adjusted Rand index or normalized mutual information for validation. Labels support evaluation rather than make the clusters ground truth.

Practical Workflow

  1. Define the entity, time interval, similarity objective, and cluster interpretation.

  2. Align or resample series with different lengths or frequencies.

  3. For ordered observations, use rolling-origin validation with sklearn.model_selection.TimeSeriesSplit; for independent entity descriptors, a held-out set of entities is appropriate.

  4. Fit imputation, scaling, feature selection, and PCA only on training data, normally inside a Pipeline.

  5. Compare candidate algorithms on internal quality, stability, runtime, and interpretability.

  6. Profile clusters using centroids, medoids, or representative trajectories, and test downstream usefulness on untouched data.

Code Example

This example treats each row as an independent two-feature temporal descriptor and compares k-means with DBSCAN on non-convex moons.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=500, noise=0.08, random_state=42)

# The synthetic rows are independent descriptors, so a random split is appropriate.
X_train, X_test = train_test_split(
    X, test_size=0.25, random_state=42
)

kmeans = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", KMeans(n_clusters=2, n_init=10, random_state=42)),
])
dbscan = Pipeline([
    ("scale", StandardScaler()),
    ("cluster", DBSCAN(eps=0.25, min_samples=5)),
])

kmeans_labels = kmeans.fit_predict(X_train)
dbscan_labels = dbscan.fit_predict(X_train)

kmeans_x = kmeans.named_steps["scale"].transform(X_train)
dbscan_x = dbscan.named_steps["scale"].transform(X_train)

print("K-Means silhouette:",
      silhouette_score(kmeans_x, kmeans_labels))

non_noise = dbscan_labels != -1
print("DBSCAN silhouette:",
      silhouette_score(dbscan_x[non_noise], dbscan_labels[non_noise]))

print("K-Means test assignments by cluster:",
      np.bincount(kmeans.predict(X_test), minlength=2))
# DBSCAN has no out-of-sample predict method in scikit-learn.

Common Pitfalls

  • Scaling incorrectly: Fitting transformations on validation or test data leaks information.

  • Ignoring alignment: Pointwise distances miss patterns occurring at different times.

  • Overfitting cluster selection: Repeatedly choosing preprocessing and parameters to maximize silhouette or another internal score overstates performance.

  • Treating unequal sizes as errors: Clusters may naturally have different sizes; forcing equal sizes can discard rare regimes unless the application requires it.

  • Assuming reproducibility comes only from a seed: Also record package versions, preprocessing choices, parameters, and fitted pipelines.

  • Ignoring compute limits: Standard scikit-learn clustering is CPU-oriented; GPUs mainly help custom batched-distance or deep-embedding pipelines.

  • Interpreting clusters as truth: Profile and validate each group before assigning it business meaning.

46

How does deep learning support cluster analysis?

Deep clustering uses neural networks to learn representations and cluster assignments from complex data, making groups that are hard to identify in raw feature space easier to analyze. It suits images, text, and nonlinear sensor data, but adds training cost, tuning, and reduced interpretability, and may still produce misleading partitions.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); PyTorch 2.x for custom deep-clustering implementations

How Deep Learning Helps

Deep learning is not one clustering algorithm; it supplies learned representations or objectives for a clustering stage:

  • Representation learning: Autoencoders and self-supervised encoders transform raw data into latent spaces where distances may better express similarity.

  • Joint optimization: Deep Embedded Clustering learns embeddings and assignments together rather than clustering fixed features.

  • Transfer learning: Pretrained CNNs, vision transformers, or text encoders can provide clustering-ready embeddings without training from scratch.

  • Probabilistic assignments: Variational models can represent uncertainty instead of forcing every observation into a hard cluster.

Common implementations include autoencoder plus K-means, DEC/IDEC, VAE plus Gaussian mixture modeling, self-supervised methods such as SCAN, deep spectral clustering, and clustering pretrained-model embeddings. DEC uses Student-t soft assignments with a KL-divergence target; IDEC explicitly balances reconstruction and clustering objectives.

A representative objective is L = L_rec + beta * L_KL: L is the total loss, L_rec measures reconstruction quality, beta weights the clustering term, and L_KL regularizes soft assignments toward a target distribution.

Classical and Density-Based Alternatives

Method How it works / cluster shapes Choose k? Handles noise? Strength; weakness; typical use Scale
K-means Euclidean centroids; compact, roughly spherical cells Yes No Fast baseline; sensitive to outliers and non-convex shapes O(n * k * d) per iteration
K-medoids Uses observed points as representatives Yes No More outlier-resistant than K-means; slower Costlier than K-means
Agglomerative Merges clusters using a linkage rule; shape varies Cut hierarchy No Flexible hierarchy; expensive on large data Commonly O(n²) memory
DBSCAN Density connectivity; finds arbitrary-shaped clusters No Yes Handles curved groups and noise; sensitive to eps Worst-case neighborhood search can approach O(n²)
HDBSCAN Density hierarchy; supports variable-sized clusters No global k Yes Useful for uncertain density; costlier and parameter-sensitive Generally costlier than DBSCAN
Gaussian mixture Gaussian probabilities; elliptical clusters Yes No Soft membership; sensitive to covariance assumptions Cost grows with components and dimensions
Spectral Graph eigenvectors; captures non-convex geometry Usually No Handles complex shapes; costly and sensitive to k Dense implementations can require O(n³) work

Agglomerative results depend on linkage: single linkage can form chains, complete linkage favors compact groups, average linkage balances the criteria, and Ward linkage minimizes increases in within-cluster variance for Euclidean data. Scikit-learn’s HDBSCAN implementation has existed since version 1.3.

Distances, Cluster Selection, and Evaluation

  • Euclidean distance measures straight-line separation and is sensitive to feature scale.

  • Manhattan distance sums coordinate-wise differences and also remains scale-sensitive.

  • Cosine distance measures directional similarity and is useful for normalized text embeddings.

Scale features before K-means, DBSCAN, PCA, SVMs, or gradient-based methods when units differ; tree-based models generally do not require scaling. Choose k using the elbow method, silhouette score, Davies-Bouldin score, BIC/AIC for Gaussian mixtures, and domain judgment. With labels, use external measures such as adjusted Rand index. Profile and validate each cluster because clusters are mathematical partitions, not ground truth.

Code Example

This scikit-learn example contrasts K-means with DBSCAN on curved data. The pipelines ensure scaling is fitted only on the training split.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=300, noise=0.08, random_state=42)
X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=42
)

def clustering_pipeline(model):
    return Pipeline([
        ("scaler", StandardScaler()),
        ("cluster", model),
    ])

kmeans_pipeline = clustering_pipeline(
    KMeans(n_clusters=2, n_init=10, random_state=42)
)
dbscan_pipeline = clustering_pipeline(
    DBSCAN(eps=0.30, min_samples=5)
)

kmeans_labels = kmeans_pipeline.fit_predict(X_train)
dbscan_labels = dbscan_pipeline.fit_predict(X_train)
X_train_scaled = kmeans_pipeline.named_steps["scaler"].transform(X_train)

def safe_silhouette(features, labels):
    """Compute silhouette after excluding DBSCAN noise points."""
    keep = labels != -1
    if np.unique(labels[keep]).size < 2:
        return float("nan")
    return silhouette_score(features[keep], labels[keep])

print(f"Training samples: {X_train.shape[0]}")
print(f"Holdout samples: {X_test.shape[0]}")
print(f"K-means silhouette: {safe_silhouette(X_train_scaled, kmeans_labels):.3f}")
print(f"DBSCAN silhouette: {safe_silhouette(X_train_scaled, dbscan_labels):.3f}")
print(f"DBSCAN noise points: {np.sum(dbscan_labels == -1)}")

# Silhouette favours convex clusters, so K-means usually scores higher here
# even though DBSCAN recovers the two moons.

Common Pitfalls

  • Leakage: Fit scalers, imputers, autoencoders, and reducers only on training data or within each validation fold.

  • Rare groups and missing values: Rare populations may be absorbed into large clusters or treated as noise; decide whether they are errors, outliers, or meaningful groups.

  • Overfitting and collapse: Monitor representation collapse, validate stability across resamples, and do not optimize only one internal metric.

  • Resources and reproducibility: Set random seeds, monitor training memory, and remember that GPU acceleration mainly benefits embedding training rather than conventional clustering.

47

How does consensus clustering improve assignment stability?

Consensus clustering improves assignment stability by clustering many perturbed or subsampled versions of the data and combining their partitions, so assignments supported across runs are preferred over those driven by one sample or initialization. This reduces sensitivity to resampling and algorithmic randomness, but costs additional computation and can reinforce a consistently incorrect structure.

Read full answer

Applies tomachine learning concepts (all versions)

How It Works

  1. Generate multiple datasets through subsampling, bootstrapping, feature perturbation, or random initialization.

  2. Run a base clustering algorithm on each dataset.

  3. Compare partitions using co-association rather than raw cluster labels.

  4. Aggregate the co-association matrix and derive a final partition, often with hierarchical clustering.

For observations i and j, co-association is:

co-association(i, j) = runs where i and j share a cluster / runs where both are available

Here, i and j are observations, the numerator counts their joint agreements, and the denominator counts runs containing both observations. A common final step is average-linkage clustering of 1 - co-association, treating the result as a dissimilarity matrix. Pairwise co-association avoids having to match arbitrary labels such as “Cluster 0” across separate runs.

Why It Improves Stability

  • Reduces sampling sensitivity: A consistent grouping is less dependent on which observations happened to enter a particular run.

  • Smooths initialization effects: Results are aggregated rather than determined by one favorable initialization.

  • Identifies robust relationships: Points repeatedly grouped together receive stronger final assignments.

  • Exposes uncertainty: Low co-association indicates that a point has unstable relationships with most other points.

Consensus improves agreement, not necessarily correctness. It cannot repair an unsuitable distance metric, incorrect number of clusters, poor base algorithm, or data without meaningful cluster structure. Stability must therefore be assessed separately from validity.

Comparing Common Clustering Methods

Method Cluster shapes found Choose k? Noise handling Scalability and trade-off
k-means Compact, roughly spherical clusters Yes Outliers can distort centroids O(n * k * d) per iteration; fast but initialization- and scaling-sensitive
k-medoids Groups around representative medoids; shapes can be more flexible Yes More robust than k-means, though outliers can still matter Typically slower than k-means
Agglomerative hierarchical Single linkage can bridge broadly, complete favors compact groups, average balances them, and Ward favors variance reduction Usually through a merge cut No strong native noise model Scikit-learn commonly requires quadratic memory and roughly quadratic work
DBSCAN Arbitrary-shaped dense groups No; choose eps and min_samples Explicitly labels noise as -1 Scikit-learn uses a brute-force neighborhood search with quadratic worst-case behavior
HDBSCAN Variable-density, potentially non-convex groups No fixed k Strong noise handling and membership or outlier scores Higher preprocessing cost than DBSCAN; scalability depends on the implementation
Gaussian mixture Elliptical, probabilistic groups Yes No explicit noise label Roughly O(n * k * d) per EM iteration plus covariance costs
Spectral Non-convex or graph-based structures Usually yes No explicit noise label by default Dense formulations can require cubic work; sparse graphs improve scalability

Consensus clustering can use many of these as base learners, although outputs such as DBSCAN noise labels must be handled consistently. On two-moons data, for example, Euclidean k-means often cuts across the curved groups, while DBSCAN with suitable parameters can separate them and identify sparse points as noise.

Practical Guidance

  • Euclidean distance measures straight-line separation, while Manhattan distance sums absolute coordinate differences. Both are strongly affected by features measured on different scales, so standardize features when their units or ranges differ.

  • Cosine distance, based on 1 - cosine similarity, measures directional difference and is useful for text embeddings where magnitude is less important than orientation.

  • Choose the number of clusters using domain judgment, the elbow method, silhouette score, or the Davies-Bouldin index. BIC and AIC are more directly applicable to Gaussian mixture models.

  • Measure run-to-run stability using co-association or the adjusted Rand index between partitions on shared observations. If trusted external labels exist, adjusted Rand index can measure agreement with those labels.

  • Impute missing values before clustering, protect rare groups from being dominated by larger ones, and use fixed random seeds when reproducibility matters.

  • Profile each cluster using feature summaries and representative observations. Clusters are organizing patterns, not automatically ground truth.

Common Pitfalls

  • Using consensus to disguise a fundamentally unsuitable base algorithm.

  • Selecting parameters or preprocessing in a way that leaks information from held-out data.

  • Assuming a higher silhouette score proves semantic usefulness or guarantees stability.

  • Treating DBSCAN noise points as an ordinary cluster.

  • Interpreting clusters without checking their feature profiles.

  • Likely follow-up questions: How would you choose the perturbation strategy, number of repeats, and final number of clusters? How would you distinguish genuine structure from a stable but incorrect consensus?

48

What is subspace clustering in high-dimensional data?

Subspace clustering is a family of methods that groups observations lying in different local low-dimensional structures within high-dimensional data. It solves the problem that one global projection may conceal several groups, but its main trade-off is potentially richer structure discovery versus expensive affinity or graph construction and sensitive parameters.

Read full answer

Applies tosubspace-clustering concepts (all versions); scikit-learn 1.3+ for `HDBSCAN`

Core Idea

The usual model assumes that the data are generated by a union of low-dimensional linear subspaces, such as (S_1 cup S_2 cup dots cup S_k), where each subspace dimension is much smaller than the original feature dimension (d). A point may lie in one subspace or, for overlapping structures, have membership in several.

A subspace is not necessarily a subset of the original columns. It may represent latent linear combinations of features, so subspace clustering is different from both global feature selection and PCA:

  • PCA finds one global low-dimensional representation.

  • Feature selection chooses one global subset of columns.

  • Subspace clustering allows different groups to have different local structures.

How It Works

A typical pipeline involves:

  1. Preprocessing: Impute missing values and scale features when the method uses Euclidean distances or density estimates.

  2. Affinity construction: Measure whether observations share a local subspace using neighborhoods, local tangent spaces, reconstruction errors, or self-representation.

  3. Representation learning: Build an affinity graph or embedding in which observations from the same structure are connected.

  4. Clustering: Apply spectral clustering, k-means, or another partitioning method to the representation.

  5. Interpretation: Profile each resulting cluster by examining its most distinctive features, members, and stability.

Euclidean distances can become less informative as dimensionality grows, and nearest neighbors may be sparse. For text embeddings or other direction-based representations, cosine distance is often more appropriate. Missing values, outliers, graph sparsity, and parameter selection are also major practical concerns.

Algorithm Comparison

Algorithm Cluster shapes found Need to choose k? Noise handling Scalability
k-means Compact, roughly spherical clusters Yes No explicit noise model Large-scale friendly; about O(n * k * d) per iteration
k-medoids Less rigid than k-means but still constrained by medoids Yes Outliers can distort medoids Often requires O(n²) pairwise distances
Agglomerative hierarchical Depends on linkage: single is flexible, complete is compact, average is intermediate, and Ward favors spherical Euclidean clusters Can choose a cut level No explicit noise model Commonly requires O(n²) memory
DBSCAN Arbitrary non-convex shapes No Identifies noise points Moderate with spatial indexing; less effective in high dimensions
HDBSCAN Arbitrary shapes and variable-density clusters No Identifies noise points Usually more expensive than DBSCAN but more flexible
Gaussian mixture model Elliptical or Gaussian-shaped clusters Yes No explicit noise label Cost grows with sample count, cluster count, and dimensionality
Spectral clustering Connected, non-convex, or manifold-like structures Usually yes No explicit noise model Often O(n²) memory and up to O(n³) eigendecomposition
Subspace clustering Groups with different local low-dimensional structures Varies by method Varies by method Graph and spectral variants can be expensive

scikit-learn provides HDBSCAN since version 1.3, although it is a general density-based clustering algorithm rather than a dedicated subspace-clustering implementation.

Choosing and Evaluating Clusters

No single metric should determine the result:

  • Elbow method: Look for diminishing returns as k increases.

  • Silhouette score: Compare within-cluster and between-cluster distances.

  • Davies-Bouldin index: Penalize large internal spread and similarity between clusters.

  • BIC/AIC: Support model selection for Gaussian mixture models.

  • Stability and domain judgment: Check consistency across resamples and whether groups are meaningful in practice.

For the silhouette score, a(i) is the mean distance from observation i to others in its cluster, while b(i) is the smallest mean distance to another cluster. The score is s(i) = (b(i) - a(i)) / max(a(i), b(i)); its average ranges from -1 to 1, with higher values indicating better geometric separation. For DBSCAN or HDBSCAN, exclude noise and require at least two remaining clusters.

If trusted labels are available, the Adjusted Rand index (ARI) can compare discovered groups with those labels. Such labels are external references, not proof that the clusters are intrinsically correct; each cluster should still be profiled and interpreted.

Common Pitfalls

  • Confusing PCA with subspace clustering: PCA provides one global projection; subspace clustering may model multiple local structures.

  • Skipping scaling: Distance- and density-based methods can behave poorly when features use different units.

  • Ignoring sparsity and dimensionality: Neighborhood graphs may become unreliable or expensive.

  • Leakage: Fit imputers, scalers, feature selectors, and data-dependent graph construction on training data only.

  • Over-tuning: Select neighborhood size, subspace dimension, and other parameters using resampling rather than one favorable split.

  • Treating clusters as truth: Clustering produces descriptive groups, so domain interpretation and cluster profiling are essential.

Common follow-up questions include how local tangent spaces differ from low-rank subspace methods, how neighborhood graphs are constructed, why spectral methods scale poorly, and how clusters can be validated when true labels are unavailable.

49

How do you handle clustering for large-scale datasets?

Large-scale clustering means fitting groups while controlling memory, computation, and latency without losing meaningful structure. The main trade-off is scalability and approximation versus cluster fidelity, stability, and interpretability, because no single algorithm handles dense, high-dimensional, noisy, and streaming data equally well.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Challenges and Solutions

  • Memory pressure: Exact pairwise-distance methods can require (O(n^2)) memory, where (n) is the sample count. Avoid explicit distance matrices; use nearest-neighbor search, sampling, partitioning, or approximate methods.

  • Computational cost: Lloyd’s k-means algorithm costs approximately (O(nkd)) per iteration, where (k) is the number of clusters and (d) the feature count. Use MiniBatchKMeans, BIRCH, distributed implementations, or representative sampling for very large data.

  • Randomness and local minima: k-means can produce different partitions across runs. Set random_state, use multiple initializations such as n_init=10, and compare results across seeds.

  • Geometry and noise: k-means is suited to convex, roughly isotropic groups and may split crescents. DBSCAN or HDBSCAN can recover non-convex density-based groups and explicitly label noise.

  • High dimensionality: Euclidean distances may become less informative as dimensions increase. Use domain-driven feature selection, dimensionality reduction, or cosine distance for text embeddings.

  • Streaming data: MiniBatchKMeans.partial_fit supports incremental updates, while BIRCH summarizes incoming observations into micro-clusters. Batch algorithms require all observations in memory and may revisit them during iterative optimization.

Distance Metrics and Scaling

Euclidean distance is sqrt(sum_i((x_i - y_i)^2)), where x and y are feature vectors and i indexes features. Manhattan distance is sum_i(abs(x_i - y_i)). Cosine distance is 1 - dot(x,y)/(norm(x)*norm(y)), where the dot product measures directional alignment and the norms measure vector length; it is common for text embeddings.

Scale features before distance-based clustering when their units or ranges differ, or one large-valued feature may dominate the result. Scaling is generally unnecessary for tree-based methods.

Algorithm Comparison

Algorithm Cluster shapes found Choose (k)? Noise handling Scalability
k-means Convex, isotropic clusters Yes No native support Good; MiniBatchKMeans for very large data
k-medoids Roughly spherical, partitional clusters Yes Resists outliers; no explicit noise labels Often (O(n^2)) distance computation
Agglomerative Flexible, linkage-dependent geometry When cutting the tree No native support Poor at large (n)
DBSCAN Non-convex clusters and density variation No Yes Neighborhood search can approach (O(n^2))
HDBSCAN Irregular clusters with varying density No fixed (k) Yes Usually more expensive than DBSCAN
Gaussian mixture Elliptical, probabilistic clusters Yes Limited Moderate; depends on covariance structure
Spectral Complex manifold-like shapes Yes No native support Poor; graph construction can require (O(n^2)) memory

Single, complete, and average linkage preserve different cluster relationships; Ward linkage favors compact, similarly sized spherical clusters. sklearn.cluster.HDBSCAN is available in the stated scikit-learn version.

Choosing and Interpreting Clusters

  • Elbow method: Identify where within-cluster reduction stops improving rapidly; it is subjective.

  • Silhouette score: Ranges from -1 to 1, with higher values indicating better separation. Exclude DBSCAN noise points labeled -1.

  • Davies-Bouldin index: Lower values indicate more distinct, less overlapping clusters.

  • BIC/AIC: Apply these to Gaussian mixture models; lower values balance fit against complexity.

  • Domain judgment: A numerically better partition must still make operational sense.

  • External metrics: With trusted labels, the adjusted Rand index compares partitions; 1 is perfect, 0 represents chance-level agreement, and negative values are worse than chance.

Clusters are hypotheses, not ground truth. Profile each cluster, examine feature distributions, and test stability across samples, seeds, and resampling methods.

Code Example

This small make_moons example illustrates geometric failure: k-means splits the curved groups, while DBSCAN follows their density structure. For production-scale data, use mini-batch, streaming, partitioned, or distributed implementations described above.

import matplotlib.pyplot as plt
import numpy as np

from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=600, noise=0.08, random_state=42)
X_train, _ = train_test_split(
    X, test_size=0.30, random_state=42
)

# Fit preprocessing on training data only.
preprocessing = Pipeline([("scale", StandardScaler())])
X_train_scaled = preprocessing.fit_transform(X_train)

kmeans = KMeans(n_clusters=2, random_state=42, n_init=10)
dbscan = DBSCAN(eps=0.30, min_samples=5)

kmeans_labels = kmeans.fit_predict(X_train_scaled)
dbscan_labels = dbscan.fit_predict(X_train_scaled)

kmeans_score = silhouette_score(X_train_scaled, kmeans_labels)

non_noise = dbscan_labels != -1
dbscan_score = None
if np.unique(dbscan_labels[non_noise]).size >= 2:
    dbscan_score = silhouette_score(
        X_train_scaled[non_noise], dbscan_labels[non_noise]
    )

print(f"K-Means silhouette: {kmeans_score:.3f}")
print(
    "DBSCAN clusters:",
    np.unique(dbscan_labels[non_noise]).size,
)
print("DBSCAN noise points:", np.sum(~non_noise))
print(
    f"DBSCAN silhouette: {dbscan_score:.3f}"
    if dbscan_score is not None
    else "DBSCAN silhouette: n/a"
)

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, labels, title in [
    (axes[0], kmeans_labels, "K-Means"),
    (axes[1], dbscan_labels, "DBSCAN"),
]:
    for label in np.unique(labels):
        mask = labels == label
        ax.scatter(
            X_train_scaled[mask, 0],
            X_train_scaled[mask, 1],
            label="noise" if label == -1 else str(label),
        )
    ax.set_title(title)
plt.tight_layout()
plt.show()

Common Pitfalls

  • Scaling after clustering or fitting preprocessing on validation data causes leakage.

  • Impute missing values using training-only statistics before distance-based clustering.

  • Choosing (k) only to maximize silhouette can produce unstable or meaningless groups.

  • Interpreting clusters without profiling their features or checking stability.

  • Assuming DBSCAN’s eps transfers between unscaled and scaled feature spaces.

  • Evaluating DBSCAN without excluding label -1 from silhouette calculations.

  • Using supervised accuracy for clustering; use internal, stability, and external metrics.

  • Expecting GPU acceleration to directly accelerate clustering; it more often accelerates upstream embedding computation.

50

How can reinforcement learning optimize cluster analysis tasks?

Reinforcement learning (RL) can optimize clustering by treating algorithm selection, assignment refinement, merging, splitting, or constraint handling as decisions and learning which choices maximize a reward. It supports flexible multi-objective optimization, but adds complexity and potentially unstable exploration, and is often unnecessary when direct optimization with a fixed algorithm is sufficient.

Read full answer

Applies tomachine learning concepts with Python 3.12 and scikit-learn 1.5+

RL Formulation

Model the workflow as a Markov decision process: the state contains assignments, prototypes, or quality statistics; an action changes the clustering; the transition produces a new state; and the reward measures clustering quality. If there is only one independent choice, such as selecting an algorithm, use a simpler bandit formulation.

A policy is evaluated using J(π) = E[Σ_{t=0}^T γ^t r_t], where J is expected return, π is the policy, E denotes expectation, Σ sums over time, t is the current step, T is the final step, γ is the discount factor, and r_t is the reward at step t.

Reward Design

A reward can combine several objectives:

R = silhouette - λ * C - μ * V

Here, R is the scalar reward, silhouette measures compactness and separation, λ and μ are non-negative penalty weights, C represents complexity, and V counts constraint violations. Domain validity should normally be included because a narrow statistical objective can produce meaningless clusters.

Clustering Algorithms

Method Cluster shapes found Choose k Noise handling Scalability
k-means Convex, roughly spherical Yes None O(n * k * d) per iteration
k-medoids Compact, less geometrically constrained Yes Outlier-tolerant, but no formal noise label Distance matrix is O(n²); exact PAM search can reach O(n³)
Agglomerative Linkage-dependent Tree cut None Often O(n²) memory; naïve worst case O(n³) time
DBSCAN Arbitrary-shaped density clusters No Yes Near O(n log n) with spatial indexing; O(n²) brute force
HDBSCAN Hierarchical clusters with varying density No Yes Implementation-dependent
Gaussian mixture Elliptical, probabilistic clusters Yes No native noise component Moderate; depends on covariance structure
Spectral Non-convex or manifold-like structures Yes No native handling Dense form uses O(n³) time and O(n²) memory

Agglomerative single linkage can chain, complete linkage favors compact groups, average linkage is a compromise, and Ward linkage minimizes within-cluster variance under Euclidean geometry. scikit-learn provides HDBSCAN from version 1.3.

Distance, Scaling, and Validation

  • Euclidean distance is sensitive to feature scale; Manhattan distance sums coordinate-wise differences; cosine distance is useful for directional text embeddings.

  • Standardize features for k-means, k-medoids, DBSCAN, HDBSCAN, PCA, SVM, and gradient-based methods. Tree models generally do not need scaling.

  • Select cluster counts with the elbow method, silhouette score, Davies-Bouldin index, domain knowledge, and stability across resamples. Use BIC or AIC for Gaussian mixtures.

  • With reference labels, adjusted Rand index measures agreement, but clustering labels may represent a different taxonomy. Profiles of each cluster must still be inspected because clusters are not ground truth.

Code Example

This epsilon-greedy bandit compares k-means and DBSCAN on two moons. Because DBSCAN has no predict method, validation observations are assigned to the nearest training core sample solely for this out-of-sample illustration.

import numpy as np
from sklearn.cluster import DBSCAN, KMeans
from sklearn.datasets import make_moons
from sklearn.metrics import silhouette_score
from sklearn.model_selection import train_test_split
from sklearn.neighbors import NearestNeighbors
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, _ = make_moons(n_samples=300, noise=0.08, random_state=42)
X_train, X_val = train_test_split(
    X, test_size=0.25, random_state=42
)


def fit_cluster(method):
    if method == "kmeans":
        estimator = KMeans(n_clusters=2, n_init=10, random_state=42)
    else:
        estimator = DBSCAN(eps=0.25, min_samples=5)

    # Preprocessing is fitted on the training partition only.
    pipeline = Pipeline([
        ("scale", StandardScaler()),
        ("cluster", estimator),
    ])
    pipeline.fit(X_train)

    cluster = pipeline.named_steps["cluster"]
    scaler = pipeline.named_steps["scale"]
    scaled_train = scaler.transform(X_train)
    scaled_val = scaler.transform(X_val)

    if method == "kmeans":
        val_labels = cluster.predict(scaled_val)
        return cluster.labels_, scaled_train, scaled_val, val_labels

    core_indices = cluster.core_sample_indices_
    if len(core_indices) == 0:
        val_labels = np.full(len(X_val), -1)
    else:
        core_labels = cluster.labels_[core_indices]
        neighbors = NearestNeighbors(n_neighbors=1)
        neighbors.fit(scaled_train[core_indices])
        nearest = neighbors.kneighbors(
            scaled_val, return_distance=False
        ).ravel()
        val_labels = core_labels[nearest]

    return cluster.labels_, scaled_train, scaled_val, val_labels


def score(labels, features):
    count = np.unique(labels).size
    if count < 2 or count >= len(labels):
        return -1.0
    return silhouette_score(features, labels)


methods = ("kmeans", "dbscan")
rewards = []

for method in methods:
    train_labels, scaled_train, scaled_val, val_labels = fit_cluster(method)
    non_noise = train_labels != -1
    train_score = score(train_labels[non_noise], scaled_train[non_noise])
    reward = score(val_labels, scaled_val)
    rewards.append(reward)
    print(method, "training silhouette:", train_score)
    print(method, "validation reward:", reward)

# Epsilon-greedy single-decision RL/bandit.
rng = np.random.default_rng(42)
q_values = np.zeros(len(methods))
alpha = 0.20
epsilon = 0.20

for _ in range(40):
    if rng.random() < epsilon:
        action = int(rng.integers(len(methods)))
    else:
        action = int(np.argmax(q_values))
    q_values[action] += alpha * (rewards[action] - q_values[action])

print("RL-selected method:", methods[int(np.argmax(q_values))])

Common Pitfalls

  • Data leakage: the reward partition becomes the agent’s training signal; reserve a separate final test partition before reporting generalization.

  • Wrong objective: silhouette or inertia alone can ignore domain constraints, missing values, outliers, and interpretability.

  • Unscaled features: distance-based methods may organize points by magnitude rather than meaning.

  • Overfitting: repeated validation-driven exploration can overfit the reward partition.

  • Misinterpretation: profile clusters and test stability; visual separation does not prove that the clusters are useful or causal.

Common follow-ups concern state and reward design, missing-value handling, cluster stability, and why RL is preferable to direct optimization.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Cluster Analysis answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects