EconStatsCraft › Cluster analysis

Cluster analysis

Groups observations that resemble each other - countries into development types, households into segments, firms into strategy groups. It always returns clusters, so the diagnostics that tell you whether the grouping is real matter more than the grouping itself.

What it needs

Variables to cluster on

Frequently asked questions

What is cluster analysis?

Cluster analysis groups observations so that those in the same group are more similar to each other than to those in other groups, without being told the groups in advance. It is unsupervised learning: there is no outcome variable, only the features you choose to compare on.

What is the difference between k-means and hierarchical clustering?

K-means needs the number of clusters k in advance and assigns each point to the nearest of k centroids, which suits large data and round, similar-sized groups. Hierarchical clustering builds a tree of nested merges (a dendrogram) and lets you choose the number of clusters afterwards by cutting the tree, which suits smaller data and exploring structure.

How do I choose the number of clusters?

Look for an elbow in the within-cluster sum of squares as k rises, and for the k with the highest average silhouette score. When the two disagree or neither has a clear peak, the data may not have strong clusters at all, which is itself a finding.

Should I standardise variables before clustering?

Yes, almost always. Clustering uses distances, so a variable measured in thousands swamps one measured in fractions. Standardising to mean 0 and standard deviation 1 gives every variable equal say, unless some deserve more weight on purpose.

How do I know the clusters are real?

Check the silhouette score (above about 0.5 is strong, below 0.25 weak), rerun with a different seed or method and see whether the same groups come back, and describe each cluster by its means. Groups that only exist for one seed, or that cannot be described in words, are probably noise.

The intuition

Clustering asks whether the rows fall into groups, without being told what the groups are. That freedom is also the danger: every clustering algorithm will return clusters, including on data with no group structure whatsoever. The honest workflow is to cluster, then to ask whether the structure found is stronger than what random data of the same shape would produce.

When to use it

To build a typology - country development types, household segments, firm strategy groups - when the groups are not observed and you want the data to suggest them.

Where the estimator comes from

k-means minimises within-cluster sum of squares, \sum_{k}\sum_{i\in C_k}\lVert x_i-\mu_k\rVert^2, alternating between assigning each point to its nearest centroid and recomputing centroids. Each step cannot increase the objective, so it converges - but to a LOCAL minimum that depends on the initialisation, which is why k-means++ seeding and multiple restarts are standard. Hierarchical clustering instead builds a tree by repeatedly merging the closest pair, where "closest" depends on the linkage: single linkage uses the nearest pair of points and chains, complete uses the farthest and makes compact balls, and Ward merges whichever pair increases within-cluster variance least, which makes it the closest analogue of k-means.

The model

\min_{C_1..C_k}\sum_{j=1}^{k}\sum_{i\in C_j}\lVert x_i-\mu_j\rVert^2, \qquad s(i)=\frac{b(i)-a(i)}{\max\{a(i),b(i)\}}

What it assumes

Distance is a sensible measure of similarity for your variables - so they must be on comparable scales, which is why standardising is on by default.
k-means additionally assumes roughly spherical, similarly sized groups. It will do badly on elongated or nested shapes.
The variables are scaled deliberately. k-means minimises squared Euclidean distance, so a variable measured in thousands dominates one measured in units, and the clusters are then about that variable alone.
The clusters are roughly spherical and of similar size. k-means imposes that shape whatever the data look like, which is why it splits one elongated group rather than finding two real ones.

Statistical properties

k-means implicitly assumes spherical clusters of similar size, because Euclidean distance to a centroid is its only criterion. It will cut a single elongated cloud in half rather than find two crescents.
The result depends entirely on scaling. An unstandardised variable with a large range dominates the distance, so standardise unless the units are already comparable.
The silhouette measures how much closer a point is to its own cluster than to the next nearest, on a scale from -1 to 1. Below about 0.25 there is no real structure, whatever the algorithm returned.
The elbow method is not a test and frequently has no elbow. The gap statistic (Tibshirani et al., 2001) compares within-cluster dispersion against a reference distribution with no clusters, and can legitimately conclude that k = 1.
k-means always returns exactly k clusters and assigns every point, including outliers, which distort centroids badly. DBSCAN can leave points unassigned as noise, which is often the truthful answer.
Clusters are not classes. They have no external validity until something outside the clustering variables distinguishes them, and naming them before that check is how a marketing segmentation becomes a fiction.

How to read the output

The elbow plot: where the fall in within-cluster sum of squares flattens is a candidate k. It is a hint, not a test.
The silhouette: above about 0.5 is a reasonable structure, 0.25 to 0.5 is weak, below 0.25 means the grouping is essentially arbitrary.
The cluster means table is where the interpretation lives. If you cannot describe each cluster in a sentence, it is not useful.
Very uneven cluster sizes often mean k is wrong or outliers are driving the split.

A worked reading

Run k-means with k = 3 on data generated with no group structure at all and it returns three clusters with a tidy chart and plausible-looking centroids. The silhouette is 0.11. That number is the entire finding: the algorithm did its job, partitioned the space, and found nothing, because the objective it minimises has no way to report failure. Always compute a validity measure, and always compare it against what the same procedure yields on random data of the same dimension - the gap statistic formalises exactly that comparison.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Different runs giving different clusters	Local minima	Use k-means++ and many restarts; instability is itself informative
One giant cluster and several tiny ones	Scaling, or genuine outliers	Standardise; consider a robust method
Silhouette near zero	No real group structure	Do not proceed to name the clusters
No elbow in the scree of within-cluster sums	Common. The elbow is not guaranteed to exist	Use the gap statistic
Single-linkage clusters forming long chains	Known behaviour of single linkage	Use Ward or complete linkage
Clusters not differing on any outside variable	They may be an artefact	A segmentation nobody can act on is not a segmentation

Common mistakes

The fundamental one: the algorithm ALWAYS returns clusters, including on data with no group structure at all. The silhouette is what tells you whether they are real.
Not standardising. Distance is then dominated by whichever variable has the largest units.
Choosing k to suit the story. Report the elbow and silhouette so the reader can judge.
k-means depends on its random start - the app uses multiple restarts, but a solution that changes wildly across runs is a warning about the data, not the software.
Clustering on variables you then use to "test" whether the clusters differ on those variables. They will, by construction. That is circular.
Treating cluster membership as an exogenous variable in a later regression without acknowledging it was estimated.

Where to read this properly

MacQueen (1967), on k-means; Ward (1963), Journal of the American Statistical Association 58(301).
Rousseeuw (1987), "Silhouettes", Journal of Computational and Applied Mathematics 20.
Tibshirani, Walther and Hastie (2001), "Estimating the Number of Clusters via the Gap Statistic", JRSS-B 63(2).
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2009), ch. 14.

Related Multivariate statistics

Principal components (PCA) · Factor analysis

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Cluster analysis

What it needs

Frequently asked questions

What is cluster analysis?

What is the difference between k-means and hierarchical clustering?

How do I choose the number of clusters?

Should I standardise variables before clustering?

How do I know the clusters are real?

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Multivariate statistics

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.