Common mistake 1 with Principal components (PCA)

Not standardising. Income in rupees alongside a 1–10 index means component one is simply income, in disguise.

Common mistake 2 with Principal components (PCA)

Naming components as though the label were a finding. A component is a weighted sum, nothing more; the name is your interpretation and should be defended.

Common mistake 3 with Principal components (PCA)

The sign of a component is arbitrary - the software could flip it and nothing would change. Never read meaning into the sign alone.

Common mistake 4 with Principal components (PCA)

Using PCA on regressors and then reporting the component coefficient as an effect. It is an effect of a bundle, not of any one variable.

Common mistake 5 with Principal components (PCA)

Applying it to binary or heavily skewed variables, where linear correlation is a poor summary.

EconStatsCraft › Principal components (PCA)

Principal components (PCA)

Turns many correlated variables into a few uncorrelated components, ordered by how much variation each explains. Used to build an index from several indicators, and to break multicollinearity when regressors move together too closely to separate.

What it needs

Variables - Three or more numeric variables that plausibly measure related things.

The intuition

Twenty correlated indicators are not twenty independent pieces of information. Principal components rotates the axes so the first new axis points along the direction of greatest spread, the second along the greatest spread perpendicular to it, and so on. Usually the first two or three carry most of the variation, and the rest is noise you can discard - not because those directions are unimportant in principle, but because nothing in your data varies along them.

When to use it

To build a single index from several indicators that measure the same broad thing; to cut a large set of regressors down before modelling; and to break multicollinearity when regressors move together too closely to be separated.

Where the estimator comes from

Centre X and take the eigendecomposition of the covariance matrix, S=V\Lambda V'. The columns of V are the loadings, the eigenvalues in \Lambda are the variances of the components, and the scores are Z=XV. Equivalently, the singular value decomposition X=U D V' gives the same V with \Lambda=D^2/(n-1), and is the numerically stable route - forming X'X squares the condition number and loses precision. The share of variance explained by component k is \lambda_k/\sum_j\lambda_j. The Eckart-Young theorem makes the optimality precise: the first k components give the rank-k matrix closest to X in Frobenius norm, so no other linear compression of the same size loses less.

The model

\Sigma=V\Lambda V',\qquad z_k=Xv_k,\qquad \text{share explained}=\frac{\lambda_k}{\sum_j \lambda_j}

What it assumes

The relationships are linear. PCA sees correlations, and correlation is a linear measure.
Variables are on a comparable scale - which is why standardising is on by default.
Variance is what you care about. A low-variance variable is treated as unimportant, whether or not it is.
The variables are on comparable scales, or standardised first. Run on raw covariances, the component with the largest units dominates, and the answer changes if you switch from rupees to lakhs.

Statistical properties

PCA is not scale-invariant. A variable measured in rupees rather than thousands of rupees will dominate the first component purely through its units, so standardise unless the variables share a meaningful scale.
Components are uncorrelated BY CONSTRUCTION, so using them as regressors eliminates multicollinearity - but the resulting coefficients are on rotated axes and usually mean nothing.
Signs are arbitrary. Flipping a component and its loadings together leaves everything unchanged, so software may report the opposite sign from a textbook and both are correct.
The scree plot and the Kaiser rule (keep eigenvalues above 1) are conventions, not tests. Horn's parallel analysis, which compares eigenvalues against those from random data of the same shape, is better grounded and usually keeps fewer.
PCA finds directions of VARIANCE, which need not be directions of interest. The component that predicts your outcome may be the fifth, and dropping it as noise is a real risk in principal-components regression.
It is a linear method. Variables related through a curve will not be compressed well, and the loadings will look like nonsense rather than announcing the problem.

How to read the output

The scree plot: keep components before the elbow. Alternatively keep every eigenvalue above 1 (Kaiser), which is what "auto" does on standardised data.
Cumulative variance explained: two components covering 80% means two numbers carry most of what six variables were saying.
The loadings say what each component IS. A first component with similar positive loadings on everything is a "general level" index; a second with positives on one group and negatives on another is a contrast between them.
Components are ordered by variance, not by importance. The first component is the direction of most variation, which is only the most interesting direction if variation is what you care about.

A worked reading

Three variables where the first is simply rescaled into thousands: unstandardised, the first component explains 100.0% of the variance and the loadings point almost entirely at that one column. Standardise and the same data gives 59.6% on the first component with a sensible spread of loadings. Nothing about the relationships changed - only the units - and the unstandardised answer was a statement about measurement rather than about the data. This is the single most common way PCA is misapplied.

Symptom → diagnosis → what to do

If you see this	It means	Do this
First component explaining nearly everything	A scale problem, almost certainly	Standardise and re-run
Loadings all the same sign and size on PC1	A general size factor	Common and often the least interesting component
Needing many components for 80%	Variables are largely uncorrelated	There is nothing to compress. Do not force it
Signs differing from a published table	Sign is arbitrary	Not an error
Kaiser keeping far more than the scree suggests	Kaiser over-retains with many variables	Use parallel analysis
Components uninterpretable	PCA does not optimise interpretability	Use factor analysis with rotation instead

Common mistakes

Not standardising. Income in rupees alongside a 1–10 index means component one is simply income, in disguise.
Naming components as though the label were a finding. A component is a weighted sum, nothing more; the name is your interpretation and should be defended.
The sign of a component is arbitrary - the software could flip it and nothing would change. Never read meaning into the sign alone.
Using PCA on regressors and then reporting the component coefficient as an effect. It is an effect of a bundle, not of any one variable.
Applying it to binary or heavily skewed variables, where linear correlation is a poor summary.

Where to read this properly

Jolliffe, Principal Component Analysis (2002), ch. 2 and 6.
Eckart and Young (1936), Psychometrika 1(3), for the optimality result.
Horn (1965), "A Rationale and Test for the Number of Factors", Psychometrika 30(2).
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2009), ch. 14.

Related Multivariate statistics

Factor analysis · Cluster analysis

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Principal components (PCA)

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Multivariate statistics

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.