Common mistake 1 with Ridge / Lasso / Elastic Net

The big one: refitting OLS on the survivors and reporting those p-values. The same data chose the variables, so the standard errors are wrong and the p-values are far too small. This is post-selection inference and it needs a method built for it - post-double-selection, or splitting the sample.

Common mistake 2 with Ridge / Lasso / Elastic Net

Reading the survivors as "the true model". At the cross-validated λ, lasso over-selects: on simulated data with four real regressors among forty it typically keeps eight or more. It is tuned for prediction, not for recovery.

Common mistake 3 with Ridge / Lasso / Elastic Net

Forgetting to standardise, when doing this by hand elsewhere. The penalty is not scale-invariant.

Common mistake 4 with Ridge / Lasso / Elastic Net

Using k-fold cross-validation on time series. It trains on the future to predict the past.

Common mistake 5 with Ridge / Lasso / Elastic Net

Lasso with two strongly collinear regressors keeps one arbitrarily and drops the other. Which one survives can flip with a trivial change to the sample; Elastic Net is more stable when that matters.

EconStatsCraft › Ridge / Lasso / Elastic Net

Ridge / Lasso / Elastic Net

Regression with a penalty on the size of the coefficients, so a model with more regressors than you could justify in an OLS still estimates. Lasso sets some coefficients to exactly zero and so selects; ridge shrinks everything and so stabilises. λ is chosen by cross-validation, and the coefficient path shows which variables leave the model first.

What it needs

Dependent variable
Candidate regressors - Give it more than you would put in an OLS - that is the point.

The intuition

With sixty candidate regressors and a hundred and twenty rows, OLS will fit the training data beautifully and predict terribly, because it spends its degrees of freedom fitting noise. Shrinkage deliberately biases the coefficients towards zero. That sounds like vandalism and is not: a little bias buys a large reduction in variance, and prediction error is the sum of both.

When to use it

When you have more candidate regressors than an OLS can sensibly carry - a long list of controls, many lags, a wide administrative dataset. Ridge when the regressors are collinear and you want stable estimates of all of them. Lasso when you want the data to narrow a long list down. If you have five regressors and a theory about each, use OLS.

Where the estimator comes from

Ridge minimises \lVert y-X\beta\rVert^2+\lambda\lVert\beta\rVert_2^2, with the closed form \hat{\beta}=(X'X+\lambda I)^{-1}X'y. Adding \lambda I makes the matrix invertible even when p>n, which is why ridge exists at all in the wide case. Lasso instead penalises the \ell_1 norm, \lVert y-X\beta\rVert^2+\lambda\lVert\beta\rVert_1, which has no closed form but does have corners in its constraint region at the axes - and a corner is exactly where a coefficient becomes precisely zero, which is why lasso selects and ridge does not. Elastic net mixes them. The bias-variance identity is what justifies all of it: E[(y-\hat{f})^2]=\text{Bias}^2+\text{Var}+\sigma^2, and \lambda trades the first against the second.

The model

\hat\beta = \arg\min_\beta \Big\{ \tfrac{1}{2n}\lVert y - X\beta \rVert_2^2 + \lambda \big[ \alpha \lVert \beta \rVert_1 + \tfrac{1-\alpha}{2}\lVert \beta \rVert_2^2 \big] \Big\}

What it assumes

Regressors are standardised before the penalty is applied - otherwise a variable measured in thousands is penalised a thousand times less than the same variable in units. This module standardises, and reports coefficients back in your units.
You care about prediction error. λ is chosen to minimise cross-validated prediction error, which is not the same objective as recovering the true model.
The observations are exchangeable enough for cross-validation to mean something. With time series, ordinary k-fold CV leaks the future into the past - use rolling-origin validation instead.
Some form of sparsity or collinearity is real. If every regressor genuinely matters equally, shrinkage buys nothing.

Statistical properties

Standardise before penalising. The penalty is applied to raw coefficient magnitudes, so a variable measured in small units gets a large coefficient and is shrunk hardest - purely because of its units.
The intercept must not be penalised, or the fit is not invariant to shifting y.
Lasso selects at most n variables when p exceeds n, which is a hard ceiling rather than a finding about the data.
With groups of correlated predictors, lasso picks one arbitrarily and zeroes the rest; ridge spreads the weight across them. Elastic net was designed for exactly this and keeps the group together.
Cross-validation chooses lambda by predictive performance, so the resulting model is tuned for prediction and NOT for inference. Post-selection p-values on the surviving variables are invalid, and the usual reported ones are badly so.
The one-standard-error rule picks the largest lambda within one standard error of the cross-validated minimum. It gives a simpler model at nearly the same accuracy and is the more defensible default.

How to read the output

λ is the whole story: large λ means heavy shrinkage and fewer survivors, small λ approaches OLS. The number reported is the cross-validated choice.
Compare the penalised and OLS columns. Shrinkage pulls coefficients toward zero - that is deliberate, trading a little bias for a large fall in variance.
The coefficient path, read right to left, shows the penalty tightening and variables leaving. The order they leave is the order the data finds them least useful.
Ridge keeps every regressor by construction. If the "kept" line says all of them, that is the penalty working as designed, not a failure to select.
In-sample R² is always higher for OLS. That comparison proves nothing - OLS minimises it by definition. Shrinkage is a bet about out-of-sample error.

A worked reading

Six growth regressors on 110 countries, with lasso and a penalty of λ = 0.01081 chosen by five-fold cross-validation. It keeps all six - which is the honest outcome here rather than a disappointing one: with six candidate regressors and no genuinely irrelevant ones to discard, there is nothing for the penalty to remove, and a method that dropped variables anyway would be doing harm. Compare the penalised and OLS columns and the shrinkage is visible but small: log initial GDP goes from -0.45465 to -0.44739, population growth from -0.49490 to -0.48291, each pulled a percent or two towards zero. In-sample R² falls from 0.6432 to 0.6426, which is what you pay. The standardised column is the one to read for importance, because the raw coefficients are on six different scales - investment has the smallest raw coefficient of the three largest standardised effects. And note what none of this licenses: the penalty was chosen by looking at the data, so a t-test on any surviving coefficient is not a valid test of anything.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Cross-validated lambda at the edge of the grid	The grid is too narrow	Widen it; the optimum is outside
Ridge and OLS almost identical	lambda near zero: no overfitting to fix	n is large relative to p. Shrinkage is not needed
Lasso keeping nothing	lambda far too large, or no signal	Check the cross-validation curve is U-shaped
Coefficients differing wildly across CV folds	Correlated predictors, unstable selection	Use elastic net or report selection frequencies
Reporting p-values on lasso-selected variables	Post-selection inference	Invalid. Use sample splitting or selective inference
Test error above the training error by a lot	Still overfitting	Increase lambda, or use the one-standard-error rule

Common mistakes

The big one: refitting OLS on the survivors and reporting those p-values. The same data chose the variables, so the standard errors are wrong and the p-values are far too small. This is post-selection inference and it needs a method built for it - post-double-selection, or splitting the sample.
Reading the survivors as "the true model". At the cross-validated λ, lasso over-selects: on simulated data with four real regressors among forty it typically keeps eight or more. It is tuned for prediction, not for recovery.
Forgetting to standardise, when doing this by hand elsewhere. The penalty is not scale-invariant.
Using k-fold cross-validation on time series. It trains on the future to predict the past.
Lasso with two strongly collinear regressors keeps one arbitrarily and drops the other. Which one survives can flip with a trivial change to the sample; Elastic Net is more stable when that matters.

Where to read this properly

Hoerl and Kennard (1970), "Ridge Regression", Technometrics 12(1).
Tibshirani (1996), "Regression Shrinkage and Selection via the Lasso", JRSS-B 58(1).
Zou and Hastie (2005), "Regularization and Variable Selection via the Elastic Net", JRSS-B 67(2).
Hastie, Tibshirani and Wainwright, Statistical Learning with Sparsity (2015).

Related Machine learning

Regression vs machine learning · Random forest · Gradient boosting

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Ridge / Lasso / Elastic Net

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Machine learning

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.