Common mistake 1 with Rolling-origin cross-validation

Using a single hold-out and reporting the winner as though it were established. That is the mistake this module exists to fix.

Common mistake 2 with Rolling-origin cross-validation

Letting any information from after the origin reach the model - a seasonal adjustment computed on the full sample, a variable transformed using the full-sample mean. This is leakage and it flatters the forecast enormously.

Common mistake 3 with Rolling-origin cross-validation

Comparing models on different samples. If one model drops the first 12 observations for lags, every model must be scored on the same origins.

Common mistake 4 with Rolling-origin cross-validation

Choosing on MAPE when the series goes near zero - the percentage explodes. Use MASE.

Common mistake 5 with Rolling-origin cross-validation

Re-tuning the model after seeing the cross-validation result, then reporting that result. It is no longer out-of-sample once you have used it to choose.

EconStatsCraft › Rolling-origin cross-validation

Rolling-origin cross-validation

The honest way to compare forecasting methods. A single hold-out gives one number per model and depends entirely on which slice of history landed in the test set; this re-forecasts from many origins and reports accuracy at every horizon.

What it needs

Series

The intuition

A single hold-out gives one number per model, and that number depends entirely on which slice of history happened to land in the test set. Rolling-origin evaluation re-forecasts from many origins and averages, so the comparison rests on dozens of forecasts rather than one. It also respects time: every forecast uses only data that existed when it would have been made.

When to use it

Whenever you are choosing between forecasting methods. A single hold-out is the default in most software and it is the weakest possible evidence: one number per model, from one arbitrary slice of history.

Where the estimator comes from

For each origin t=t_0,t_0+s,\dots, fit on y_{1:t} and forecast \hat{y}_{t+1},\dots,\hat{y}_{t+h}, then record the error at each horizon separately. The EXPANDING window uses all history up to t; the ROLLING window uses a fixed-length recent slice, which is preferable when the relationship drifts. Errors are summarised by horizon, since accuracy at h=1 and h=12 are different questions. Scale-free comparison uses MASE, \mathrm{MASE}=\frac{\frac{1}{h}\sum_j|e_{t+j}|}{\frac{1}{T-m}\sum_{t>m}|y_t-y_{t-m}|}, dividing by the in-sample mean absolute error of the seasonal naive forecast, so MASE below 1 beats that benchmark and the measure is comparable across series of any units.

The model

\text{RMSE}_h=\sqrt{\frac{1}{|O|}\sum_{t\in O}\bigl(y_{t+h}-\hat y_{t+h|t}\bigr)^{2}},\qquad \text{MASE}=\frac{\text{MAE}}{\frac{1}{n-m}\sum_{i=m+1}^{n}|y_i-y_{i-m}|}

What it assumes

The future resembles the past well enough that past forecasting performance predicts future performance.
Each forecast uses only data available at its own origin - no peeking. That is what the rolling design enforces.
The series is long enough that the first training window is still a sensible model.
The model is refitted at each origin rather than fitted once and rolled forward. Fitting on the whole sample and then evaluating on windows inside it uses the future to choose the parameters, which is the exact error cross-validation exists to prevent.

Statistical properties

A random train/test split is invalid for time series. It trains on the future to predict the past, and it inflates apparent accuracy substantially for any persistent series.
One hold-out is a single draw. Across 30 different realisations of the same process, the hold-out mean absolute error had a standard deviation of 28% of its own size - so a 20% difference between two models on one split is well within noise.
Error must be reported by horizon. A method that wins at one step ahead frequently loses at twelve, and averaging over horizons hides exactly the trade-off a user needs.
MASE is preferred to MAPE, which is undefined at zero, asymmetric between over- and under-prediction, and unusable for series that can be negative.
The Diebold-Mariano test asks whether a difference in accuracy is statistically distinguishable rather than merely visible. Forecast errors are autocorrelated across overlapping origins, so it needs a HAC variance.
Refitting the model at every origin is honest but slow; updating parameters only occasionally is a common and usually harmless compromise that should be stated.

How to read the output

MASE below 1 means the method beats a naive forecast; above 1 means it does not. It is scaled, so it is comparable across series with different units.
Read the error-by-horizon chart, not just the pooled table. A method that wins one step ahead often loses at eight, and the horizon you care about is the one to choose on.
The "origins fitted" column matters: a model that only converged on half the windows is fragile, whatever its average score.
Small RMSE differences between the top few methods are usually noise. Prefer the simpler model when they are close.

A worked reading

Sixty-four observations, 15 origins, horizons of one to three steps, 45 forecasts in total on an expanding window. The ranking is the interesting part: the MEAN forecast wins on every measure (RMSE 1.7483, MAE 1.1018, MASE 0.857), ahead of auto-ARIMA and Theta, with Naive and Drift last by a wide margin. A MASE below 1 means beating the naive benchmark on its own terms, and three of the six models manage it. The reason to evaluate this way rather than on one hold-out is that a single hold-out gives one number from a distribution wide enough to reverse most of these rankings - the gap between ARIMA at 1.8008 and Theta at 1.8193 is well inside the noise of any single split. Pooling 45 forecasts does not remove that noise but shrinks it by roughly the square root of the count, and the differences that survive pooling are the ones worth acting on.

Symptom → diagnosis → what to do

If you see this	It means	Do this
One model best at every horizon	Unusual; a genuine winner or a leak	Check that no future information enters
Errors not growing with horizon	Suspicious for most series	Verify the origins really are moving
MASE above 1	Worse than seasonal naive	A real finding, and a common one
Very few origins	The average is as noisy as a single split	Start earlier or use a smaller step
Rolling and expanding disagreeing sharply	The relationship is drifting	That is itself the finding; prefer rolling
MAPE infinite or absurd	Values at or near zero	Use MASE

Common mistakes

Using a single hold-out and reporting the winner as though it were established. That is the mistake this module exists to fix.
Letting any information from after the origin reach the model - a seasonal adjustment computed on the full sample, a variable transformed using the full-sample mean. This is leakage and it flatters the forecast enormously.
Comparing models on different samples. If one model drops the first 12 observations for lags, every model must be scored on the same origins.
Choosing on MAPE when the series goes near zero - the percentage explodes. Use MASE.
Re-tuning the model after seeing the cross-validation result, then reporting that result. It is no longer out-of-sample once you have used it to choose.

Where to read this properly

Tashman (2000), "Out-of-Sample Tests of Forecasting Accuracy", International Journal of Forecasting 16(4).
Hyndman and Koehler (2006), "Another Look at Measures of Forecast Accuracy", IJF 22(4) - where MASE comes from.
Diebold and Mariano (1995), Journal of Business and Economic Statistics 13(3).
Bergmeir and Benitez (2012), "On the Use of Cross-Validation for Time Series Predictor Evaluation", Information Sciences 191.

Related Time series - one series

Time-varying coefficients (Kalman) · Unit root tests · Correlogram (ACF / PACF) · ARIMA / SARIMA · Theta method · Unobserved components · STL forecasting · Forecast combination · Markov regime switching · Change frequency (resample) · ARCH / GARCH volatility · Decompose & filter · Spectral analysis · Exponential smoothing · Forecast accuracy

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Rolling-origin cross-validation

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Time series - one series

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.