Common mistake 1 with Forecast accuracy

Choosing a model on in-sample AIC and calling it the best forecaster. Often it is the worst.

Common mistake 2 with Forecast accuracy

Tuning the model until the hold-out looks good. That is just in-sample fitting with extra steps.

Common mistake 3 with Forecast accuracy

Not comparing against the naïve benchmark. For many financial series nothing beats it, and saying so is a legitimate finding.

Common mistake 4 with Forecast accuracy

Reading a single hold-out comparison as a ranking. One split gives one draw from a wide distribution - rolling-origin evaluation exists because that draw routinely reverses the order.

EconStatsCraft › Forecast accuracy

Forecast accuracy

Holds back the end of the sample, fits every standard forecasting model on the rest, and ranks them on out-of-sample error - including whether any of them actually beats a naïve random walk.

What it needs

Series

The intuition

A model can always be made to fit the past better by adding parameters. The only honest test of a forecast is data the model has never seen. This module hides the end of the sample, fits everything on the rest, and asks who actually wins - including whether anyone beats simply guessing that tomorrow equals today.

When to use it

Whenever you claim one model forecasts better than another. In-sample fit cannot answer that question.

Where the estimator comes from

Split at T-h. Estimate on 1,\dots,T-h, forecast h steps, compare with the held-out values. \mathrm{RMSE}=\sqrt{\frac{1}{h}\sum_t e_t^2} punishes large misses quadratically; \mathrm{MAE}=\frac{1}{h}\sum_t |e_t| is linear; \mathrm{MAPE}=\frac{100}{h}\sum_t |e_t/y_t| is scale-free but explodes near zero. Theil's U divides your RMSE by the naive random walk's. Diebold–Mariano tests whether the mean loss differential \bar{d}=\frac{1}{h}\sum_t (e_{0t}^2-e_{1t}^2) differs significantly from zero, using its long-run variance.

The model

\mathrm{RMSE}=\sqrt{\frac{1}{h}\sum_{t=1}^{h}(y_t-\hat{y}_t)^{2}}\qquad U=\frac{\mathrm{RMSE}_{\text{model}}}{\mathrm{RMSE}_{\text{naive}}}

What it assumes

The hold-out is representative - no regime change between training and test.
The hold-out is long enough for the comparison to mean anything.
The models were specified before seeing the hold-out. A backtest of models chosen by looking at the whole series measures how well you remembered it.
The forecast origin respects what was actually known at the time. Using revised data to forecast a period whose first release was different overstates accuracy for anything macroeconomic.

Statistical properties

In-sample fit and out-of-sample accuracy frequently rank models in OPPOSITE orders.
Theil's U < 1 means you beat the random walk; for many financial series nothing does, and reporting that is a legitimate result.
A single split is noisy; rolling-origin evaluation is more reliable when the sample allows it.
MAPE is undefined at zero and asymmetric - it punishes over-prediction more than under-prediction.
Diebold–Mariano is not valid for comparing nested models estimated on the same sample; use Clark–West there.

How to read the output

Lower RMSE is better; RMSE punishes big misses harder than MAE does.
MAPE is in percent and so comparable across series, but it explodes when the actual value is near zero.
Theil's U below 1 means the model beats a naïve random walk. Above 1 means it is worse than guessing today's value.
Mean error is bias: consistently positive means the model systematically under-predicts.
Diebold–Mariano tests whether the difference in accuracy is statistically real.

A worked reading

176 observations to train, 24 held out, every model forecasting the whole hold-out in one shot. The winner is the MEAN - a flat line at the training average - with RMSE 0.4182 and Theil's U of 0.8185, ahead of ARIMA(1,0,0) at 0.4315 and every differenced ARIMA behind that. The naive random walk comes last at 0.5109. Diebold-Mariano confirms the ordering rather than leaving it to eyeballing: Mean beats naive at p = 0.0172, the differenced ARIMAs beat it at p ≈ 0.001, and drift is significantly WORSE at p = 0.0023. Two lessons sit in that table. A constant can be a serious forecast - on a mean-reverting series over a 24-period horizon it is hard to beat, and any model that cannot beat it is not earning its complexity. And in-sample fit would have ranked these models in nearly the opposite order, which is the entire reason the hold-out exists.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Best in-sample model is worst out-of-sample	Over-fitting	Prefer BIC; prefer simpler models
Nothing beats the naive forecast	Series is close to a random walk	Report that honestly - it is the finding
Mean error consistently one sign	Systematic bias	Add a trend or drift term
Accuracy collapses at longer horizons	Normal	Report accuracy by horizon, not just overall
You tuned the model against the hold-out	Leakage	You need a third, untouched test set

Common mistakes

Choosing a model on in-sample AIC and calling it the best forecaster. Often it is the worst.
Tuning the model until the hold-out looks good. That is just in-sample fitting with extra steps.
Not comparing against the naïve benchmark. For many financial series nothing beats it, and saying so is a legitimate finding.
Reading a single hold-out comparison as a ranking. One split gives one draw from a wide distribution - rolling-origin evaluation exists because that draw routinely reverses the order.

Where to read this properly

Diebold & Mariano (1995)
Clark & West (2007)
Hyndman and Athanasopoulos (2021), Forecasting: Principles and Practice, 3rd ed., OTexts, chapter 5 (The forecaster's toolbox)
Makridakis et al. (2018) - the M4 competition results

Related Time series - one series

Time-varying coefficients (Kalman) · Unit root tests · Correlogram (ACF / PACF) · ARIMA / SARIMA · Rolling-origin cross-validation · Theta method · Unobserved components · STL forecasting · Forecast combination · Markov regime switching · Change frequency (resample) · ARCH / GARCH volatility · Decompose & filter · Spectral analysis · Exponential smoothing

All analyses

Time series - one series

Time series - several series

Yield curve (Nelson–Siegel / Svensson) · Dynamic factor model · Dynamic regression (ARIMAX) · Structural VAR · VAR · Granger causality · Cointegration · VECM · ARDL & bounds test

Panel data

Panel regression · Panel cointegration tests · Heterogeneous panel: Mean Group & PMG · Gravity model of trade · Dynamic panel GMM · Panel diagnostics · Panel unit-root tests

Multivariate statistics

Principal components (PCA) · Factor analysis · Cluster analysis

Marketing research

Conjoint analysis · MaxDiff (best–worst scaling) · TURF - reach of a product line · Price sensitivity (Van Westendorp) · Market basket analysis · Perceptual map · Scale reliability (Cronbach’s alpha) · Key driver analysis

Causal

Spatial autocorrelation · Spatial regression · Heckman selection · Instrumental variables (2SLS) · Difference-in-differences · Staggered difference-in-differences · Regression discontinuity · Matching & weighting · Oaxaca–Blinder decomposition

Finance & risk

Black–Scholes option pricing · Bond price, duration & convexity · Merton distance to default · EMI / home loan planner · CAPM / beta estimation · Multi-factor model · Portfolio optimization · Value at Risk & Expected Shortfall · Performance & drawdown · Time value of money & NPV/IRR · Equity valuation (DDM, FCF, residual income) · Cost of capital (WACC) · Binomial option pricing · Option strategies (payoff & profit) · Forwards, futures & swaps · Credit risk & Basel capital · Extreme value theory (tail risk)

Actuarial

Life table · Life annuity · Life insurance pricing · Chain ladder reserving · Buhlmann credibility · Survival analysis · Mortality graduation · Aggregate loss model · Ruin probability · Multi-state Markov model · Loss severity fitting

Tests & breaks

Monte Carlo experiment · F-test, Wald & Chow · Structural breaks & stability

Machine learning

Regression vs machine learning · Ridge / Lasso / Elastic Net · Random forest · Gradient boosting

Open the workspace →

Skip to results

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Forecast accuracy

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Time series - one series

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.