Common mistake 1 with Forecast combination

The big one: choosing weights on the same data you use to report accuracy. The combination is then scored on data it was tuned to and will look far better than it is. This module uses three separate slices to avoid exactly that.

Common mistake 2 with Forecast combination

Combining ten variants of the same model. Diversity of error is what creates the gain, not the number of members.

Common mistake 3 with Forecast combination

Estimating many weights from a short validation stretch. With five models and eight validation points, OLS weights are noise.

Common mistake 4 with Forecast combination

Assuming the best individual model is the best member of a combination - a weak-but-different model often contributes more than a strong-but-similar one.

EconStatsCraft › Forecast combination

Forecast combination

Bates and Granger showed in 1969 that averaging forecasts usually beats choosing between them, because different models make different mistakes. This fits several, learns weights on data none of them saw, and judges the result on data the weights never saw either.

What it needs

Series to forecast

The intuition

Different forecasting methods make different mistakes. Average them and the mistakes partly cancel, which is why a simple mean of several forecasts is a famously hard benchmark to beat. But the folklore overstates it: averaging protects you from picking the WORST model, and it does not generally beat the best one - and if one member is badly wrong, it drags the average down with it.

When to use it

Almost always, once you have more than one credible model. Combination is one of the most reliably replicated findings in the whole forecasting literature.

Where the estimator comes from

For two unbiased forecasts with error variances \sigma_1^2,\sigma_2^2 and correlation \rho, the combination w f_1+(1-w)f_2 has variance w^2\sigma_1^2+(1-w)^2\sigma_2^2+2w(1-w)\rho\sigma_1\sigma_2, minimised at w^=\frac{\sigma_2^2-\rho\sigma_1\sigma_2}{\sigma_1^2+\sigma_2^2-2\rho\sigma_1\sigma_2}. The gain comes from \rho<1: identical forecasts cannot help each other. Bates and Granger derived this in 1969; the awkward empirical fact, known as the forecast combination puzzle, is that estimating w^ usually does WORSE out of sample than setting w=1/N, because the variances and correlation must themselves be estimated and that noise costs more than the optimal weighting gains.

The model

\hat y^{c}_{t+h}=\sum_{i=1}^{k}w_i\hat y^{(i)}_{t+h},\qquad \sum_i w_i=1;\qquad w^{\text{inv}}_i=\frac{1/\text{MSE}_i}{\sum_j 1/\text{MSE}_j}

What it assumes

The individual forecasts are produced without seeing the data used to score them.
The weights are estimated on a stretch that the models never saw, and applied to a further stretch the weights never saw.
The models make partly independent errors - combining two near-identical models achieves nothing.
The forecasts are for the same target over the same horizon. Combining a one-step and a three-step forecast, or forecasts of a level and of a growth rate, averages quantities that are not the same thing.

Statistical properties

Equal weights are hard to beat and are the right default. Estimated optimal weights require estimating a covariance matrix from few observations, and the estimation error usually exceeds the theoretical gain.
The gain comes from DIVERSITY, not from quality. Adding a fifth method that behaves like the other four adds nothing; adding one that errs differently helps even if it is individually worse.
A badly wrong member does real damage. Averaging is not robust - one member with ten times the error of the others will dominate the mean absolute error of the combination.
The median is more robust than the mean for exactly that reason, and is worth reporting alongside when the members disagree sharply.
Combining forecasts that are all biased in the same direction cannot remove the bias. Combination addresses variance, not systematic error.
Trimmed means - dropping the highest and lowest before averaging - capture most of the robustness of the median while keeping more information.

How to read the output

Compare the best combination against the best individual model. If combining helps, that is the usual result and the reason to do it.
OLS weights can be negative or above one. That is legitimate - a negative weight means one forecast is useful as a correction to another - but it over-fits the validation stretch more readily.
If equal weights beat the cleverer schemes, believe it. That is the famous "forecast combination puzzle".
The number of weights estimated relative to the number of observations is the thing to watch. Five weights from eight observations is not estimation, and it is why constrained and equal-weight schemes exist.

A worked reading

Five methods on a quarterly series: fit on observations 1-48, weights learned on 49-56, everything judged on 57-64. On the test stretch the individual RMSEs run from 2.4239 (Naive) to 2.4773 (ETS), and the best combination - equal weights - scores 2.4273. So combining came 0.1% WORSE than simply using the naive forecast, and the result card says so rather than claiming a win. That is worth seeing, because the forecasting literature's headline finding is that combinations usually beat their members, and "usually" is doing real work in that sentence: here the members are close together and the best one is already near the floor, so there is nothing for averaging to buy. Two details repay attention. The OLS weighting scheme returns 2.261 on Naive and -4.024 on Drift - wildly overfitted weights from eight observations, which is why constrained schemes exist. And ARIMA has the BEST mean absolute error (1.3881) while ranking sixth on RMSE, so which model you call best depends on whether you care more about typical errors or large ones.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Combination worse than the best member	Common and expected	It protects against choosing badly, not against the best
One member wildly off	It is dragging the mean	Trim it, or use the median
Members almost identical	No diversity, nothing to gain	Combination will match them and cost effort
Estimated weights beating equal weights in sample	Overfitting the weights	Check out of sample; usually it reverses
A negative optimal weight	Legitimate when correlation is high	But very unstable; constrain to non-negative
All members biased the same way	Combination cannot fix bias	Correct the bias first

Common mistakes

The big one: choosing weights on the same data you use to report accuracy. The combination is then scored on data it was tuned to and will look far better than it is. This module uses three separate slices to avoid exactly that.
Combining ten variants of the same model. Diversity of error is what creates the gain, not the number of members.
Estimating many weights from a short validation stretch. With five models and eight validation points, OLS weights are noise.
Assuming the best individual model is the best member of a combination - a weak-but-different model often contributes more than a strong-but-similar one.

Where to read this properly

Bates and Granger (1969), "The Combination of Forecasts", Operational Research Quarterly 20(4).
Clemen (1989), "Combining Forecasts: A Review", International Journal of Forecasting 5(4).
Smith and Wallis (2009), "A Simple Explanation of the Forecast Combination Puzzle", Oxford Bulletin 71(3).
Makridakis, Spiliotis and Assimakopoulos (2020), on the M4 competition results.

Related Time series - one series

Time-varying coefficients (Kalman) · Unit root tests · Correlogram (ACF / PACF) · ARIMA / SARIMA · Rolling-origin cross-validation · Theta method · Unobserved components · STL forecasting · Markov regime switching · Change frequency (resample) · ARCH / GARCH volatility · Decompose & filter · Spectral analysis · Exponential smoothing · Forecast accuracy

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Forecast combination

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Time series - one series

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.