Common mistake 1 with Markov regime switching

Label switching: the model numbers the regimes arbitrarily, and a re-run can swap them. Never report "regime 1" without saying what its estimated mean is.

Common mistake 2 with Markov regime switching

Testing the number of regimes with a likelihood ratio test. The parameters are unidentified under the null, so the usual chi-square distribution does not apply - this is the Davies problem.

Common mistake 3 with Markov regime switching

Reading smoothed probabilities as if they were available in real time. They are not; they are a description of history.

Common mistake 4 with Markov regime switching

Fitting many regimes to a short sample. Two regimes and fifty observations is already ambitious.

Common mistake 5 with Markov regime switching

Concluding "the model found the 1991 crisis" when you chose two regimes on a series you already knew had one crisis.

EconStatsCraft › Markov regime switching

Markov regime switching

Some series are not one process - they are two, with the economy moving between them. This estimates each regime’s mean and variance and the probability of being in each one at every date, without being told when the switches happened.

What it needs

Series

The intuition

Some series do not have one set of parameters, they have two: expansions and recessions, calm markets and crises, before and after a policy change. A Markov-switching model does not require you to say when the regimes occur - it estimates both the parameters within each regime and the probability that each date belonged to each one, which is why it can date a recession without being told where recessions are.

When to use it

When a single set of parameters cannot describe the whole sample: expansions versus recessions, calm versus crisis markets, before and after a policy change whose date you do not want to impose.

Where the estimator comes from

Let s_t\in\{1,\dots,K\} be an unobserved state following a Markov chain with transition probabilities p_{ij}=P(s_t=j\mid s_{t-1}=i), and let the observation density depend on the state, y_t\mid s_t=k\sim N(\mu_k+\phi_k y_{t-1},\sigma_k^2). The likelihood integrates over every possible state path, which would be K^T terms, but Hamilton\'s filter does it recursively: predict P(s_t=k\mid \mathcal{F}_{t-1}), update with the new observation by Bayes, and accumulate the log-likelihood. The smoothed probabilities P(s_t=k\mid \mathcal{F}_T) come from a backward pass, and the expected duration of regime k is 1/(1-p_{kk}) - a direct and interpretable reading of the transition matrix.

The model

y_t=\mu_{S_t}+\varepsilon_t,\quad \varepsilon_t\sim N(0,\sigma^2_{S_t}),\qquad P(S_t=j\mid S_{t-1}=i)=p_{ij}

What it assumes

The regime process is Markov - the probability of next period’s regime depends only on this period’s.
The number of regimes is fixed by you in advance. It cannot be tested by a standard likelihood ratio test.
Within a regime the parameters are constant and the errors Normal.
The regimes are persistent enough to be identified. If switching is frequent relative to the sample, the likelihood has trouble separating a two-regime process from one with fat-tailed errors.
There is enough data in each regime. A regime occupying five per cent of a short sample has its parameters estimated from very few observations, however confident the output looks.

Statistical properties

Regime labels are arbitrary. Nothing distinguishes "regime 1" from "regime 2" except the parameters, so the same model may return them swapped between runs - the label-switching problem.
The likelihood is multimodal. Different starting values reach different optima, so multiple starts are essential and a single fit should not be trusted.
Testing K = 1 against K = 2 is non-standard: under the null the transition probabilities are unidentified, so the usual likelihood ratio does not have a chi-squared distribution. Use the Hansen or Carrasco-Hu-Ploberger bounds.
Expected duration 1/(1-p) is the most interpretable output. A p11 of 0.95 on monthly data means an expected regime length of 20 months, which can be checked against known history.
The model will find two regimes in data generated by a single process with fat tails, because a high-variance state fits outliers well. Compare against a GARCH alternative before concluding regimes exist.
Smoothed probabilities use the whole sample and are far more decisive than filtered ones. Real-time recession dating must use filtered probabilities, and they are much less clean.

How to read the output

The estimated means identify the regimes: the one with the low mean and high variance is the crisis state.
The diagonal of the transition matrix is persistence. Expected duration is 1/(1 − persistence) - a regime lasting under two periods on average is probably noise.
The smoothed probabilities use the whole sample. For a real-time reading you need the filtered probabilities instead.
Check the high-probability dates against events you can name. If they line up with crises you recognise, the model has found something real.
Smoothed probabilities use the whole sample and filtered ones use only the past. For dating regimes historically the smoothed series is right; for anything resembling a real-time decision only the filtered one was available.
The transition matrix implies an expected duration for each regime - one over one minus the staying probability. If that duration is implausible for the phenomenon, the model has found something other than what you meant.

A worked reading

Fitted to a growth series, the model returns two states with p11 = 0.95 and p22 = 0.78, so expected durations are 1/(1-0.95) = 20 periods for the first and 1/(1-0.78) = 4.5 for the second - a long calm state and a short volatile one, which is the recognisable shape of expansions and recessions. That duration reading is the strongest check available: if the estimated regimes had durations of two periods, they would be fitting outliers rather than dating a cycle, whatever the log-likelihood said.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Regimes swapping between runs	Label switching	Order by a parameter, such as the mean
One regime occupying three observations	Fitting outliers, not a regime	Constrain the minimum duration or reduce K
Very different results across starts	Multimodal likelihood	Use many starts and report the best
Transition probability at 1.00	An absorbing state	Usually a boundary artefact rather than economics
Two regimes found in simulated single-regime data	A known failure mode with fat tails	Compare against GARCH
Smoothed probabilities near 0 or 1 everywhere	Normal, and partly an artefact of using the whole sample	Show filtered for real-time claims

Common mistakes

Label switching: the model numbers the regimes arbitrarily, and a re-run can swap them. Never report "regime 1" without saying what its estimated mean is.
Testing the number of regimes with a likelihood ratio test. The parameters are unidentified under the null, so the usual chi-square distribution does not apply - this is the Davies problem.
Reading smoothed probabilities as if they were available in real time. They are not; they are a description of history.
Fitting many regimes to a short sample. Two regimes and fifty observations is already ambitious.
Concluding "the model found the 1991 crisis" when you chose two regimes on a series you already knew had one crisis.

Where to read this properly

Hamilton (1989), "A New Approach to the Economic Analysis of Nonstationary Time Series", Econometrica 57(2).
Kim and Nelson, State-Space Models with Regime Switching (1999).
Hansen (1992), "The Likelihood Ratio Test under Nonstandard Conditions", Journal of Applied Econometrics 7(S1).
Carrasco, Hu and Ploberger (2014), Econometrica 82(3), for the identification-robust test.

Related Time series - one series

Time-varying coefficients (Kalman) · Unit root tests · Correlogram (ACF / PACF) · ARIMA / SARIMA · Rolling-origin cross-validation · Theta method · Unobserved components · STL forecasting · Forecast combination · Change frequency (resample) · ARCH / GARCH volatility · Decompose & filter · Spectral analysis · Exponential smoothing · Forecast accuracy

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Markov regime switching

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Time series - one series

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.