Common mistake 1 with Monte Carlo experiment

Concluding that robust standard errors fix misspecification. Run the omitted-variable case with HC3: the coverage stays at zero, because the estimator is centred on the wrong number and no interval width repairs that.

Common mistake 2 with Monte Carlo experiment

Assuming one robust estimator handles everything. HC3 restores coverage under heteroskedasticity and does nothing under serial correlation - which this module shows, because HC3 on the autocorrelated case is as broken as the classical one.

Common mistake 3 with Monte Carlo experiment

Reading a coverage of 93% or 96% as a failure. With a few thousand replications there is simulation noise of a percentage point or so either way.

Common mistake 4 with Monte Carlo experiment

Turning replications up expecting a different answer. More replications sharpen the estimate of the coverage; they do not change what the coverage is.

EconStatsCraft › Monte Carlo experiment

Monte Carlo experiment

Estimates the same model on thousands of synthetic samples so you can see the sampling distribution itself, not one draw from it. It is the only way to check whether the standard errors are telling the truth - the coverage of a nominal 95% interval is measured rather than assumed.

The intuition

Every other module estimates something from data you cannot check. Monte Carlo inverts that: you choose the truth, generate data from it, run the estimator thousands of times, and compare what it reports against what you know. It is the only way to see an estimator's properties directly rather than taking a theorem's word for it - and the only way to find out what happens when the theorem's conditions do not hold.

When to use it

To settle any question of the form "does this matter". Every other result in this app is ONE draw from a sampling distribution, with the standard error as an estimate of that distribution’s width taken on faith. This is the only place you can check whether that faith is warranted.

Where the estimator comes from

Fix a data-generating process with known parameter \beta. For r=1,\dots,R: draw a sample, compute \hat{\beta}_r and its reported standard error \widehat{\mathrm{se}}_r, and record whether the nominal 95% interval covered \beta. Then BIAS is \frac{1}{R}\sum_r\hat{\beta}_r-\beta; the TRUE standard error is the standard deviation of \{\hat{\beta}_r\} across replications; and COVERAGE is the share of intervals containing \beta, which should be 0.95. The Monte Carlo standard error of a coverage estimate is \sqrt{p(1-p)/R}, so R=3{,}000 pins a coverage near 0.95 to about \pm0.4 percentage points - precise enough that a reading of 0.776 is unambiguous.

The model

\text{coverage} = \frac{1}{R}\sum_{r=1}^{R} \mathbf{1}\left[\,|\hat\beta_r - \beta| \le 1.96\,\widehat{se}_r\,\right]

What it assumes

Nothing about your data - it simulates its own. The dataset you have loaded is ignored, which is the point: the truth has to be known for any of this to be checkable.
That you change one thing at a time. The value is in the comparison, and a run with two things changed cannot attribute the difference to either.
The number of replications is large enough for the quantity you are reading. Estimating a rejection rate near 5% to within a percentage point needs on the order of two thousand replications, and a bias needs far fewer.
The random seed is not doing the work. A result that changes materially when the seed changes is a result about the seed.

Statistical properties

The comparison that matters is the AVERAGE REPORTED standard error against the actual spread of estimates. If the first is smaller, the formula is lying and every t-statistic built on it is too large.
Bias and coverage are separate failures. An estimator can be unbiased and still have terrible coverage, because coverage depends on the standard error being right as well as the point estimate.
Results are conditional on the data-generating process you chose. A Monte Carlo shows what happens under those assumptions and nothing more, so the design of the experiment is the substance.
The number of replications controls only the precision of the answer, not the answer itself. Report the Monte Carlo standard error so a reader can tell a real difference from simulation noise.
Setting the seed makes it reproducible; changing it should not change conclusions. If it does, R is too small.
This is how every robust standard error in this app was checked. A caveat that fires on a condition nobody has simulated is a guess.

How to read the output

Bias is about the CENTRE of the distribution: is the estimator aimed at the right number. Coverage is about its WIDTH: are the reported intervals honest. They are independent problems and the distinction is the most useful thing here.
"SE accuracy" compares the average reported standard error with how much the estimates really moved. Below 100% the standard errors are understating the uncertainty, and coverage falls below 95%.
The size of a 5% test is the same fact from the other side: with a true null it should reject 5% of the time. A reading of 22% means one test in five is a false positive.
A simulation shows what happens under the process you specified, and only that. It is a check on the estimator's arithmetic, not evidence about any real dataset.

A worked reading

Generate 3,000 samples of 100 observations with heteroskedastic errors and a true slope of 1, then estimate by OLS with CLASSICAL standard errors. The mean estimate is 1.0046, so bias is 0.0046 - OLS is unbiased under heteroskedasticity, exactly as the theory says. But the estimates actually spread with a standard deviation of 0.2734 while the average reported standard error is 0.1723, only 63.0% of the truth. Coverage of the nominal 95% interval is 77.6%. Nothing is wrong with the coefficient; everything is wrong with the confidence interval, and that is the distinction the whole exercise exists to make visible.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Coverage well below 95%	Standard errors too small	The reported uncertainty is understated
Coverage above 95%	Standard errors too large	Conservative: valid but wasteful of power
Bias near zero, coverage poor	The point estimate is fine, the SE is not	The commonest and most dangerous pattern
Results changing with the seed	Too few replications	Raise R and report the Monte Carlo standard error
Estimates below the truth far from 50%	A skewed sampling distribution	The symmetric interval is the wrong shape
Everything perfect	The DGP matches the estimator exactly	Break an assumption; that is the interesting experiment

Common mistakes

Concluding that robust standard errors fix misspecification. Run the omitted-variable case with HC3: the coverage stays at zero, because the estimator is centred on the wrong number and no interval width repairs that.
Assuming one robust estimator handles everything. HC3 restores coverage under heteroskedasticity and does nothing under serial correlation - which this module shows, because HC3 on the autocorrelated case is as broken as the classical one.
Reading a coverage of 93% or 96% as a failure. With a few thousand replications there is simulation noise of a percentage point or so either way.
Turning replications up expecting a different answer. More replications sharpen the estimate of the coverage; they do not change what the coverage is.

Where to read this properly

Davidson and MacKinnon, Econometric Theory and Methods (2004), ch. 4.
White (1980), Econometrica 48(4) - the robust covariance matrix this experiment motivates.
Morris, White and Crowther (2019), "Using Simulation Studies to Evaluate Statistical Methods", Statistics in Medicine 38(11).
Cameron and Trivedi, Microeconometrics (2005), ch. 11.

Related Tests & breaks

F-test, Wald & Chow · Structural breaks & stability

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Monte Carlo experiment

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Tests & breaks

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.