Common mistake 1 with Random forest

Reading importance as causal. If loan size and income both proxy for something unobserved, the forest will happily rank them highly while telling you nothing about what changing either would do.

Common mistake 2 with Random forest

Trusting the tree’s built-in impurity importance, which is biased towards variables with many distinct values. Use permutation importance, as this module does.

Common mistake 3 with Random forest

Deploying a model whose training and hold-out scores diverge sharply. That is memorisation, and it will not survive contact with new applicants.

Common mistake 4 with Random forest

Forgetting fair-lending constraints. A variable that predicts well may be illegal or unacceptable to use, and the model has no opinion about that.

EconStatsCraft › Random forest

Random forest

Grows hundreds of decision trees on bootstrap samples and averages them. Handles thresholds and interactions without being told they exist, which is why lenders and insurers use it. Reports permutation importance rather than the tree's own impurity measure, because the latter flatters continuous variables.

What it needs

Outcome - Classification or regression, detected from the outcome.
Predictors

The intuition

A single deep tree fits its training data almost perfectly and generalises badly, because it will happily split on noise. Grow hundreds of trees on bootstrap samples and average them, and the errors that are specific to each sample cancel while the real structure survives. The extra trick that makes it a random FOREST is restricting each split to a random subset of variables, which stops every tree from agreeing with every other.

When to use it

The task is prediction, the data are tabular, and you suspect the relationship is not a straight line. This is the workhorse of credit scoring and insurance pricing for exactly that reason.

Where the estimator comes from

Each tree is grown on a bootstrap resample, and at every split only m of the p features are considered - conventionally m=\sqrt{p} for classification and p/3 for regression. Averaging B trees with pairwise correlation \rho and individual variance \sigma^2 gives \mathrm{Var}=\rho\sigma^2+\frac{1-\rho}{B}\sigma^2. The second term vanishes as B grows, so more trees never hurt; the first does not, which is why decorrelating the trees by restricting m matters more than adding trees. Roughly 37% of observations are left out of each bootstrap, and predicting those from the trees that did not see them gives the out-of-bag error - a cross-validation estimate for free.

The model

\hat{f}(\mathbf{x})=\frac{1}{B}\sum_{b=1}^{B}T_{b}(\mathbf{x}),\qquad T_{b}\text{ grown on a bootstrap sample}

What it assumes

Rows are independent draws. With panel or time-series structure a plain random split overstates performance badly.
The future resembles the past. A model fitted to a benign credit cycle will mis-rank borrowers in a downturn.
Nothing about linearity, normality or homoskedasticity - that freedom is the whole appeal, and the reason there is no coefficient to interpret at the end.
The predictors available at training time will be available at prediction time, with the same meaning. A variable that is recorded later or defined differently in production silently destroys the model's accuracy without any error message.

Statistical properties

More trees never overfit. Increasing B reduces variance monotonically and then flattens; the only cost is time. Depth and leaf size are the parameters that control overfitting.
Out-of-bag error is close to k-fold cross-validation and costs nothing extra, which makes it the natural tuning criterion.
Impurity-based importance is BIASED towards high-cardinality and continuous variables, because they offer more places to split. Permutation importance is the honest alternative and should be preferred.
With correlated predictors, importance is split arbitrarily between them, so a genuinely strong variable can look unimportant because its twin absorbed the credit.
Forests cannot extrapolate. A prediction is an average of training values, so no input outside the training range produces an output outside it - a real limitation for time series with trend.
They handle interactions and nonlinearity without being told, which is precisely why they beat a logit on data generated by conjunctive rules.

How to read the output

The hold-out score is the only performance number worth quoting.
Permutation importance says how much worse predictions get when a column is shuffled. It has no sign and no units: it cannot tell you whether more leverage raises or lowers risk.
Partial dependence shows the shape the model learned. Kinks and plateaus there are what a linear model would have averaged away.
For a lender, the decile chart matters more than the AUC: it answers whether the riskiest tenth really does default more.

A worked reading

On the 5,000-row credit book, where default is driven by two conjunctive rules and a U-shape in age, a logistic regression reaches roughly 0.78 AUC and a forest around 0.92. The gap is the whole point: a logit reports one average slope per variable and cannot express "high debt-to-income AND a prior delinquency" without being told to look for it, whereas a tree discovers the conjunction by splitting twice. But the forest cannot tell you the interaction exists in any readable form, which is why partial-dependence plots exist and why the two methods are complements rather than rivals.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Training accuracy of 1.00	Normal for a forest; look at OOB	Not evidence of overfitting on its own
OOB error far above training error	Genuine overfitting	Raise the minimum leaf size or limit depth
Importance dominated by an ID-like column	High-cardinality bias	Drop it and use permutation importance
Predictions never beyond the training range	By construction	Do not use a forest to extrapolate a trend
Adding trees changing nothing	Converged	Correct behaviour. Stop adding them
Two correlated variables both looking weak	Importance split between them	Group them, or check with a single-variable model

Common mistakes

Reading importance as causal. If loan size and income both proxy for something unobserved, the forest will happily rank them highly while telling you nothing about what changing either would do.
Trusting the tree’s built-in impurity importance, which is biased towards variables with many distinct values. Use permutation importance, as this module does.
Deploying a model whose training and hold-out scores diverge sharply. That is memorisation, and it will not survive contact with new applicants.
Forgetting fair-lending constraints. A variable that predicts well may be illegal or unacceptable to use, and the model has no opinion about that.

Where to read this properly

Breiman (2001), "Random Forests", Machine Learning 45(1).
Strobl et al. (2007), "Bias in Random Forest Variable Importance Measures", BMC Bioinformatics 8.
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2009), ch. 15.
Athey and Imbens (2019), "Machine Learning Methods Economists Should Know About", Annual Review of Economics 11.

Related Machine learning

Regression vs machine learning · Ridge / Lasso / Elastic Net · Gradient boosting

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Random forest

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Machine learning

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.