Common mistake 1 with Gradient boosting

Adding rounds until the training score looks good. That is the definition of overfitting; the hold-out score is what moves in the wrong direction.

Common mistake 2 with Gradient boosting

Assuming the defaults are right for your data. Boosting is more sensitive to its settings than a forest is.

Common mistake 3 with Gradient boosting

Using it for a causal question. The accuracy is real and the causal content is zero.

Common mistake 4 with Gradient boosting

Reading feature importance as an effect. It measures how often and how usefully a variable was split on, which rewards variables with many distinct values and splits importance arbitrarily between correlated predictors.

EconStatsCraft › Gradient boosting

Gradient boosting

Builds trees in sequence, each one fitted to what the previous ones got wrong. On tabular data of the kind banks and insurers hold, this is usually the most accurate method available - and the one most easily pushed into overfitting, so the training and hold-out columns are reported side by side.

What it needs

Outcome
Predictors

The intuition

A forest builds hundreds of independent trees and averages them. Boosting builds them one at a time, and each new tree is fitted to what the ensemble so far gets WRONG. That sequential focus on the residual is why boosting usually wins on tabular data, and also why it can overfit in a way a forest cannot: nothing stops it from eventually fitting the noise, so it needs early stopping.

When to use it

You want the best achievable prediction on tabular data and are willing to tune a little. It is the default choice in most bank and insurer model stacks.

Where the estimator comes from

Gradient boosting treats the ensemble as a function and does gradient descent in function space. Starting from F_0, at each step fit a small tree h_m to the negative gradient of the loss at the current predictions - for squared error this is exactly the residual - and update F_m(x)=F_{m-1}(x)+\nu\,h_m(x), where \nu is the learning rate. Shrinking each step with a small \nu and taking more of them generalises much better than taking a few large steps, which is the empirical finding that made the method work. XGBoost adds a second-order expansion of the loss and explicit penalties on leaf count and leaf weights.

The model

f_{m}(\mathbf{x})=f_{m-1}(\mathbf{x})+\nu\,h_{m}(\mathbf{x}),\qquad h_{m}\approx-\frac{\partial L}{\partial f_{m-1}}

What it assumes

The same independence and stability assumptions as a random forest.
That you will check the training-versus-hold-out gap. Boosting will drive training error towards zero if you let it, and that is not a good thing.
The training rows and the rows you will predict come from the same process. Boosting extrapolates badly by construction - a tree predicts a constant outside the range it saw - so a shift in the covariate distribution degrades it faster than it degrades a linear model.
The observations are independent. With time series or panel data, a random train/test split leaks the future into the past and the reported accuracy is meaningless; split by time or by entity instead.

Statistical properties

Boosting CAN overfit as trees are added, unlike a forest. The number of trees is a tuning parameter chosen by early stopping on a validation set, not something to maximise.
Learning rate and tree count trade off directly: halve the rate and you need roughly twice the trees. Small rates around 0.01 to 0.1 with many trees are the standard recipe.
Trees are deliberately shallow - depth 3 to 6 - because each is only meant to be a weak learner correcting a small part of the residual. Deep trees in a boosted ensemble overfit fast.
It is far more sensitive to hyperparameters than a random forest, which is close to a no-tuning method. That is the real cost of the extra accuracy.
Subsampling rows and columns per tree adds randomness and usually improves generalisation, borrowing the forest's decorrelation idea.
Because trees are added sequentially, boosting cannot be parallelised across trees the way a forest can, though the split search within a tree can be.

How to read the output

Learning rate and number of rounds trade off against each other: halve the rate and you need roughly twice the rounds.
A large training-to-hold-out gap means too many rounds, too deep trees, or too small a leaf size.
Compare against the random forest. If boosting wins only slightly, the forest is less fiddly and more robust to settings.
The learning rate and the number of trees trade off against each other: halving the rate and doubling the trees gives roughly the same fit. Quote both or neither, because one without the other says nothing about how hard the model was fitted.

A worked reading

On the same credit data the boosted model reaches roughly 0.92 AUC, comparable to the forest, but the route there is different and instructive. Watch the validation curve: it falls steeply for the first few hundred trees, flattens, and then begins to RISE. A forest's curve never rises. That difference is the whole distinction between averaging independent estimates and sequentially chasing the residual, and it is why a boosted model reported without an early-stopping criterion should not be believed.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Validation error rising after a point	Overfitting as trees accumulate	Early stopping. This is expected behaviour
Training error at zero	Trees too deep or too many	Reduce depth; lower the learning rate
Barely beating a single tree	Learning rate too high	Lower it and add trees
Taking a very long time	Small learning rate with many trees	The intended trade. Subsample rows to speed it up
Much worse than a forest	Untuned. Forests need no tuning; boosting does	Tune depth, rate and tree count together
Wildly different results across seeds	Subsampling with too few trees	Add trees, or fix the seed and say so

Common mistakes

Adding rounds until the training score looks good. That is the definition of overfitting; the hold-out score is what moves in the wrong direction.
Assuming the defaults are right for your data. Boosting is more sensitive to its settings than a forest is.
Using it for a causal question. The accuracy is real and the causal content is zero.
Reading feature importance as an effect. It measures how often and how usefully a variable was split on, which rewards variables with many distinct values and splits importance arbitrarily between correlated predictors.

Where to read this properly

Friedman (2001), "Greedy Function Approximation: A Gradient Boosting Machine", Annals of Statistics 29(5).
Chen and Guestrin (2016), "XGBoost: A Scalable Tree Boosting System", KDD.
Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2009), ch. 10.
Grinsztajn, Oyallon and Varoquaux (2022), "Why do tree-based models still outperform deep learning on tabular data?", NeurIPS.

Related Machine learning

Regression vs machine learning · Ridge / Lasso / Elastic Net · Random forest

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Gradient boosting

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Machine learning

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.