Common mistake 1 with Regression vs machine learning

Using a random split on time-series data. That leaks the future into training and produces a number that cannot be achieved in practice.

Common mistake 2 with Regression vs machine learning

Tuning settings against the hold-out set and then quoting that hold-out score. Once you have chosen on it, it is a training set.

Common mistake 3 with Regression vs machine learning

Treating a win for the forest as evidence about causes. It is evidence about prediction, and about nothing else.

Common mistake 4 with Regression vs machine learning

Comparing models on different preprocessing. If one model saw imputed missing values and another saw the rows dropped, the comparison is between pipelines rather than between models.

EconStatsCraft › Regression vs machine learning

Regression vs machine learning

Fits the regression, a single tree, a random forest and gradient boosting on identical training rows and scores them all on identical held-out rows. Answers the question students always ask - is machine learning better? - with a number rather than an opinion, and says plainly what you give up when the answer is yes.

What it needs

Outcome to predict - A 0/1 event or a number. Detected automatically.
Predictors - Text columns are turned into dummies for you.

The intuition

Every model looks good on the data it was fitted to. The only honest comparison holds out data the model has never seen, fits each candidate on the same training rows, and scores them on the same test rows. What makes this a method rather than a chore is the discipline: one split, applied identically, with no peeking at the test set while choosing anything.

When to use it

Before you decide that machine learning is or is not worth it. The answer depends entirely on whether the truth has thresholds and interactions, and that is an empirical question about your data, not a matter of taste.

Where the estimator comes from

Split the sample once into training and test sets, fit each candidate on the training set alone, and evaluate on the test set with a loss appropriate to the outcome - mean squared error or R^2 for a continuous outcome, AUC, log loss or accuracy for a binary one. AUC is P(\hat{p}_{\text{positive}}>\hat{p}_{\text{negative}}), the probability that a randomly chosen positive case is scored above a randomly chosen negative one, which makes it invariant to the classification threshold and to class imbalance in a way accuracy is not. Where hyperparameters must be chosen, they are chosen by cross-validation WITHIN the training set, never on the test set, or the test error is no longer an out-of-sample estimate.

The model

\text{fit on } \mathcal{D}_{\text{train}},\quad \text{score on } \mathcal{D}_{\text{test}},\qquad \mathcal{D}_{\text{train}}\cap\mathcal{D}_{\text{test}}=\varnothing

What it assumes

The rows are exchangeable, so a random split is meaningful. With time-series data it is not - use rolling-origin cross-validation instead, because a random split lets the model see the future.
The hold-out rows were never used for anything, including choosing settings.
The sample is large enough that the hold-out estimate is not mostly noise.
The metric matches the decision. Accuracy on an imbalanced outcome rewards predicting the majority class, and a model that never predicts the rare event can top the table while being useless.

Statistical properties

A single split is itself random. With a few hundred rows the test error has a standard error large enough that a small ranking difference between models is noise; repeated splits or cross-validation give a sounder comparison.
Accuracy is a poor measure under class imbalance. At 5% positives, predicting "never" scores 95% accuracy and is useless, which is why AUC or log loss should be reported instead.
Any choice made by looking at test performance - a hyperparameter, a feature set, even which models to include - leaks the test set and biases the estimate optimistically.
Preprocessing must be fitted on the training data only. Standardising with the full-sample mean before splitting leaks information, and the leak is small but real.
For time series a random split is invalid, because it trains on the future. Use a rolling origin, which the cross-validation module implements.
A complex model that ties a simple one on held-out data has lost. Interpretability, speed and stability all favour the simpler model when accuracy is equal.

How to read the output

Read only the hold-out column. Training scores always favour the most flexible model and tell you nothing about how it will behave on new borrowers.
If the regression is close to the best flexible model, take the regression: you get coefficients, standard errors and an argument you can defend.
If a flexible model wins by a wide margin, the truth has structure your linear model cannot express. Consider adding that structure by hand - a kink, an interaction - which often recovers most of the gain and keeps interpretability.
A small gap between models is not a ranking. Hold-out scores have sampling error of their own, and differences inside that error tell you nothing about which model to deploy.

A worked reading

On the credit book with a 70/30 split, logistic regression reaches about 0.78 AUC and both tree ensembles about 0.92. That gap is large enough to survive any reasonable split, and it has a cause worth naming: the data was generated with two conjunctive rules and a U-shape in age, none of which a linear index can express. The lesson generalises in one direction only. A 14 point AUC gap says the relationship is not linear; it does not say trees are better in general, and on data generated by a linear model the ranking reverses.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Test error below training error	An easy test split, or a leak	Check the split; repeat it
All models scoring nearly the same	The signal is linear, or there is little of it	Report the simplest one
Very high accuracy with rare positives	Class imbalance	Report AUC and the confusion matrix, not accuracy
Ranking flipping across seeds	The split is too small to distinguish them	Use repeated cross-validation
A model far ahead of everything	Suspect a leak	Look for a feature that encodes the outcome
Test set used for tuning	The estimate is no longer out of sample	Tune inside the training set only

Common mistakes

Using a random split on time-series data. That leaks the future into training and produces a number that cannot be achieved in practice.
Tuning settings against the hold-out set and then quoting that hold-out score. Once you have chosen on it, it is a training set.
Treating a win for the forest as evidence about causes. It is evidence about prediction, and about nothing else.
Comparing models on different preprocessing. If one model saw imputed missing values and another saw the rows dropped, the comparison is between pipelines rather than between models.

Where to read this properly

Hastie, Tibshirani and Friedman, The Elements of Statistical Learning (2009), ch. 7.
Hanley and McNeil (1982), "The Meaning and Use of the Area under a ROC Curve", Radiology 143(1).
Varma and Simon (2006), "Bias in Error Estimation when Using Cross-Validation for Model Selection", BMC Bioinformatics 7.
Kuhn and Johnson, Applied Predictive Modeling (2013), ch. 4.

Related Machine learning

Ridge / Lasso / Elastic Net · Random forest · Gradient boosting

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Regression vs machine learning

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Machine learning

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.