Common mistake 1 with Classical statistical tests

Running many tests and reporting only the significant one. That is how you manufacture a false positive.

Common mistake 2 with Classical statistical tests

Treating "p ≥ 0.05" as proof of no difference. It means you could not detect one.

Common mistake 3 with Classical statistical tests

Using a paired test on unpaired data, or vice versa.

Common mistake 4 with Classical statistical tests

Running several group comparisons and reporting the significant one. Each test has its own error rate, and five at 5% gives roughly one false positive per analysis by construction.

EconStatsCraft › Classical statistical tests

Classical statistical tests

The standard toolkit from SciPy: compare two groups or many, test normality, test independence in a contingency table, and test correlations - each with an effect size and a plain-English verdict.

What it needs

Variable(s) - For one-sample, paired, normality and correlation tests.
Measurement
Group by - The column that defines the groups.
Row variable
Column variable

The intuition

Every test here compares a signal to the noise around it. The t-statistic is literally (difference between groups) ÷ (how much that difference bounces around under repeated sampling). If the ratio is large, the difference is bigger than sampling noise can comfortably explain.

When to use it

When your question is "do these two (or more) groups differ?" rather than "how does X affect Y holding Z constant?". For the second question you want regression.

Where the estimator comes from

Under H_0 the difference in means has expectation zero and estimable variance, so t=(\bar{x}_1-\bar{x}_2)/\mathrm{SE} follows a t distribution with the appropriate degrees of freedom. Welch does not pool the variances, using \mathrm{SE}=\sqrt{s_1^2/n_1+s_2^2/n_2} and adjusting the degrees of freedom by the Satterthwaite formula, which is why it stays valid when the groups have different spreads. ANOVA generalises this: F=\mathrm{MS}_{\text{between}}/\mathrm{MS}_{\text{within}}, the same idea with more than two groups.

The model

t=\frac{\bar{x}_1-\bar{x}_2}{\operatorname{SE}(\bar{x}_1-\bar{x}_2)} \qquad F=\frac{\text{between-group variance}}{\text{within-group variance}}

What it assumes

Observations are independent - this is the assumption that actually gets violated.
The t-test assumes roughly Normal data, or a large enough sample for the CLT.
Student's t assumes equal variances; Welch's does not, which is why Welch is the safer default.
ANOVA assumes equal variances across all groups - check with Levene.

Statistical properties

The t-test is exact under Normality and asymptotically valid without it, by the CLT.
Welch's test is almost never worse than Student's and is often much better - there is little reason to default to the pooled version.
Rank tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis) test a shift in distribution, not specifically in means, and lose only about 5% efficiency versus the t-test when the data really are Normal.
Power rises with sample size and effect size, and falls with variance. A null result on a small sample is uninformative, not evidence of no effect.

How to read the output

The p-value answers "if there were truly no difference, how often would I see a gap this big?"
Always read the effect size next to it. With 10,000 observations a meaningless difference will be significant.
Cohen's d: 0.2 small, 0.5 medium, 0.8 large.
ANOVA says "somewhere there is a difference"; Tukey HSD says where.

A worked reading

Welch's t = 8.4185, p < 0.0001, on a difference of 0.2144 log points between the two groups (means 2.4681 and 2.2538 on 575 and 625 observations). Read it as: one group earns about 21 log points - roughly 24% - more on average, and sampling noise alone would essentially never produce a gap this large. Two cautions belong with that. Welch's version is used rather than Student's because it does not assume the two variances are equal; here they are almost identical (0.4425 against 0.4387) so the two tests would agree, but that is luck rather than something to rely on, and Welch costs nothing when the assumption does hold. And a p-value this small says the difference is not zero, not that it is important - with 1,200 observations a difference of a tenth the size would also have cleared any threshold. The effect size is what makes it interesting, and 0.2144 log points is the number to report and defend.

Symptom → diagnosis → what to do

If you see this	It means	Do this
Levene rejects	Group variances differ	Use Welch's t-test, not Student's
Strong skew or outliers	Normality is doubtful in a small sample	Use Mann–Whitney or bootstrap the difference
Significant p but tiny d	Large sample detecting a trivial difference	Report the effect size and say plainly that it is small
ANOVA rejects	Some group differs, but which?	Tukey HSD, which corrects for multiple comparisons
χ² with expected counts below 5	The χ² approximation is unreliable	Combine categories, or use Fisher's exact test

Common mistakes

Running many tests and reporting only the significant one. That is how you manufacture a false positive.
Treating "p ≥ 0.05" as proof of no difference. It means you could not detect one.
Using a paired test on unpaired data, or vice versa.
Running several group comparisons and reporting the significant one. Each test has its own error rate, and five at 5% gives roughly one false positive per analysis by construction.

Where to read this properly

Wooldridge (2019), Introductory Econometrics: A Modern Approach, 7th ed., Cengage - Math Refresher C.6 (Appendix C.6 in earlier editions)
Delacre, Lakens & Leys (2017) - why Welch should be the default
Cohen (1988), Statistical Power Analysis - where the 0.2 / 0.5 / 0.8 conventions come from

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Classical statistical tests

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Explore

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.