Common mistake 1 with Market basket analysis

Ranking by confidence. This is the classic error and it finds only your most popular items: the retail sample here has 26 rules with confidence above 50% and a lift of about 1, because most baskets contain bread.

Common mistake 2 with Market basket analysis

Reading a rule as causal. Putting two associated items on the same shelf may sell no more of either - the association was there before the shelf.

Common mistake 3 with Market basket analysis

Mining a long list and reporting what survives. With many items, thousands of pairs are tested and some clear any threshold by chance. A rule list is a shortlist to test, not a set of findings.

Common mistake 4 with Market basket analysis

Ignoring that lift is inflated for rare consequents. When an item appears in 1% of baskets, small absolute differences produce large lifts, so the top of a lift ranking is populated by rare items almost mechanically.

EconStatsCraft › Market basket analysis

Market basket analysis

Which items appear together more often than chance. Ranked by LIFT rather than confidence, because confidence alone just finds whatever is popular - a rule ending in bread is confident after almost anything.

What it needs

Basket or transaction id - One row per item per basket - long format.
Item

The intuition

Market basket analysis looks for products that are bought together more often than chance would predict. The trap is that the most frequent pairs are simply the most frequent products - bread and milk appear together because both appear everywhere. Lift is the correction: it compares the observed co-occurrence against what independence would give, and it is the only one of the three standard measures that does.

When to use it

Cross-sell and bundling, shelf and website layout, promotion design. Also any co-occurrence question outside retail - which symptoms present together, which courses students take together, which policies countries adopt together.

Where the estimator comes from

For a rule A\Rightarrow B: SUPPORT is P(A\cap B), the share of baskets containing both; CONFIDENCE is P(B\mid A)=P(A\cap B)/P(A), the share of A-baskets that also hold B; and LIFT is \text{lift}=\frac{P(A\cap B)}{P(A)P(B)}=\frac{P(B\mid A)}{P(B)}, the factor by which buying A multiplies the probability of buying B. Lift of 1 is independence. The Apriori algorithm makes the search tractable using the downward-closure property: if an itemset is infrequent, every superset of it is infrequent too, so whole branches can be pruned without being counted.

The model

\text{lift}(X \to Y) = \frac{P(Y \mid X)}{P(Y)} = \frac{P(X \cap Y)}{P(X)P(Y)}

What it assumes

Long format: one row per item per basket, with a column identifying the basket. A wide table of 0/1 columns has to be reshaped first.
That a basket is the right unit. Aggregating a customer’s year of shopping into one "basket" finds that they buy bread and also, separately, wine - which is not what a basket rule means.
Pairs, here. Longer itemsets exist and are where the Apriori algorithm earns its name, and also where a reader stops being able to act on the rule.
That the basket composition is not driven by the layout or promotion you are about to change. Rules learned under one store plan describe behaviour under that plan.

Statistical properties

Confidence is misleading on its own. A rule "buy X, therefore buy milk" can have 70% confidence purely because 70% of all baskets contain milk, and the rule adds nothing - its lift is 1.0.
Lift is symmetric: the lift of A implies B equals the lift of B implies A. So lift identifies association, never direction, and the arrow in the write-up is a choice rather than a finding.
Very low support rules are unstable. A rule seen in five baskets can have a spectacular lift and mean nothing, so a minimum support threshold is essential and should be reported.
Association is not causation and not even sequence. Nothing records which item entered the basket first, so "customers who buy A then buy B" is not something this data can support.
The number of candidate rules explodes combinatorially, so thousands of rules pass any threshold. Ranking and pruning are the real work, not the algorithm.
Leverage and conviction are alternative measures that behave better than lift for very rare or very common items, and are worth reporting alongside.

How to read the output

Lift is the number. 1 means independent - the rule tells you nothing, however confident it looks. Above about 1.2 is worth a second look; below it is noise.
Confidence is asymmetric and lift is symmetric. A → B and B → A have different confidences and the same lift, so the direction you act on is a business judgement rather than a statistical one.
Leverage is the same fact in units of baskets: how many more baskets contain the pair than independence would predict. Useful when a high lift sits on a tiny support and is not worth acting on.
Support gates everything else. A rule with a lift of 8 on 0.2% of baskets may rest on a few dozen transactions, and the lift of a rare pair is estimated far less precisely than the table suggests.

A worked reading

2,500 baskets over 14 items, average basket size 3.91, and 123 rules above the support floor. Sorted by lift the top pair is pasta → pasta sauce at 3.1024, then beer → nappies at 2.2456 - the second being the rule the whole literature is named after. Now look at what sorting by CONFIDENCE would have done instead. "pasta sauce → pasta" has 69% confidence and "beer → nappies" only 41%, so confidence ranks them in one order while lift ranks them in another, and lift is the one to trust: confidence is high whenever the consequent is common, regardless of whether the antecedent told you anything. Note also that lift is symmetric - pasta → pasta sauce and pasta sauce → pasta both score 3.1024 - while confidence is not (61% against 69%). That asymmetry is the only thing in the table that hints at direction, and it is not causal evidence: it reflects which item is rarer, not which purchase came first.

Symptom → diagnosis → what to do

If you see this	It means	Do this
High confidence, lift near 1	The consequent is simply common	The rule is worthless. This is the main trap
Enormous lift on tiny support	A handful of baskets	Raise the minimum support
Rules that are obvious	Common in retail data	The value is in the surprising middle of the ranking
Thousands of rules	Thresholds too loose	Tighten support and lift together
A rule and its reverse both reported	Lift is symmetric	Report one, and do not imply direction
Items that are variants of one product	Trivially associated	Aggregate to a sensible product level first

Common mistakes

Ranking by confidence. This is the classic error and it finds only your most popular items: the retail sample here has 26 rules with confidence above 50% and a lift of about 1, because most baskets contain bread.
Reading a rule as causal. Putting two associated items on the same shelf may sell no more of either - the association was there before the shelf.
Mining a long list and reporting what survives. With many items, thousands of pairs are tested and some clear any threshold by chance. A rule list is a shortlist to test, not a set of findings.
Ignoring that lift is inflated for rare consequents. When an item appears in 1% of baskets, small absolute differences produce large lifts, so the top of a lift ranking is populated by rare items almost mechanically.

Where to read this properly

Agrawal, Imielinski and Swami (1993), "Mining Association Rules between Sets of Items in Large Databases", SIGMOD.
Agrawal and Srikant (1994), "Fast Algorithms for Mining Association Rules", VLDB - the Apriori algorithm.
Brin, Motwani and Silverstein (1997), on lift and interest measures.
Hahsler, Grun and Hornik (2005), "arules", Journal of Statistical Software 14(15).

Related Marketing research

Conjoint analysis · MaxDiff (best–worst scaling) · TURF - reach of a product line · Price sensitivity (Van Westendorp) · Perceptual map · Scale reliability (Cronbach’s alpha) · Key driver analysis

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

Market basket analysis

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Marketing research

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.