Common mistake 1 with MaxDiff (best–worst scaling)

Reading the top item as "good enough to launch". MaxDiff cannot say that: it ranks, it does not scale against an absolute standard. Pair it with a purchase-intent question if that is the decision.

Common mistake 2 with MaxDiff (best–worst scaling)

Assuming the top-ranked items make the best PORTFOLIO. They usually do not, because preferences overlap - the product-test sample here is built so the highest MaxDiff item is absent from the best three-item line. That question is TURF.

Common mistake 3 with MaxDiff (best–worst scaling)

Averaging utilities across segments that want different things. The average person may not exist; estimate by segment if the segments disagree.

Common mistake 4 with MaxDiff (best–worst scaling)

Fielding items that differ in length or specificity. A concrete, vivid item beats an abstract one on wording alone, and MaxDiff will report that as a preference.

EconStatsCraft › MaxDiff (best–worst scaling)

MaxDiff (best–worst scaling)

Respondents pick the best and worst item from each small set, and the items come out on one scale. It discriminates where a battery of 1–5 ratings does not, because it forces a trade-off instead of letting everything be important.

What it needs

Respondent
Item
Task or set - Which set of items this row appeared in.

The intuition

Ask people to rate twelve flavours out of ten and most will give sevens and eights to everything, because rating scales invite agreement and different people use them differently. Ask instead which is BEST and which is WORST out of three, repeatedly, and there is nowhere to hide - every answer is a forced comparison, the scale-use problem disappears, and the resulting utilities are on a common ruler.

When to use it

Prioritising a long list where a rating scale collapses: features to build, messages to test, benefits to lead with, flavours to launch. Also whenever respondents from different cultures or segments use rating scales differently, since a forced choice removes scale-use bias entirely.

Where the estimator comes from

Under random utility, respondent i assigns item j a utility u_{ij}=\beta_j+\varepsilon_{ij} with \varepsilon extreme-value. The probability that j is chosen best from set S is the multinomial logit form \exp(\beta_j)/\sum_{k\in S}\exp(\beta_k), and the probability that l is chosen worst is the same with signs reversed. The best-worst pair is modelled as the choice of the ORDERED pair maximising \beta_j-\beta_l, giving P(j\text{ best},\,l\text{ worst})=\frac{\exp(\beta_j-\beta_l)}{\sum_{(a,b)\in S,\,a\ne b}\exp(\beta_a-\beta_b)}. Only differences are identified, so one item is fixed at zero. Shares are the softmax of the fitted utilities.

The model

P(\text{best}=i,\ \text{worst}=j\mid S)=\frac{e^{u_i}}{\sum_{k\in S}e^{u_k}}\cdot\frac{e^{-u_j}}{\sum_{k\in S\setminus\{i\}}e^{-u_k}}

What it assumes

A balanced design - every item shown roughly equally often, and appearing with every other item about equally. The module reports the exposure ratio, and unbalanced exposure makes the raw counting scores incomparable.
That the sets are small, typically four or five items. Beyond about six people stop evaluating and start scanning.
Independence of irrelevant alternatives, as in any logit. Two near-identical items in the same set share their support and both look weaker.
That respondents answered the exercise rather than clicking through it. MaxDiff is repetitive by design, and a respondent who picks the first and last item every time produces perfectly valid-looking data - screen for straight-lining and for implausibly short completion times before estimating anything.

Statistical properties

Best-worst extracts more information per question than a single choice, because worst-choice data is roughly as informative as best-choice and comes free with the same task.
It eliminates scale-use bias: there is no way to be a generous or a stingy rater when every answer is a comparison within a fixed set.
Utilities are identified only up to a location shift, so one item is normalised. Differences and shares are meaningful; a single utility is not.
The design matters as much as the model. Each item must appear roughly equally often and be paired with a balanced variety of others, or some comparisons are never observed and the utilities are poorly identified.
Shares from the softmax assume IIA, exactly as in multinomial logit, so near-identical items will split share implausibly. Hierarchical Bayes with respondent-level utilities is the standard repair and also gives individual-level scores.
The rank order is far more robust than the cardinal shares. Report the ranking with confidence, the shares with care.

How to read the output

The 0–100 score is RELATIVE - the best item always scores 100 and the worst 0, even if every item is terrible. It has no zero point and no absolute meaning.
The logit utilities are on an interval scale, so differences are comparable, and the shares are what a "share of preference" claim needs.
Where the counting score and the logit ranking disagree, look at the exposure balance first - that is usually the cause.
Utilities are identified only up to a constant, so the level means nothing and only differences do. Two studies cannot have their utilities compared directly, even with the same items, unless they share a design.

A worked reading

500 respondents, twelve flavours, three items per task, 2,000 tasks in total with exposure balanced to 1.00x. Mango comes out top with a logit utility of 1.9591 and a preference share of 28.0%, against Strawberry at 1.6918 and 21.4%. Read the utilities as DIFFERENCES rather than levels: subtract 1.6918 from 1.9591 and exponentiate, and the odds of Mango being chosen over Strawberry in a head-to-head are about 1.3 to 1 - a clear preference, and nothing like the 28-to-21 that the shares might suggest, because shares depend on the whole set on offer while the odds ratio does not. The 0-100 score column is a rescaling of the same utilities and carries no extra information; it exists because stakeholders ask for one. Note the worst-choice counts are used as well as the best: Pistachio is picked best 146 times and worst 140, which is a polarising item rather than a mediocre one, and a best-only design would have scored it identically to something nobody feels strongly about.

Symptom → diagnosis → what to do

If you see this	It means	Do this
All utilities near zero	No differentiation, or a broken design	Check items appear a balanced number of times
One item never chosen best or worst	Its utility is unidentified	It needs to appear in more tasks
Shares not matching the score ranking	Scores and softmax shares are different scales	Not an error
Two similar items splitting share oddly	IIA, as in any logit	Use hierarchical Bayes
Aggregate utilities hiding two segments	Averaging opposite preferences	Estimate individual-level utilities and cluster them
Very wide confidence intervals	Too few tasks per respondent	Four to six tasks per item is the usual minimum

Common mistakes

Reading the top item as "good enough to launch". MaxDiff cannot say that: it ranks, it does not scale against an absolute standard. Pair it with a purchase-intent question if that is the decision.
Assuming the top-ranked items make the best PORTFOLIO. They usually do not, because preferences overlap - the product-test sample here is built so the highest MaxDiff item is absent from the best three-item line. That question is TURF.
Averaging utilities across segments that want different things. The average person may not exist; estimate by segment if the segments disagree.
Fielding items that differ in length or specificity. A concrete, vivid item beats an abstract one on wording alone, and MaxDiff will report that as a preference.

Where to read this properly

Finn and Louviere (1992), "Determining the Appropriate Response to Evidence of Public Concern", Journal of Public Policy and Marketing 11(2).
Marley and Louviere (2005), "Some Probabilistic Models of Best, Worst, and Best-Worst Choices", Journal of Mathematical Psychology 49(6).
Louviere, Flynn and Marley, Best-Worst Scaling (2015).

Related Marketing research

Conjoint analysis · TURF - reach of a product line · Price sensitivity (Van Westendorp) · Market basket analysis · Perceptual map · Scale reliability (Cronbach’s alpha) · Key driver analysis

All analyses

Data & variables

Each example is simulated from a known model, so you can check whether the estimator recovers the truth - and nothing here is anyone's private data.

Filter variables

Drag a variable into a slot on the right - or just click it.

Choose an example dataset above to begin.

Choose a transformation - it opens in the data editor, with a live preview.

Apply to

Multiply by

Columns ctrl-click for several

Group by ctrl-click for several

Within groups

Statistic

Bands

How many

Cut points

Keep rows where

Value

and

Fill with

Value

Also mark which rows were filled

Rules

Base period = 100

Which part

Percent at each tail

Threshold

Survey weight optional

Direction

Names column

Values column

Largest first

Delete them, not just count them

Read this column as

Tidy the text

New name optional

EconStatsCraft rigorous, plain-spoken econometrics

Econometrics that explains itself.

Regression, panel data, time series and the whole battery of tests. Every result is followed by a verdict you can defend.

Load a dataset to see it here.

MaxDiff (best–worst scaling)

What it needs

The intuition

When to use it

Where the estimator comes from

The model

What it assumes

Statistical properties

How to read the output

A worked reading

Symptom → diagnosis → what to do

Common mistakes

Where to read this properly

See also

Related Marketing research

All analyses

Charts

Maps

Explore

Regression

Time series - one series

Time series - several series

Panel data

Multivariate statistics

Marketing research

Causal

Finance & risk

Actuarial

Tests & breaks

Machine learning

Econometrics that explains itself.