MaxDiff Simulation

1. What Is MaxDiff Simulation?

A MaxDiff (best-worst scaling) study asks people which item they like most and least, again and again, across many small sets. After estimation (Hierarchical Bayes), every respondent ends up with an individual preference score for each item. On its own, that is just a pile of numbers. MaxDiff Simulation is the page where you turn those scores into decisions.

The page is organised as a short sequence of analyses, and you meet them in this order:

1. Item Rankings. The page you land on first. How do the items rank, under several definitions of "preferred"? This is your starting point: read it before anything else.
2. Assortment. If only a chosen set of items were on offer, what share would each win?
3. TURF Analysis. Which small set of items reaches the most people? (TURF is explained in full in Section 6.)
4. Anchor Analysis. For Anchored MaxDiff studies only: what is each item's absolute purchase probability, not just its relative rank?

The car-feature example used throughout this guide

A car manufacturer ran a MaxDiff study on eight features and now wants to know which features to feature prominently. We will use this study in every section:

2. Prerequisites

The MaxDiff Simulation page is a consumer of analysis results, not an estimator. Before it shows anything, two things must already exist.

3. The MaxDiff Simulation Workspace

The page has a left sidebar for choosing what to analyse and a main area with one page per analysis. The sidebar is always visible.

Run picker

The Run dropdown (a standard <select> control) lists every completed MaxDiff run in the project. Each option carries its own task, so picking a run from a different task (for example switching between a normal and an Anchored MaxDiff study) switches the task automatically. Choicery detects whether the selected run is anchored and shows the Anchor Analysis page accordingly.

Filters (segment filter)

The Filters button restricts every analysis on the page to a subset of respondents, defined by their answers to other survey questions (for example "women under 35" or "premium buyers"). When a filter is active, rankings, share-of-choice, and TURF reach recompute on that subset. The Anchor Analysis calibration cards remain a full-run diagnostic and say so explicitly. This is the same shared segment filter used by the Market Simulator.

Page navigation

The main area has up to four pages, selected from the Pages list:

4. Item Rankings (Five Metrics)

The Item Rankings page ranks all MaxDiff items. A single Metric dropdown switches between five views of the same individual utilities. Each metric answers a slightly different question, so they can produce different orderings, which is exactly why all five are offered.

The five metrics

Why First Choice can disagree with Relative Preference

First Choice is "winner takes all" per respondent, so it rewards items that are somebody's absolute favourite, even a polarising item that many others rank last. Relative Preference is a smoother average. An item that is nobody's number one but everybody's solid number two can rank higher on Relative Preference than on First Choice. Reading both protects you from a single misleading view.

5. Assortment Simulator

The Assortment page answers: "If the only options were the ones I tick, what share would each capture?" Tick the items in the Select Assortment card, click Simulate, and Choicery computes a share of choice on the individual utilities and averages across respondents. At least two items must be ticked; with fewer, the page answers "Select at least 2 items."

Use this page to compare line-ups directly: "if we only promoted Safety, Fuel Efficiency, and Design, Safety takes almost half the attention." It is the MaxDiff cousin of CBC market simulation, but on items rather than full product profiles.

6. TURF Analysis

This is the heart of the page. You set up to four controls, click Run TURF, and Choicery searches for the item sets that reach the most respondents.

The four settings

How the default K is chosen. When you do not set K yourself, Choicery uses the top quartile of the item list, capped at 5: K = max(1, min(5, ceil(number of items / 4))). In practice that is 6 items → 2, 8 → 2, 10 → 3, 12 → 3, 16 → 4, 20 or more items → 5. A fixed default of 5 would make reach trivially high in short lists: with "top 5 of 6" almost every item reaches almost every respondent, so every assortment looks equally good. Tying K to the list length keeps "reached" meaningful across studies of different sizes. A K you type in is used as entered; it is only limited to one below the number of items, and the results card says so when that happens.

There is no execution-mode setting. A search that does not fit the interactive work budget (Section 12) is handed to the analysis worker automatically when a worker is available, and the results appear when it finishes (the page waits up to 15 minutes). Without a worker, Choicery returns a message asking for a respondent segment, a smaller assortment or fewer items.

Reach definitions

The Reach Definition is the most important choice on this page. It decides how strict "acceptable" is, and it can completely change the answer. Choicery offers five definitions for every MaxDiff run, plus two more on anchored runs:

Reading the results

Before you run anything the results card is headed TURF Results. After Run TURF it is replaced by a card headed Optimal K-Item Assortment (with K the assortment size you chose), and the table inside it carries the sub-heading Top N Combinations, or Best Combination when only one is returned. It lists the best item combinations, ranked by reach. The best row is highlighted. Each combination shows its item IDs (the bracketed numbers), the item names, its reach and its frequency. Under the Weighted by probability reach definition the table changes its two right-hand columns: the ID column is headed Portfolio, and instead of Freq. you get Total, the summed portfolio probability across respondents.

Reach and frequency

• Reach is the percentage of respondents reached by the set (at least one acceptable item). Higher is better and is what TURF maximises.
• Frequency is the average number of acceptable items per respondent within the set. Higher frequency means more redundancy: people have several items they like, which can mean the set is over-serving some people while leaving others uncovered. For pure coverage, a high reach with modest frequency is ideal.

The TURF ladder

Alongside the best combinations, Choicery builds a ladder up to your Ladder Size. It is a greedy build: start with the single item that reaches the most people, then add the item that adds the most new reach, and so on. Each row names the added item, its Gain (the marginal reach that item adds on top of the previous step), the Cumulative Reach of everything selected so far, and a Status column that reads "Adds reach" or, once reach has plateaued, "Redundancy only". The cumulative reach bar shows how quickly you saturate.

Reach often plateaus before the ladder is finished: at some point no remaining item can raise it any further. The ladder does not stop there. It keeps building to the size you asked for and marks those steps Redundancy only, with a Gain of +0.0%. At those steps, remaining candidates are ranked by Frequency instead of additional reach. With a binary reach rule, Frequency is the average number of selected items that pass each respondent's reach criterion; under Weighted by probability, it is the average summed item-probability mass. A candidate can tie on both measures, so adding it does not necessarily increase Frequency. A note under the table names the plateau value. Note that the plateau is not always 100%: with a binary reach rule, part of the sample can be unreachable by any item in the pool.

7. Anchor Analysis (Anchored MaxDiff only)

Standard MaxDiff is purely relative: it tells you item A beats item B, but not whether anybody actually wants either. Anchored MaxDiff adds a calibration screen where each respondent answers "Yes" or "No" for every item. Combining the relative HB utilities with those yes/no answers yields an absolute purchase probability per item. This page only appears for anchored runs.

The page opens with a collapsible How Anchored MaxDiff Works panel that summarises the method in two sentences. Below it sit three result cards:

On anchored runs you also gain two extra reach definitions on the TURF Analysis page: Calibration "Yes" answers (raw), where "reached" means the respondent actually said yes on the calibration screen, and Utility above anchor (model-based), where "reached" means the item's HB-estimated utility exceeds the respondent's anchor threshold. The raw option is the most literal; the model-based option uses the smoothed HB estimate and is robust to single noisy calibration clicks.

8. Exports and Reports

Every result on the page can leave it. There are four export channels.

The Download PDF button appears when results are loaded and your account has export permission for the project (without it the sidebar shows only the Methodology Guide link). If rankings, TURF, or assortment results are not current for the active filters, Choicery prepares them before queuing the report. While rendering, the button becomes Cancel PDF; polling stops after two minutes with a timeout message.

What is inside the XLSX exports

The Item Rankings XLSX exports bundle every metric. Use Summary XLSX for a compact shareable workbook or Full XLSX when respondent-level sheets are required:

• A Rank Comparison sheet (each item's rank under all methods, sorted by Relative Preference).
• A summary sheet per metric: Relative Preference (share and standard deviation), First Choice Summary and Softmax Pref Summary (share, standard error, and the lower and upper 95% confidence bounds), and Shapley Summary (total reach, percent reach, contribution).
• Full XLSX only: an individual sheet per respondent-level metric (First Choice Individual, Softmax Pref Individual, Shapley Individual), one row per respondent and one column per item. Relative Preference has no individual sheet.
• A Settings sheet documenting the run, respondent and item counts, the checksums of the input and result, the segment-filter flag, and method notes.

The TURF Results XLSX contains:

• A Summary sheet: the top combinations, the ladder, and per-item reach.
• An Individual Results sheet: per respondent, which items are reach-eligible and how each of the ten best combinations scored for them.
• A Settings sheet recording the assortment size, ladder size, reach definition, top-K, and whether a segment filter was applied.

9. Best Practices & Pitfalls

Do

• Start with the default reach definition (Item is in respondent's top-K). It is the classic counting approach and avoids the trivial-100%-reach trap.
• Compare the TURF set against the naive "top-K by ranking" set. The gap is the story.
• Read the ladder for diminishing returns. If reach barely moves after item 3, you have found your natural assortment size; do not pay for a fourth item that adds two points.
• Use the Filters button to check whether the best set differs by audience. The optimum for premium buyers may not be the optimum overall.
• On anchored studies, prefer the reach definition Calibration "Yes" answers (raw) and cross-check P(Accept) against the raw calibration rate.

Don't

• Don't read TURF off the Item Rankings page. The best three items individually are rarely the best three together.
• Don't use "above the mean (lenient)" for a many-item study and then celebrate 100% reach. It is meaningless.
• Don't treat MaxDiff share of choice as absolute demand. There is no None option; the shares are relative within the ticked set.
• Don't compare reach numbers across different reach definitions. They are different questions; only compare sets evaluated under the same definition.
• Don't push assortment size too high. The combination search is capped (Section 12); a huge size with many items can exceed the limit and Choicery will ask you to narrow it.

10. Key Definitions

11. Reach Definitions: The Mathematics

Let uij be respondent i's utility for item j (probability scale, summing to 100 over j), and let zij be the same utility zero-centered per respondent. For a portfolio S, respondent i is reached when any item in S passes the rule.

Weighted by probability

Selected items contribute jointly. For standard MaxDiff, their exponentiated zero-centered utilities are compared with the other a − 1 alternatives in a MaxDiff set of size a:

Weighted reach has no 100% ceiling to aim for, and its level depends on a. The a − 1 competing alternatives always keep part of the probability, so even the full item set scores below 100%. For a respondent with flat (zero-centered) utilities the full set of n items scores exactly n / (n + a − 1): with 12 items and 5 per set that is 75%. Spread-out utilities push the value above that reference point, and it only approaches 100% for respondents whose preferences are very pronounced. Because a sits in the denominator, the same portfolio scores higher in a study with 3 items per set than in one with 5, so weighted reach levels are not comparable across studies with different set sizes. Compare portfolios within one study, or use a binary rule such as top-K when you need a head count. The TURF results show this reference value for the study at hand whenever Weighted by probability is selected.

For Anchored MaxDiff, the anchor is the comparison alternative. With respondent-specific threshold ti, use anchored utility zij − ti and denominator 1:

First Choice

Top-K

K is capped below the number of items, so an item set can never reach everyone by definition alone.

Top half (median) and above the mean

Calibration "Yes" answers (raw)

Utility above anchor (model-based)

12. The TURF Optimization Problem

Finding the best portfolio of size K is a combinatorial optimisation. For n items there are C(n, K) possible sets. Choicery handles this in two complementary ways.

Exhaustive search (the TURF Results table)

For the headline result, Choicery enumerates all combinations of the requested size, computes reach and frequency for each, and keeps the best ones. This is exact: it is guaranteed to find the true optimum for that size. To stay responsive it enforces two limits:

Combinations are ranked by reach first, then by frequency as a tie-breaker. The exhaustive search is why very large assortment sizes on many-item studies can hit the cap: C(40, 8) already exceeds 76 million.

Greedy ladder (the TURF Ladder table)

The ladder uses a greedy heuristic instead of exhaustive search. It is the classic algorithm for the maximum-coverage problem:

S = {}                      # empty portfolio
while |S| < ladder_size:
    pick the item j (not in S) that maximises reach(S ∪ {j})
    add j to S
    record reach(S) and the increase

Greedy is fast and intuitive, and it shows the order in which items earn their place. Because maximum coverage is submodular, the greedy solution is provably within a factor of 1 − 1/e ≈ 63% of the optimum for that size, and in practice usually much closer. It is therefore a good approximation, but not guaranteed optimal. When you need the exact best set of a fixed size, read the exhaustive TURF Results table; when you want to understand the build-up and the diminishing returns, read the ladder.

13. Shapley Contribution to Reach

The Shapley Contribution metric on the Item Rankings page answers a fairness question: "Across all the portfolios an item could belong to, how much reach does it bring on average?" It borrows the Shapley value from cooperative game theory, where players (items) form coalitions (portfolios) and we want each player's fair share of the total value created.

Choicery evaluates contribution with the same portfolio value function used by Weighted TURF. For standard MaxDiff, selected exponentiated utilities enter the portfolio numerator jointly; for Anchored MaxDiff, the anchor is the comparison alternative. The per-item contributions are averaged over respondents and normalised for display.

The exact Shapley value is computed whenever the study is small enough for the full marginal-contribution enumeration. Above that size Choicery switches to a deterministic permutation approximation with a fixed seed, which trades exactness for bounded runtime: the reported contributions then carry a small sampling error, so very close items can swap ranks. When sampling is used, the Shapley card on the Item Rankings page says so and names the number of permutations; the same information is recorded on the Settings sheet of the Item Rankings workbook ("Shapley Calculation" and "Shapley Permutations").

Interpretation: an item with high Shapley contribution pulls its weight in many different portfolios, a hallmark of a versatile, broadly useful item. An item can rank well on Relative Preference yet have a modest Shapley contribution if it mostly duplicates the coverage of more popular items.

14. The Anchored Calibration Model

Anchored MaxDiff converts the relative HB scale into absolute acceptance probabilities. Choicery estimates, per respondent, an anchor threshold θi on the utility scale such that an item is accepted when its utility exceeds the threshold, through a logistic link:

A low threshold means the respondent accepts most items; a high threshold means only their very top items pass. The Anchor Threshold card reports the mean, standard deviation, minimum, and maximum of θ across respondents, so you can see how selective your sample is and how much that selectiveness varies.

The model-based P(Accept) usually sits close to the raw calibration "Yes" rate but smooths individual noise, which is why the page shows both side by side. A large, systematic gap between them is a signal to inspect data quality or calibration-screen wording.

15. Known Limitations

• TURF inherits MaxDiff's assumptions. If the MaxDiff design or HB estimation is poor, the utilities are poor, and no assortment optimisation can rescue them. Check HB convergence first.
• The reach definition is a modelling choice, not a fact. Different definitions can name different "best" sets. Report which one you used and, ideally, show that the headline set is robust across a couple of definitions.
• No price or cost. TURF maximises reach, not profit. An item that adds reach may be expensive to offer; the page does not weigh that. Combine TURF with cost data outside Choicery for ROI decisions.
• Exhaustive search is capped. Very large item pools with large assortment sizes exceed the 250,000-combination limit; use a smaller size, exclude clearly weak items, or lean on the greedy ladder. Searches below that combination cap but above the interactive work budget can run as a bounded background job.
• Greedy is approximate. The ladder is near-optimal but not guaranteed optimal; trust the exhaustive TURF Results table for the exact best fixed-size set.
• Relative vs absolute. Standard MaxDiff cannot tell you whether anyone wants the items at all. If that question matters, run Anchored MaxDiff.

16. References