Anchored MaxDiff: Best-Worst Scaling with Calibration

1. What Is Anchored MaxDiff?

Anchored MaxDiff is ordinary MaxDiff (best-worst scaling) with one extra step. Respondents first work through best-worst tasks: each task shows a few items and asks for the best and the worst. After the last task, one calibration screen lists every item with a Yes/No question, for example "Would you consider actually choosing this topping for your pizza?". These answers place a fixed reference point, the anchor, on the same scale as the items.

Plain MaxDiff gives the rank order and the gaps between items, but no zero of "acceptable": the best item wins even if nobody wants any of them. The anchor fixes that zero, so you can read whether an item lies above the line (wanted) or below it.

The best-worst part is identical to plain MaxDiff. The MaxDiff guide covers the same editor; this guide repeats everything you need and adds the calibration.

What you get out of it

• Individual item utilities for every respondent, as in plain MaxDiff.
• An individual anchor threshold, estimated on the same scale.
• P(Accept) per item: the share of respondents who would accept it, and consideration sets per respondent.
• The usual views: preference ranking, segmentation and TURF reach.

2. When to Use It (and When Not)

Use Anchored MaxDiff when a rank order is not enough and you need to know which items clear a line of real acceptance. If you only need to prioritise, plain MaxDiff is shorter for respondents and fully sufficient.

3. Opening the Editor

The editor is the Anchored MaxDiff Configuration page. You reach it from the Survey Flow:

The left rail

• Setup: General (opens first, including the Calibration Settings), Attributes (the item list), Layout.
• Templates: Templates.
• Output: Generate, Preview.
• Help: Anchored MaxDiff Guide opens this guide in a new tab.

Required before fielding: the item list, the calibration wording (pre-filled, but you should adapt it) and a generated design. Layout and Templates are optional.

What saves automatically and what needs Save changes

The two locks

• Item list locked once a design exists: "Attributes are locked. Delete the generated design before editing attributes. Existing attributes and settings will be kept." with the button Delete generated design to edit attributes (Figure 2).
• Whole design locked once responses exist: the banner "Design locked - responses already collected" names the active and trashed responses and says: "To modify the design, open Data → Bin and permanently delete all responses first. To preserve them, duplicate the project before making changes. Preview, downloads, and design diagnostics remain available while the design is locked." All editing controls, including the calibration texts, are then disabled.

If the item list and the design no longer match, a Design Warning at the top says so (for example "Design is missing N item(s) from your current configuration. Please regenerate the design.").

4. The Editor Step by Step

4.1 General and Calibration Settings

No.	Control	What it does
1	Task Name *	Required. Shown in the survey flow, reports and exports (export columns start with anchored_maxdiff_ plus this name; the calibration answers get the suffix _cal). Up to 50 characters, starting with a letter, then letters, numbers, underscores or hyphens. Pizza: anchored_maxdiff.
2	Question / Instruction	The text above every best-worst task (rich text, up to 10,000 characters and 10 images). Default "Please select the best and worst item."
3	Footer	Optional text below each task. It also appears below the calibration screen.
4	Best Label	"Label for the "best" option (left column)". Default "Best"; Pizza "Most preferred". Required, up to 1,000 characters, no images.
5	Worst Label	"Label for the "worst" option (right column)". Default "Worst"; Pizza "Least preferred". Same rules.
6	Item Label	"Header text for the item column". Default "Item"; Pizza "Toppings". Also the header of the calibration list.
7	Calibration Settings	The block that makes the study anchored. Always visible on General.
8	Calibration Question	"Question shown on the calibration screen after best-worst tasks". Default for a new block: "Would you consider purchasing or using this item?"; Pizza: "Would you consider actually choosing this topping for your pizza?". Rich text, up to 10,000 characters and 10 images. If you clear it, the previously saved question is kept.
9	Yes Label	"Label for the affirmative response". Default "Yes". Required: an empty label blocks the save ("Anchor yes label is required."). Up to 1,000 characters, no images.
10	No Label	"Label for the negative response". Default "No". Required ("Anchor no label is required.").

4.2 Attributes: the item list

The step is labelled Attributes and the list "Attributes *", but each row is one item. The same list appears on the calibration screen.

No.	Control	What it does
1	Attributes are locked.	Shown once a design exists. Delete generated design to edit attributes deletes the design after a confirmation; items and settings stay. Not possible while responses exist.
2	Attributes *	The item list ("Enter attributes one per row. Press Enter for the next attribute; use Shift+Enter for a line break."). T opens formatting, × removes a row. Right of the label: autosave status and the TXT import.

Below the list: Add Attribute and the counter "N attributes".

4.3 Layout

No.	Control	What it does
1	Response Column Position	Left & Right (default), Right Side, Left Side or Rows (horizontal: items as column headers, a best row and a worst row below; Item Label not used).
2	Column Order	Best label first (default) or Worst label first, left or top.
3	Item Alignment	Left (default), Center or Right.
4	Show Item Numbers	Numbers the items in the task. Off by default.
5	Alternating Row Colors	Shades every second row. Off by default; on in Pizza.
6	Color Style	Colored (default, best green, worst red) or Neutral (black labels, standard radio buttons).
7	Row Height	Compact, Normal (default) or Spacious.
8	Table Width	Auto (default) or Custom with Custom Width (%) 30 to 100 in steps of 5 (default 80).
9	Preview	Live preview of the task with the first four items.

Layout applies to the best-worst tasks. The calibration screen has its own fixed layout (section 6). Click Save changes after editing.

4.4 Templates

No.	Control	What it does
1	Save as Template	Template name (required), Description (optional), Save (needs at least one item).
2	Available Templates	Saved templates with Apply, download as .txt, rename and delete; the button reloads the list.

A template stores the whole block configuration: items, question, footer, labels, layout and the Calibration Question, Yes Label and No Label, but not the design. Templates belong to your account. Apply is blocked while a design exists ("Delete the generated design before applying a template.") and replaces the current configuration after confirmation.

4.5 Generate

Header "Generate Design", subtitle "Create a balanced experimental design for your Anchored MaxDiff study". The calibration needs no design of its own: it always lists every item of the design once.

No.	Control	What it does
1	Versions	Number of different task sets; respondents are assigned in rotation. Default 200, allowed 1 to 300. Disabled for Legacy Simple.
2	Tasks	Best-worst tasks per respondent, before the calibration screen. Default 8 (Free plan 6), allowed 1 to 30.
3	Items per Task	Default 4, allowed 3 to 8, and at most the number of items minus 1. Free plan at most 4.
4	Method	Balanced (default): many versions, optimised item, pair, per-version and position balance. Legacy Simple: one fixed task set for everyone.

The readiness box above the form explains a disabled Generate Design button: Attributes required. (too few items for the chosen Items per Task), Design parameters required. (Items per Task outside 3 to 8), or the warning Limited task-set variation. when each task omits only one item. Generation saves pending item edits, needs a saved General step, asks "Replace existing design?" before replacing a design (the old one is backed up), is refused while responses exist, and can be cancelled from the progress panel.

Balanced makes item counts equal when versions × tasks × items per task is divisible by the number of items (otherwise they differ by at most one), then balances pairs, versions and screen positions. Legacy Simple builds one task set by rotating the least-used items. Details are in the MaxDiff guide.

No.	Control	What it does
1	Show Design Diagnostics	Opens the inline diagnostics summary (section 4.6).
2	Delete design	Deletes the design after "Delete generated design?". Disabled while responses exist.

The status bar icons are Download Design CSV, Download Diagnostics (text report) and Full Diagnostics.

4.6 Design Diagnostics

The tiles mean the same as in plain MaxDiff:

The full diagnostics page adds Per-Version Exposure, Positional Balance, the pair matrix, Design Information and Recommendations, plus the section Anchor Calibration: Method "Full-pool binary", Calibration Items, BW Design Items, Item Pool Match, Prompts / Version, Order (Randomized), and a table per item with In BW, In Calibration, BW Exposures, Anchor Prompts and Anchor/BW. Its caption: "Anchored MaxDiff uses the generated best-worst design, then shows one binary calibration prompt for each configured item. The anchor threshold is estimated after HB analysis from these Yes/No responses." Warnings appear for an empty calibration pool, an item pool that does not match the design, or a blank question or label.

4.7 Preview

No.	Control	What it does
1	Participant	Picks the design version. The preview steps through all tasks and then a final Calibration step, ending with "Preview complete - All MaxDiff and calibration steps were completed." Answers are not saved.

Without a design, the Example Preview shows one task built from the first items and no calibration screen.

5. How Large Should Your Study Be?

The best-worst part follows the same rules as plain MaxDiff. The calibration adds one Yes/No per item, so a long list also makes the calibration screen long.

Many items

With at most 30 tasks and 8 items per task, about 3 appearances per item are possible for up to 80 items (8 per task), 50 items (5 per task) or 40 items (4 per task). For longer lists Orme (2020, ch. 12.7) describes Sparse MaxDiff (each item about once, about 1,000 respondents, aggregate analysis) and the adaptive Bandit MaxDiff (80 to 500 items). Choicery has no adaptive design; a sparse design can be built with Balanced, but individual scores from it lean heavily on the HB population model. In Anchored MaxDiff a long list also means a long calibration screen with one row per item.

Sample size

The literature reviewed gives no sample-size rule for MaxDiff; the question is not settled. As orientation: Schramm and Lichters (2024) used 448 respondents in a 2 × 2 experiment, Schramm et al. (2025) 1,468. Plan each reported subgroup as its own sample.

Anchoring methods and their evidence

• Schramm and Lichters (2024), a preregistered 2 × 2 experiment (n = 448, 16 games, 16 sets of 4): incentive alignment raised predictive validity whichever anchor was used. Hit rates over three validation tasks: direct anchor 63.7 percent (incentive-aligned) vs 46.1 percent (hypothetical); indirect anchor 66.4 vs 50.5 percent. Direct and indirect anchoring had comparable predictive validity. Hypothetical MaxDiff overestimated demand. The indirect anchor gave a much higher no-buy probability, partly because most answers were the middle option "some"; the direct anchor predicted larger consideration sets (Table 1, p. 9; pp. 10 to 12).
• Schramm, Lichters and Guhl (2025), a preregistered, incentive-aligned study (n = 1,468, 16 chocolates, 16 sets of 4). It is an SSRN preprint, not peer reviewed. The status-quo anchor predicted free and forced product choice best; the indirect anchor predicted consideration-set size and dual-response tasks best; the limit card was best for rankings but worst for free choice; the direct anchor needed the fewest tasks to reach its best prediction. "None of the anchor approaches is an 'all-rounder'." With the direct anchor, show all items, not a subset. For MaxDiff sections longer than about 2 minutes they suggest considering status-quo or indirect anchoring (pp. 2, 20 to 21, 34 to 40).
• Orme (2020, ch. 12.6) questions whether any anchor is absolute and comparable across people.

6. What the Respondent Sees

First the best-worst tasks, exactly as in plain MaxDiff (Figure 8): question, "Task N of 8", the table, footer, Next. Back is disabled within the tasks. The respondent marks one item in each column; missing or identical picks are reported with the same messages as in MaxDiff, using your labels.

After the last task of the block comes the calibration screen:

• The Calibration Question at the top.
• Every item of the block's design once, in a random order fixed per participant (seeded by the participant number).
• For each item two options: the Yes Label and the No Label. The list header shows the Item Label.
• Every row must be answered: "Please answer all items. {count} item(s) remaining."; missing rows are highlighted.
• The block footer below the list; no Back button.

A Yes places an item above the respondent's line, a No below it. The best-worst choices set the spacing of the items; the calibration answers pin the scale to a fixed point. Both are stored with the response and enter estimation together.

7. Estimating Utilities with Hierarchical Bayes

Open the analysis, select the Anchored MaxDiff task and use MaxDiff Preference Estimation (HB) on the Preference Analysis (HB) tab. It is the MaxDiff HB engine; the calibration answers enter the likelihood as extra observations, so the anchor threshold is estimated on the same scale as the items.

• Run Name, Burn-in and Draws/Chain (5,000 each by default).
• In-App or Create Package. The package contains the calibration answers (data/anchor_calibration.csv) and estimates the threshold jointly, like the in-app run. When you import results, include Anchor_Thresholds.csv; without it Choicery falls back to a post-hoc maximum-likelihood threshold per respondent.
• Covariates (optional; the panel states the in-app limit, "In-app runs allow up to 10 encoded covariate columns" in Figure 10). Anchored runs also report a covariate effect on the threshold.
• Advanced MCMC Settings: Seed (42), Chains (2), Parallel, Variance Adj. (4.0), Prior σ² (1.0), Individual R-hat, random holdout task, validation split.

The full description of every setting is in the Preference Analysis (HB) guide.

8. Interpreting Anchored Results

Convergence and calibration coverage first

The meta bar shows Fitted RLH (Posterior Draw Average), R-hat (max), ESS (min) and the Acceptance Rate as in plain MaxDiff, plus two anchored counters. R-hat must be below 1.1 (below 1.05 is good); the Pizza demo run (1.211, ESS 6, only 5,000 draws per chain) is not there yet. Its worst value belongs to a variance term (Sigma); such terms mix slowly and need many more draws, as the MaxDiff guide explains. RLH chance is 1 / items per task (0.25 with 4 items).

• Calibration: respondents whose calibration answers could be matched to their best-worst data. If it is below N, Choicery warns you to check response and design consistency.
• Thresholds: respondents with an individual anchor threshold. In an in-app run it equals Calibration; a lower count only appears when an imported run had to fall back to post-hoc anchoring, and a note under the meta bar says so.

Respondents who answered Yes to all or No to all

A respondent who gave the same answer to every calibration item does not pin the threshold down; in the joint HB fit the threshold is shrunk toward the population. Such respondents are usually removed in research practice, so Choicery leaves them out by default: the Filter tab has the switch Exclude participants who answered Yes to all or No to all anchor questions (only in projects with an Anchored MaxDiff block), on unless you turn it off. The rule is checked per block: a respondent is left out of a block when at least two anchor questions of that block were answered and all answers are Yes, or all are No. The Filter tab shows the count, for example "8 excluded: 3 all Yes, 5 all No"; the MaxDiff Results Excel lists the switch and the count in the Settings Summary sheet.

The switch only affects analyses; exports and the Data Editor keep every respondent. With the switch on, P(Accept), consideration sets and TURF reach come from the remaining respondents. Someone who rejects the whole category is then not counted. If those people matter, turn the switch off; their thresholds are then estimated as prior-shrunk and marked in the Identification Status column of maxdiff_Anchor_Thresholds.csv.

Reading preference and acceptance together

The Relative view shows two columns. Preference is the ordinary MaxDiff share (sums to 100 percent). P(Accept) is the share of respondents whose utility for the item lies above their threshold; it is independent per item and does not sum to 100 percent. The Index view rescales Preference so the average item is 100.

The two readings can disagree, which is the point of anchoring. In Figure 11, Pepperoni ranks second on Preference (8.0 percent) ahead of Mushrooms (7.2 percent), but fewer respondents accept it (P(Accept) 59.1 vs 62.9 percent). Salami sits almost exactly on the line (50.3 percent); a small change in wording or sample could move it either way. Items far from 50 percent are safe calls.

The Anchor Analysis page

The MaxDiff Simulation (header navigation) has a page Anchor Analysis for anchored runs:

• Purchase Probability by Item: P(Accept) next to the raw calibration Yes rate ("Cal. Yes"). They can differ because the model smooths individual noise.
• Anchor Threshold: mean, standard deviation, minimum and maximum of the individual thresholds.
• Consideration Sets: per respondent the items with P(Accept) above 50 percent, and the distribution of set sizes.

Downloads

On the Export step, MaxDiff Results (Excel) carries four extra sheets: Anchor Summary, Purchase Probability, Anchored Utilities (utility minus the respondent's threshold) and Consideration Sets; the anchor figures are also added to the Settings Summary. Anchored MaxDiff Responses (Long Format) is a ZIP with three files that the download card calls data.csv, attributes.csv and calibration.csv; the actual file names carry a prefix with label, task and design code (for example <label>_<task>_<code>_data.csv). The ZIP can be imported again.

Next step: MaxDiff Simulation

For anchored runs the TURF reach can use Calibration "Yes" answers (raw) or Utility above anchor (model-based). See the MaxDiff Simulation guide.

9. Best Practices & Pitfalls

Do

• Add the anchor only when a go or no-go decision depends on it.
• Word the calibration question as a clear, absolute Yes/No that matches the decision; keep it identical across waves.
• Show 4 to 5 items per task and plan about 3 × K / k tasks.
• Walk through the whole preview, including the calibration step.
• Check R-hat, RLH and the Calibration and Thresholds counters before reporting.
• Report items near 50 percent P(Accept) as borderline.
• Include Anchor_Thresholds.csv when you import a replication package.

Do not

• Do not phrase the calibration question as a comparison.
• Do not leave the default question unchanged when it does not fit your items ("purchasing or using this item" rarely fits a topping list).
• Do not compare anchored scores across studies unless the calibration question was identical.
• Do not drop the relative ranking from the report; the two readings answer different questions.
• Do not let the list grow without need: every item adds a calibration row.

10. Key Definitions

11. Best-Worst Scaling & the MaxDiff Model

Best-worst scaling (Finn & Louviere, 1992; Louviere & Woodworth, 1983) rests on random utility theory (McFadden, 1974). Each item i has a latent utility for respondent h; the respondent picks the highest-utility item as best and the lowest as worst. Marley and Louviere (2005) formalised the probabilistic models. Choicery uses the sequential best-worst logit: the best pick is a multinomial logit over the items in the set, and the worst pick is the same logit on the negated utilities:

This is the multinomial-logit likelihood of the whole CBC family; only the construction of the observations differs. Because respondents choose rather than rate, the scores avoid scale-use bias (Cohen & Neira, 2003; Chrzan & Orme, 2019).

12. The Relative-Scale Problem & Anchoring Methods

MaxDiff only asks people to compare items with one another. Comparisons fix the differences between items but leave the origin free: you can shift every item up or down together without changing a single best-worst probability. Anchoring adds one piece of absolute information to fix the origin.

Indirect (dual-response) anchoring

Proposed by Jordan Louviere and described by Orme (2009), it adds a follow-up to each set: are all, some or none of these items important (or would you buy them)? A "none" says every shown item lies below the anchor; a "some" says the best is above and the worst below. Orme (2020, glossary) advises not more than four items per set with this method.

Direct binary anchoring

Lattery (2010) proposed asking directly, item by item, whether each item is above the threshold, as Yes/No after all sets. It needs one extra screen. This is the method Choicery implements, with all items of the design on the screen, as Schramm et al. (2025, p. 39) recommend for the direct anchor.

Status-quo and limit-card anchors

A status-quo anchor adds a "would not buy any" alternative to each set. Schramm et al. (2025) also tested a new limit-card anchor. They compared both with the direct and indirect anchors (see section 5). Choicery offers neither.

Calibration-question anchoring

Rating-scale calibration questions are also cited as anchors (Orme, 2020, ch. 12.5, citing Bacon et al., 2007; Orme, 2009; Lattery, 2010; Horne et al., 2012), but not evaluated there. Choicery uses the binary screen.

13. From Anchor to Absolute Scores

The anchor is an extra threshold parameter per respondent, estimated on the same utility scale as the items, not an extra item in the sets. The best-worst choices follow the likelihood of section 11; each Yes/No answer is a comparison between the item and the threshold a:

Because both kinds of answers are fitted jointly, a is in the same units as the item utilities. Shifting by the anchor gives the anchored score:

The shift keeps every difference the best-worst data produced and adds a meaningful zero. The same difference gives P(Accept) = sigmoid(λ_i − a), and the consideration set is the items with P(Accept) above 0.5.

Two caveats. First, a respondent with all Yes or all No gives no information about their own line; the joint HB fit still estimates the threshold, shrunk toward the population, and records it as prior-shrunk. Second, the joint fit needs the calibration answers at estimation time. In the app and in the replication package the threshold is sampled inside the HB run. Only when results arrive without Anchor_Thresholds.csv, or for older cached runs, does Choicery estimate each threshold afterwards by maximum likelihood from the calibration answers and the fitted utilities.

14. Experimental Design

The best-worst part uses the design of plain MaxDiff: items about equally often, pairs about equally often together, items spread over the screen positions, and many versions so no respondent carries the whole design. Choicery's Balanced generator makes item counts exactly equal when versions × tasks × items per task is divisible by the number of items and otherwise lets them differ by at most one; it then balances pairs, versions and positions (see the MaxDiff guide). The Design Diagnostics report the achieved values.

The calibration adds no combinatorial design: one screen with every item of the design once, in a random order per participant. Its only design decision is the wording of the Yes/No question. The number of calibration rows equals the number of items, one more reason to keep the list focused.

15. References