Interaction Search

1. What Are Interaction Effects?

Interaction effects occur when a consumer’s preference for one attribute depends on the level of another attribute. In a standard conjoint model, we assume that preferences are additive, the utility contribution of each attribute level is independent of every other attribute. Interaction Search tests whether this assumption holds.

Standard conjoint models estimate main effects only: each attribute level gets a single utility weight that applies regardless of what levels appear on the other attributes. This additive structure keeps the model simple and interpretable. But sometimes, certain attribute combinations create value (or destroy it) beyond what the individual main effects predict.

In Choicery, Interaction Search runs a likelihood-ratio chi-square test on every pair of attributes to detect significant interactions. It compares a main-effects-only model against a model that includes cross-product terms for each attribute pair, one pair at a time.

2. Prerequisites

Before running Interaction Search in Choicery, make sure you have the following:

3. Running Interaction Search in Choicery

3.1 Creating a Run

Step 1 Estimation shows a run selector at the top (or an empty-state card reading No Interaction Search Run Yet with the participant count, if this task has no run yet), followed by the estimation card titled Interaction Search (2LL). Figure 1 shows it for the pizza oven CBC (cbc_pizza_oven, 690 respondents) with the default settings.

No.	Control	What it does
1	Run Name	Suggested and counted up per task: IS_CBC_ for CBC, IS_SCBC_ for CBC Shop, IS_Run_ for the other types. Up to 50 characters, unique within the project.
2	Confidence Level	0.90, 0.95 (default) or 0.99. Sets alpha for the significance decisions: 0.10, 0.05 or 0.01.
3	Practical Threshold (%)	Minimum gain in percent certainty, in percentage points, that a pair needs in addition to significance (default 1.0). Section 4.4 shows why this value matters.
4	Run	Starts the analysis (section 3.5).
5	Significance hint	States how results are flagged: raw p-value ≤ alpha and FDR q-value ≤ alpha, both together with Improvement ≥ Practical Threshold.
6	Variable Coding	Categorical or Linear per attribute (section 3.2). The Pizza runs keep every attribute categorical.
7	Advanced Settings	Solver settings of the MNL estimation (section 3.4).

Once a run has completed, its fields are shown read-only. Use New Run (in the run dropdown or the button on the right of the run bar) to unlock the form again; it keeps the settings of the run you were looking at. The trash icon next to a run deletes it. A task can hold at most 10 completed runs; delete an old run to create another.

3.2 Variable Coding

Below the run fields, the Variable Coding section lists every attribute of the task with a dropdown offering two options, Categorical and Linear. Choosing Linear reveals one numeric input per level, where you enter the raw value that level represents. Internally the values are centered and scaled by their standard deviation, so effects involving a linear attribute are reported per standardized unit (see the Units chip in the explorer). The run refuses to start if a linear attribute has an empty or non-numeric level value.

3.3 None Option

An Include None Option checkbox appears only for the task types where the no-choice answer is a separate follow-up question: DR-CBC always, and BW-CBC when the design has a none option. It is pre-checked in those cases. For CBC and CBC Shop the checkbox is not shown; if the design contains a none option, it is included automatically.

The tooltip on the checkbox describes what is included: for DR-CBC a “No” follow-up is treated as None, for BW-CBC a “No” answer after the Best choice is treated as None. Unchecking it estimates from the product-choice stage only.

For BW-CBC only the Best choices (plus the None follow-up when included) enter the model; Worst choices are not modeled by Interaction Search. The results view and the Managerial Summary sheet state this.

3.4 Advanced Settings (Collapsible)

No.	Field	What it does
1	Max Iterations	Optimization steps of the MNL solver (20 to 5000, default 250). The default is enough for most datasets; raise it if a model is reported as not converged.
2	Convergence Tolerance	When the optimizer treats the log-likelihood as stable (1e-12 to 0.01, default 1e-6).
3	Note	A modified 2LL test based on the Preference Analysis (HB) is planned but not available in this release.

3.5 Running the Analysis

Click the Run button to start the analysis. Choicery first estimates a baseline main-effects MNL model, then tests every attribute pair by adding its cross-product terms and comparing model fit.

• The estimation card is replaced by an Interaction Search Progress card with a progress bar and a status line that runs through “Preparing choice data…”, “Building design matrix…”, “Estimating baseline main-effects model…” and “Testing interaction 3/10” followed by the names of the pair, with a pair counter such as “Pair 3/10” underneath.
• You can Cancel the analysis at any time using the button in the progress card header.
• On completion the card reads Interaction Search Complete and the view switches to the Results step after about a second.
• If the baseline model does not converge, no pair can be flagged significant: every pair then shows “Not converged / not fitted” in the Optimization / inference column of the summary table, and the Excel Managerial Summary carries a warning. Raise Max Iterations and re-run.

4. Interpreting Results

4.1 The Metrics Bar

The results view (step 2 Results, headed Interaction Results) opens with a standing note that this is a pooled MNL screening: each pair is tested separately, preference heterogeneity can resemble interaction, and holdout data should decide the final model. Runs created before the current validation checks additionally show a warning asking you to re-estimate them. Below that comes a one-line metrics bar:

In the Pizza run the main-effects model reaches a percent certainty of 41.8 and an RLH of 0.446. Ten pairs were tested; the smallest p-value is below 0.001, yet neither count shows a candidate. The reason is the practical threshold, as section 4.4 explains.

4.2 Level Interaction Explorer

The first card below the metrics bar is the Level Interaction Explorer, subtitled “Inspect pair-level synergies by attribute levels”. Its controls are:

• Attribute Pair, a dropdown listing every tested pair. Significant pairs are marked with a “[sig]” suffix and sort to the top; a pair without estimable level effects shows a placeholder note instead of a table.
• Top Rows, with Top 8, Top 12 (default) and Top 20. This limit applies to the slope and raw-term tables only; the level-effects table always shows every level pair, and the control is hidden in that case.
• Raw-p candidates only (including practical threshold), a switch that restricts the dropdown to pairs flagged by the raw-p decision.
• A PNG button in the card header that exports the whole explorer card as an image.

Above the table, a short list shows the result Raw-p candidate, Raw-p criteria not met or Inference unavailable for the selected pair, followed by p, q, χ², the Pct Cert gain in percentage points, the optimization state (Converged or Not converged / not fitted), whether utility inference is available, and the units of the effects shown.

Categorical × Categorical

When both attributes are categorically coded, the explorer shows an Interaction Level Effects table in long format: one row per level combination with the columns attribute A level, attribute B level, Effect (β), Std Error, t Ratio and p-value. Positive effects are shown in green with a plus sign (the combination is valued more than the main effects alone would predict), negative effects in red. The effect is the interaction utility delta, that is the β of the interaction term, not an effect size such as Cohen's d. Reference-level cells implied by effects coding are included and receive standard errors and p-values whenever their contrast is identifiable. A dash marks cells whose inference is unavailable (non-identifiable contrast, or the covariance of the pair model could not be computed); the note below the table explains this.

Categorical × Linear

When one attribute is categorical and the other is linear, the explorer shows a Slope deltas by level table. Each level of the categorical attribute gets its own slope adjustment, showing how the linear effect changes for that level. Rows are sorted by absolute magnitude and limited by the Top Rows setting.

Linear × Linear

When both attributes are linear, the interaction is a single coefficient, the cross-product of the two linear terms, printed as one line.

4.3 Interaction Search Summary

Below the explorer, the Interaction Search Summary card lists every attribute pair. Its header repeats the settings used (“α = 0.05 · practical threshold 0.25 percentage points” in Figure 5) and carries a CSV button that downloads the table as displayed. Pair names are written as “Brand x Price”. The columns are:

Rows flagged by the raw-p decision get a tinted background, and p- or q-values at or below the configured alpha (0.05 for the default confidence level) are set in bold when inference is available.

With the practical threshold at 0.25, Maximum temperature x Controls is a raw-p and a BH-q candidate. Controls x Price is significant on both p (0.00965) and q (0.0482), but its improvement of 0.09 points stays below the threshold, so it is not flagged. Every other pair has p ≥ 0.109. Section 10 comes back to Controls x Price: it is a typical example of a signal that heterogeneity can produce.

4.4 Significance: Raw p and FDR q

Choicery reports significance two ways, side by side. Both share the same practical-significance requirement, and both additionally require that the baseline model and that pair’s full model converged, that the observed-information matrix of each fit is full rank (no quasi-complete separation), that the interaction model’s log-likelihood is not below the baseline, and that the pair has at least one estimable interaction contrast (effective DF > 0). Pairs failing any of these show “inference unavailable” and n/a instead of a decision:

Requiring a minimum effect size prevents a common trap: with large samples, even trivially small interactions become statistically significant.

4.5 Inestimable Pairs

Prohibited level combinations can leave some interaction cells empty. Their columns are then collinear with the main effects and carry no information. Choicery detects this from the rank of the design matrix:

• If all interaction columns of a pair are collinear, the pair is inestimable. It is reported with DF 0 and no test statistics (χ², p-value and q-value show n/a, and so do the Raw candidate and BH candidate columns), is never flagged significant, and is excluded from the Benjamini-Hochberg family so it cannot dilute the correction.
• If only some columns are collinear, the pair is partially estimable and the chi-square test uses the reduced effective degrees of freedom.

Both cases are visible in the DF column and the Optimization / inference column of the summary table, in the Managerial Summary sheet and the per-pair sheets of the Excel workbook (nominal versus effective degrees of freedom), and as inestimable_count and partially_estimable_count in the results JSON.

5. Exporting Results

Step 3 Export is headed Export Data and groups the downloads into two cards, Interaction Results and Run Data:

The Aggregate MNL model is not a separate analysis in Choicery. It is produced as the baseline of every Interaction Search run and lives in the Main Effects (Aggregate MNL) sheet of the workbook (utilities, log-likelihood, percent certainty, RLH). The downloads page states this explicitly.

Two more exports sit outside the Export step:

• Download PDF in the header of the Results step, which renders the interaction results as a PDF report.
• The CSV and PNG buttons inside the results cards, which export the summary table and the explorer card as displayed.

6. Best Practices & Common Pitfalls

Do:

• Treat Interaction Search as the screening step it is. It tells you which pairs would be worth modeling explicitly, rather than guessing. Note that Choicery’s in-app Preference Analysis (HB) estimates main effects only; there is no checkbox for adding attribute interaction terms. Acting on a finding therefore means changing the study design (for example moving to conditional pricing) or re-specifying the model outside Choicery.
• Use the default practical threshold (1.0%) to filter out trivially small effects. Adjust only if you have a specific reason.
• Check if “significant” interactions disappear when using HB with heterogeneity. Many aggregate interactions are actually heterogeneity artifacts (Orme & Williams, 2016: only 4 of 22 commercial datasets benefited from explicit interaction terms).
• Focus on substantively interpretable interactions. Brand × Price is common and meaningful. Color × Warranty might be statistically significant but hard to explain, be skeptical of such pairs.
• Use the Level Interaction Explorer to understand the direction of the effect, not just its existence.

Don’t:

• Don’t add ALL significant interactions to your model. Start with the strongest 1 or 2 and evaluate model improvement. Adding too many interaction terms increases model complexity and can lead to overfitting.
• Don’t panic if you find many significant interactions. With large samples, even trivial effects become “significant.” Always check the Improvement column.
• Don’t use Interaction Search with very small samples (<50 respondents). The aggregate MNL may not converge reliably, and low statistical power means you will miss genuine interactions.
• Don’t confuse statistical significance with practical importance. Always check the Improvement column alongside the p-value.
• Don’t forget: this test uses an aggregate MNL, not HB. Results show population-average interactions. Some of what appears as an interaction may be a heterogeneity artifact, a pattern that arises because different segments have different main effects, not because of a true within-person interaction.

7. Key Definitions

8. The Statistical Test

8.1 Aggregate MNL Baseline Model

Choicery first estimates a main-effects-only multinomial logit (MNL) model on the pooled (aggregate) choice data. This model assumes that all respondents share the same preference weights and that attribute effects are additive:

The model is estimated by maximizing the log-likelihood over all observed choices. This gives the baseline log-likelihood LL_main.

8.2 Full Model with Interaction Terms

For each attribute pair (A, B), Choicery estimates a second model that includes all main effects plus the cross-product terms for that specific pair:

This gives LL_full for each pair. Since the interaction model nests the main-effects model (setting all γ_pq = 0 recovers the baseline), the likelihood-ratio test applies.

8.3 Likelihood-Ratio Test

Under the null hypothesis (no interaction), this statistic follows a chi-square distribution with df degrees of freedom, where df equals the number of additional parameters in the interaction model.

8.4 Degrees of Freedom by Coding Type

The number of additional parameters (and hence the degrees of freedom for the chi-square test) depends on how each attribute is coded:

9. Multiple Comparison Correction

9.1 The Problem

When testing multiple attribute pairs simultaneously, the probability of at least one false positive increases rapidly. With k attributes, you test k(k − 1) / 2 pairs. At α = 0.05, the expected number of false positives is 5% × number of tests.

9.2 Bonferroni Correction

The simplest approach: divide the alpha threshold by the number of tests. For 28 tests at α = 0.05, the Bonferroni-adjusted threshold becomes 0.05 / 28 = 0.00179. This controls the family-wise error rate (the probability of any false positive), but is very conservative, it substantially reduces power to detect true interactions.

9.3 Benjamini-Hochberg (Used by Choicery)

Choicery uses the Benjamini-Hochberg (BH) procedure (Benjamini & Hochberg, 1995), which controls the false discovery rate (FDR) rather than the family-wise error rate. The FDR is the expected proportion of false positives among all discoveries (rejected nulls).

The procedure works as follows:

The resulting q-values (adjusted p-values) represent the minimum FDR at which each pair would be declared significant. A q-value < 0.05 means: if you call this pair significant, you can expect that at most 5% of all your “significant” calls are false discoveries.

In Choicery, m is the number of design-testable pairs (effective DF > 0). Inestimable pairs (see Section 4.5) are left out of the family, so they cannot inflate m and weaken the correction for the pairs that were tested. Pairs that are design-testable but whose fit failed (no likelihood-ratio inference) stay in the family and enter the correction with an internal p-value of 1, which is conservative for the other pairs; their own q-value is shown as n/a.

9.4 Worked Example

10. The Heterogeneity Caveat

This is perhaps the most important methodological consideration for Interaction Search, and one that is frequently overlooked.

Consider the following scenario:

The key finding from Orme and Williams (2016) is striking: in an analysis of 22 commercial conjoint datasets, only 4 datasets showed genuine interactions that improved HB model fit when explicit interaction terms were added. The remaining apparent interactions in aggregate models were adequately captured by HB’s individual-level heterogeneity.

Practical implications

• Use Interaction Search as a screening tool, not as definitive proof. If a pair is significant in the aggregate test, verify by examining HB individual-level utilities.
• If the interaction disappears when you look at HB utilities across segments (e.g., via Latent Class), it was likely a heterogeneity artifact.
• True interactions persist at the individual level. For example, in conditional pricing designs (where each brand has its own price range), a Brand × Price interaction is real because the price meaning genuinely differs by brand.

11. When to Add Interactions to Your Model

Given the heterogeneity caveat, when should you actually add interaction terms to your conjoint model? Here are practical guidelines:

Validation approach

If you decide to add interaction terms to your HB model, validate the decision with holdout tasks:

1. Estimate HB with main effects only. Record holdout hit rate.
2. Estimate HB with main effects + the interaction. Record holdout hit rate.
3. If the interaction model improves holdout prediction, the interaction is likely genuine and worth keeping.
4. If holdout prediction stays the same (or decreases), the interaction is likely a heterogeneity artifact that HB already handles through individual-level estimation.