Detecting Preference Cycles in Forced-Choice Conjoint Experiments

Abstract

In this paper we describe implications of an under-explored theoretical property of the Average Marginal Component Effect (AMCE) estimand for conjoint experiments: its violation of independence. We show how this results from the AMCE's incorporation of information about irrelevant attributes by averaging over both direct and indirect comparisons of features. In doing so, the AMCE imposes an artificial transitivity and can produce positive AMCEs even when respondents are less likely to choose a profile with one feature over the baseline in direct comparisons. We introduce an alternative estimand, the Average Feature Choice Probability (AFCP), which only considers direct comparisons and corresponds to the frequency an attribute is chosen in paired comparisons. We decompose the AMCE into a weighted average of AFCPs and we describe the necessary conditions under which they correspond. Finally, we develop a statistical test for the presence of preference intransitivities and illustrate our method with the reanalysis of a number of conjoint experiments.

Multiple Hypothesis Testing in Conjoint Analysis

“…Compared to other studies that propose improvements on conjoint survey designs, this paper exclusively focuses on statistical inference. Existing studies have examined estimands and interpretation (Abramson et al 2020;Abramson, Koçak, and Magazinnik 2022;Bansak et al 2022;de la Cuesta et al 2022;Egami and Imai 2019;Ganter 2021), implementation (Bansak et al 2018;2021b), social desirability bias (Horiuchi, Markovich, and Yamamoto 2020), and subgroup analysis (Clayton et al 2021;Leeper, Hobolt, and Tilley 2020). While this paper does not directly engage with any of these, the issue of multiple testing is relevant to any statistical inference with conjoint analysis due to its multiple comparison feature, unless the purpose of the analysis is exclusively exploration of higher-order interaction effects (Egami and Imai 2019).…”