Identification of Preferences in Forced-Choice Conjoint Experiments: Reassessing the Quantity of Interest

Abstract

Forced-choice conjoint experiments have become a standard component of the experimental toolbox in political science and sociology. Yet the literature has largely overlooked the fact that conjoint experiments can be used for two distinct purposes: to uncover respondents’ multidimensional preferences, and to estimate the causal effects of some attributes on a profile’s selection probability in a multidimensional choice setting. This paper makes the argument that this distinction is both analytically and practically relevant, because the quantity of interest is contingent on the purpose of the study. The vast majority of social scientists relying on conjoint analyses, including most scholars interested in studying preferences, have adopted the average marginal component effect (AMCE) as their main quantity of interest. The paper shows that the AMCE is neither conceptually nor practically suited to explore respondents’ preferences. Not only is it essentially a causal quantity conceptually at odds with the goal of describing patterns of preferences, but it also does generally not identify preferences, mixing them with compositional effects unrelated to preferences. This paper proposes a novel estimand—the average component preference—designed to explore patterns of preferences, and it presents a method for estimating it.

Detecting Preference Cycles in Forced-Choice Conjoint Experiments

“…This is similar to the recommendation in Ganter (2021) for averaging over pairwise preferences to obtain a single summary measure. However, if some levels in an attribute are restricted while others are unrestricted, the randomization distribution across will differ across different AFCPs for the same attribute.…”

“…We diverge from Ganter (2021), primarily by emphasizing that the concept of preference should be understood as a relational concept between two features as opposed to an absolute ranking assigned to a single feature. In particular, we highlight that if a researcher wishes to make comparisons between ACPs -as they will typically do -they will still encounter all of the problems regarding indirect and direct preference comparisons that we describe in this paper.…”

“…We advance this literature by showing that even in the most credible of research designs -a randomized trial -additional, potentially strong, assumptions are required to make credible statements about political preferences. Specifically, we contribute to the literature in political methodology on the interpretation of results from conjoint experiments (Ganter 2021;Abramson, Kocak, and Magazinnik 2022). Most closely, we address the interpretation of the AMCE of Bansak et al (2021) as the average effect of manipulating a given feature on a candidate's vote-share.…”

New Experimental Evidence on Anti-Asian Bias in White Parents’ School Preferences

“…We use conditional logit models to analyze the role of schools’ racial composition in respondents’ school preferences. These models are well suited for a conjoint experimental design because they take into account the contrasting choice set of school profile characteristics for both the selected and not-selected schools (Ganter 2023; Hauber et al 2016). This strategy uses a utility model to estimate parent choices.…”

Multiple Hypothesis Testing in Conjoint Analysis

“…Compared to other studies that propose improvements on conjoint survey designs, this paper exclusively focuses on statistical inference. Existing studies have examined estimands and interpretation (Abramson et al 2020; Abramson, Koçak, and Magazinnik 2022; Bansak et al 2022; de la Cuesta et al 2022; Egami and Imai 2019; Ganter 2021), implementation (Bansak et al 2018; 2021b), social desirability bias (Horiuchi, Markovich, and Yamamoto 2020), and subgroup analysis (Clayton et al 2021; Leeper, Hobolt, and Tilley 2020). While this paper does not directly engage with any of these, the issue of multiple testing is relevant to any statistical inference with conjoint analysis due to its multiple comparison feature, unless the purpose of the analysis is exclusively exploration of higher-order interaction effects (Egami and Imai 2019).…”