(Mis)Measuring Sensitive Attitudes with the List Experiment

Abstract

List experiments (LEs) are an increasingly popular survey research tool for measuring sensitive attitudes and behaviors. However, there is evidence that list experiments sometimes produce unreasonable estimates. Why do list experiments “fail,” and how can the performance of the list experiment be improved? Using evidence from Kenya, we hypothesize that the length and complexity of the LE format make them costlier for respondents to complete and thus prone to comprehension and reporting errors. First, we show that list experiments encounter difficulties with simple, nonsensitive lists about food consumption and daily activities: over 40 percent of respondents provide inconsistent responses between list experiment and direct question formats. These errors are concentrated among less numerate and less educated respondents, offering evidence that the errors are driven by the complexity and difficulty of list experiments. Second, we examine list experiments measuring attitudes about political violence. The standard list experiment reveals lower rates of support for political violence compared to simply asking directly about this sensitive attitude, which we interpret as list experiment breakdown. We evaluate two modifications to the list experiment designed to reduce its complexity: private tabulation and cartoon visual aids. Both modifications greatly enhance list experiment performance, especially among respondent subgroups where the standard procedure is most problematic. The paper makes two key contributions: (1) showing that techniques such as the list experiment, which have promise for reducing response bias, can introduce different forms of error associated with question complexity and difficulty; and (2) demonstrating the effectiveness of easy-to-implement solutions to the problem.

The Effect of a Wordless, Animated, Social Media Video Intervention on COVID-19 Prevention: Online Randomized Controlled Trial

“…Rinse my nose with salt water daily.” The treatment group received the same 5 items and 1 additional “sensitive” item, “Go out with my friends,” which indicates behavioral intent to social distance (or not) during lockdown restrictions. We used the list experiments to reduce social desirability bias [ 26 , 27 ] and designed them in line with best practices [ 28 ].…”

Dominance-Driven Autocratic Political Orientations Predict Political Violence in Western, Educated, Industrialized, Rich, and Democratic (WEIRD) and Non-WEIRD Samples

“…Following conventions (Glynn, 2013; Kramon & Weghorst, 2019), we first estimated the proportion of interviewees who agreed with the treatment item for three subsamples. Specifically, we split the sample in terciles on the basis of their scores on the AOS-D, resulting in subsamples with a relatively “democratic orientation” ( n = 329), “moderate orientation” ( n = 327), and “autocratic orientation” ( n = 356).…”

“…As mentioned, the standard procedure for analyzing data from a double-list experiment yields an estimate of the proportion of interviewees in a sample who agree with the treatment item, not estimates of agreement for individual interviewees. Hence, research using list experiments has typically examined associations with individual-difference variables by splitting in subsamples (e.g., subsamples based on education; Kramon & Weghorst, 2019). Following this, we examined the association with autocratic orientation by computing estimates of agreement for three subsamples (high, average, and low scores for autocratic orientation) and then comparing these estimates.…”

An entertainment-education approach to prevent COVID-19 spread: study protocol for a multi-site randomized controlled trial

“…Second, we will assess if the CoVideo improves behavioral intent toward preventive COVID-19 behaviors. Since many participants will be primed to give socially desirable answers to the behavioral intent questions, we will use a list randomization approach [ 15 – 17 ]. There will be five list experiments, which are shown in Table 2 .…”