List Experiment Design, Non-Strategic Respondent Error, and Item Count Technique Estimators

Abstract

The item count technique (ICT-MLE) regression model for survey list experiments depends on assumptions about responses at the extremes (choosing no or all items on the list). Existing list experiment best practices aim to minimize strategic misrepresentation in ways that virtually guarantee that a tiny number of respondents appear in the extrema. Under such conditions both the “no liars” identification assumption and the computational strategy used to estimate the ICT-MLE become difficult to sustain. I report the results of Monte Carlo experiments examining the sensitivity of the ICT-MLE and simple difference-in-means estimators to survey design choices and small amounts of non-strategic respondent error. I show that, compared to the difference in means, the performance of the ICT-MLE depends on list design. Both estimators are sensitive to measurement error, but the problems are more severe for the ICT-MLE as a direct consequence of the no liars assumption. These problems become extreme as the number of treatment-group respondents choosing all the items on the list decreases. I document that such problems can arise in real-world applications, provide guidance for applied work, and suggest directions for further research.

Opting Out of the Social Contract: Tax Morale and Evasion

“…Only then will tax morale across Latin America improve. Ahlquist, 2014Ahlquist, , 2018Blair et al, 2018). Ahlquist (2014Ahlquist ( , 2018 examines the estimator's sensitivity to various forms of non-strategic error and finds that the estimator's computational stability is quite fragile, especially when compared to the difference-in-means estimator.…”

Fuzzy Profiles

“…In our case, we specified the raw aggregate value of 1 as "fully out," 5 as "fully in," and 3 as "neither" since the scales on which these measures are based are tied to psychometric theory and nicely correspond to the aforementioned thresholds. 5 Once the three points were specified for each raw measure, the software performed the transformation of all variable values into fuzzy set scores based on the log odds of full membership (Ragin, 2006). Following the aforementioned prior research, we also added a small constant of 0.001 to all membership scores equal to exactly 0.50 to ensure these observations did not get dropped from the analyses for technical reasons.…”

List Experiments with Measurement Error

“…While the literature has since provided additional tools for alleviating strategic measurement error in list experiments (e.g., Blair, Imai, and Lyall 2014; Aronow et al. 2015), it has not yet addressed the consequences of nonstrategic measurement error, arising for example from “the usual problems of miscoding by administrators or enumerators as well as respondents misunderstanding or rushing through surveys” (Ahlquist 2018). Like floor and ceiling effects, these behaviors run against the assumptions of the standard maximum likelihood model () for list experiments (Blair and Imai 2012), and can induce severe model misspecification biases, especially when the underlying trait is rare (Ahlquist 2018).…”