Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments

Abstract

Respondent inattentiveness threatens to undermine experimental studies. In response, researchers incorporate measures of attentiveness into their analyses, yet often in a way that risks introducing posttreatment bias. We propose a design-based technique-mock vignettes (MVs)-to overcome these interrelated challenges. MVs feature content substantively similar to that of experimental vignettes in political science, and are followed by factual questions (mock vignette checks [MVCs]) that gauge respondents' attentiveness to the MV. Crucially, the same MV is viewed by all respondents prior to the experiment. Across five separate studies, we find that MVC performance is significantly associated with (1) stronger treatment effects, and 2) other common measures of attentiveness. Researchers can therefore use MVC performance to re-estimate treatment effects, allowing for hypothesis tests that are more robust to respondent inattentiveness and yet also safeguarded against post-treatment bias. Lastly, our study offers researchers a set of empirically-validated MVs for their own experiments.

The Domestic Sources of International Reputation

“…21 Subjects were asked to identify the issue in dispute in our pretreatment vignette and presented with multiple choice options. This is similar to the approach recommended by Kane, Velez, and Barabas (2020). .…”

The Generalizability of Online Experiments Conducted During the COVID-19 Pandemic

“…5 The "User Agent" string captured from the end user's browser by online survey software like Qualtrics provides detailed information about how respondents arrive at the survey. We offer two approaches for classifying respondents as inattentive on the basis of 4 Kane, Velez and Barabas (2020) suggest including "mock vignettes" which can be viewed as a taskspecific ACQ. Berinsky, Margolis and Sances (2014); Berinsky et al (2019) urge researchers to use multiple ACQs and classify subjects based on different levels of attentiveness.…”

“… 4 Kane, Velez and Barabas (2020) suggest including “mock vignettes” which can be viewed as a task-specific ACQ. Berinsky, Margolis and Sances (2014); Berinsky et al (2019) urge researchers to use multiple ACQs and classify subjects based on different levels of attentiveness. …”

Is “Constitutional Veneration” an Obstacle to Constitutional Amendment?

“…To address this difficulty, we ran a follow-up experiment in December 2020 on the survey respondent aggregator Lucid. 10 In total, we recruited 470 participants, but we restrict our analysis to the 336 respondents that successfully completed a Mock Vignette Check (Kane, Velez and Barabas, 2020). 11…”

“… 11 As Peyton, Huber and Coppock (2020) show, inattention was a major problem among Lucid respondents during the COVID-19 pandemic. The Mock Vignette Check came from Kane, Velez and Barabas (2020) and was placed in the survey immediately before the treatment. We describe the MVC, as well as other attention checks we used to screen subjects, in the “Lucid Attention Checks” section of the Appendix. …”