A Text-As-Data Approach for Using Open-Ended Responses as Manipulation Checks

Abstract

Participants that complete online surveys and experiments may be inattentive, which can hinder researchers’ ability to draw substantive or causal inferences. As such, many practitioners include multiple factual or instructional closed-ended manipulation checks to identify low-attention respondents. However, closed-ended manipulation checks are either correct or incorrect, which allows participants to more easily guess and it reduces the potential variation in attention between respondents. In response to these shortcomings, I develop an automatic and standardized methodology to measure attention that relies on the text that respondents provide in an open-ended manipulation check. There are multiple benefits to this approach. First, it provides a continuous measure of attention, which allows for greater variation between respondents. Second, it reduces the reliance on subjective, paid humans to analyze open-ended responses. Last, I outline how to diagnose the impact of inattentive workers on the overall results, including how to assess the average treatment effect of those respondents that likely received the treatment. I provide easy-to-use software in R to implement these suggestions for open-ended manipulation checks.

For living well, behaviors and circumstances matter just as much as psychological traits

“…Texts in online samples could be shorter than ones collected through phone interviews or through the collection of hand-written text. There may also be other quality concerns as responses to open-ended surveys are sometimes used to identify “low-quality” or inattentive respondents ( 52 ), although these issues may be more transparent in analyses of open-ended responses than closed-ended ones.…”

Cause‐related marketing in pandemic context—The effects of cause‐brand fit and cause‐brand alliance on customer‐based legitimacy and reputation

“…Unfinished, incomplete cases were not taken into consideration. We also relied on the response quality in open‐ended questions by retaining the relevant responses and excluding invalid/inaccurate responses as using this type of elimination provides greater accuracy than attention checks (Ziegler, 2021).…”

Political responsiveness and centralized religious leaders: lessons from the Catholic Church

“…related to [insert most salient issue from previous question]" to capture if participants have a disagreement with the Church regarding their most concerning issue. After viewing the news headlines, respondents answer an open-ended manipulation check to determine how attentive they are to the experimental task (Ziegler, 2022). Then, participants express the degree to which they think the Church is responsive, the degree to which they trust the Church and the degree to which they anticipate increasing their organizational as well as political participation.…”