Are Manipulation Checks Necessary?

Abstract

Researchers are concerned about whether manipulations have the intended effects. Many journals and reviewers view manipulation checks favorably, and they are widely reported in prestigious journals. However, the prototypical manipulation check is a verbal (rather than behavioral) measure that always appears at the same point in the procedure (rather than its order being varied to assess order effects). Embedding such manipulation checks within an experiment comes with problems. While we conceptualize manipulation checks as measures, they can also act as interventions which initiate new processes that would otherwise not occur. The default assumption that manipulation checks do not affect experimental conclusions is unwarranted. They may amplify, undo, or interact with the effects of a manipulation. Further, the use of manipulation checks in mediational analyses does not rule out confounding variables, as any unmeasured variables that correlate with the manipulation check may still drive the relationship. Alternatives such as non-verbal and behavioral measures as manipulation checks and pilot testing are less problematic. Reviewers should view manipulation checks more critically, and authors should explore alternative methods to ensure the effectiveness of manipulations.

Potential negative consequences of mindfulness in the moral domain

“…This procedure is well established and allows fully controlled laboratory studies (e.g., Hafenbrack et al, 2014;Kiken & Shook, 2011;Long & Christian, 2015;Schofield, Creswell, & Denson, 2015). Thus, it can be concluded that brief mindfulness exercises can increase state mindfulness compared to various control groups (for an overview of manipulation check problems, see Hauser, Ellsworth, & Gonzalez, 2018). Notably, in contrast to the four remaining studies, in this study, we included a no-treatment control condition.…”

“…However, Lueke and Gibson (2016) found that state mindfulness was increased after a 10-min mindfulness exercise (become aware of sensations in the body, such as breathing) compared to a pure control condition (listening to an audiotape describing an English countryside), as well as to an attention condition (paying attention to the word "parish" while listening to the audiotape of the pure control condition). Thus, it can be concluded that brief mindfulness exercises can increase state mindfulness compared to various control groups (for an overview of manipulation check problems, see Hauser, Ellsworth, & Gonzalez, 2018). Given that mindfulness intervention exercises can at first feel challenging for inexperienced people (Creswell, 2017), future research should apply extended mindfulness trainings.…”

Subjective semantic surprise resulting from divided attention biases evaluations of an idea’s creativity

“…They also found that only those participants that passed the IMC were affected by the treatment. Because explicit manipulation checks can amplify, undo, or interact with the effects of a treatment, recent research suggests non-intrusive behavioral checks of attention 44 . We follow this method in the current study.…”

“Rarely safe to assume”: Evaluating the use and interpretation of manipulation checks in experimental social psychology

“…In short, essentially the same methodological and statistical considerations that apply to the validation of measurements also apply to the validation of manipulations. However, although there is a long tradition of thorough validation research and theory for measurements (Cronbach & Meehl, 1955;Loevinger, 1957;Nunnally, 1970;Raykov & Marcoulides, 2011), the empirical validation of experimental treatments in social psychology has been dominated by simpler techniques collectively known as manipulation checks (Sigall & Mills, 1998;Hauser et al, 2018).…”

“…Correlates: Indirect manipulation checks. Sometimes, however, it is not possible to directly measure the construct of interest, or there may be serious concerns that a direct measure could render a manipulation ineffective if it alerts participants to the purpose of the manipulation (Hauser, Ellsworth, & Gonzalez, 2018;Mills, 1969). It can also be useful to measure related constructs in order to provide convergent validity evidence (Cronbach & Meehl, 1955).…”