Making inferential leaps: Manipulation checks and the road towards strong inference

Abstract

Experiments in psychology often target hypothetical constructs to test some causal hypothesis or theory. In light of this goal, it is pertinent to use a manipulation that actually changes the focal hypothetical construct, and only that construct. In assessing whether such manipulation "success" can be assumed, researchers often include manipulation validity checks in their designs-a measure of the focal construct which should be responsive to the manipulation. One interpretation of a positive manipulation check is that it lends credence to a particular causal interpretation of a phenomenon. Scrutinizing the results of such manipulation checks supposedly enables a more stringent test of a causal hypothesis. This paper submits that manipulation checks do not improve our inferences to causal explanations, but may in practice result in weaker hypothesis tests. Rather than being useful, manipulation checks are at best uninformative, but more likely compromise the appraisal of a causal hypothesis. The second half of this paper advocates four methodological heuristics, offered as alternatives to manipulation validity checks, to more severely test causal hypotheses. The heuristics call for a burgeoning focus on (a) assessing the specificity of manipulations, (b) evaluating theoretical risk, (c) attempts to cast doubt on alternatives, and (d) appraising the relative merits of explanations. I conclude that rather than relying on manipulation checks as a 'Band-Aid' method to alleviate validity concerns, inferential rigor can be improved by virtue of these heuristics.

Credibility Beyond Replicability: Improving the Four Validities in Psychological Science

“…Concerns about the lack of validation and inadequacy of validation techniques can also be raised regarding experimental manipulations, and this has also gotten renewed attention, including a vigorous debate about the uses and misuses of manipulation checks (e.g., Ejelöv & Luke, 2020;Gruijters, 2022). Similarly, behavioral, cognitive, and physiological measures are often assumed to be valid without proper verification.…”

The logical structure of experiments lays the foundation for a theory of reproducibility

“…For example, manipulations that are not tested for specificity may end up manipulating non-focal constructs or only weakly manipulate the focal construct (i.e. leading to small effect sizes) [34].…”

Mapping the royal road to science: Towards a heuristic for interpreting experimental evidence

“…Elsewhere I have argued that a causal conception of validity can be refitted to the context of experimental manipulation (Gruijters, 2022).…”

“…More frankly, Cook and Campbell's (1979) extension of construct validity theory towards manipulations is in my view basically regrettable. Extending Borsboom's (2004) causal definition of validity to manipulations allows for a simple and appropriate definition of when a manipulation can be seen as valid: a manipulation is valid if, and only if, it causes variation on the target attribute (see also Gruijters, 2022).…”

“…It is likely that fat handed manipulations are common in the field of psychology (Eronen, 2020;Eronen & Bringmann, 2021;Gruijters, 2022). In The figure displays a causal graph of all possible explanatory paths involved in a simple bivariate experiment.…”