Context Dependency as a Predictor of Replicability

Abstract

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

“…3, para. 1) disregards the (in our view, important) distinction between conceptually relevant and conceptually irrelevant design space dimensions (see Gollwitzer & Schwabe, 2022).…”

Does perceived scarcity of COVID-19 vaccines increase vaccination willingness? Results of an experimental study with German respondents in times of a national vaccine shortage

“…Against this background, it also has to be mentioned here that a significant number of participants was excluded as the experimental manipulation has not worked for them, i.e., their perceived availability of COVID-19 vaccines was not affected by the experimental condition. On one hand, using manipulation checks as an exclusion criterion guarantees that the participants under research have been influenced by the given information in the intended direction, which is a necessary precondition to draw any causal conclusions between the experimental variation and the variation on the dependent variables [ 32 , 33 ]. On the other hand, this practice is increasingly criticized in some social science disciplines [ 42 ] because of the risk of increasing biases between the sub-samples.…”

“…Following the recommendations of Leiner (2019), participants with a high relative speed index (RSI > 2), which is a measurement provided by the survey platform used and is based on processing time that identifies suspicious data patterns associated with poor quality, and reading times of equal to or less than 2 SD below the sample mean were excluded from data analysis [30,31]. There is a growing body of criticism on doing experimental research without controlling for a successful experimental manipulation within the sample under research [32,33]. Given that people's prior knowledge and information could have strongly influenced whether people believed the treatment texts or not, I preregistered the manipulation check variable as an exclusion criterion, as done in other experimental research [34][35][36].…”

Integrating the manual stimulation of acupuncture points into psychotherapy: A systematic review with clinical recommendations.

“…A crisis in science has, however, been emerging over the past two decades with the realization that scientists attempting to replicate many pivotal studies could not (Baker, 2016). This problem exists in the behavioral as well as in the biological sciences (Gollwitzer & Schwabe, 2021; Wiggins & Christopherson, 2019). For instance, a massive project involving 270 researchers using methodologically rigorous designs found that less than half of 90 major studies published in three top-ranking psychology journals were able to be replicated (Open Science Collaboration, 2015).…”