Fearing the Future of Empirical Psychology: Bem's (2011) Evidence of Psi as a Case Study of Deficiencies in Modal Research Practice

Abstract

In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deficiencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufficient attention to verifying the soundness of measurement and experimental procedures, and (c) flawed implementation of null hypothesis significance testing. We argue that these deficiencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.

Falsifiability is not optional.

“…Social psychology prior to the "replication crisis" (pre-2011) (LeBel & Peters, 2011;Pashler & Harris, 2012). Though it is difficult to quantify precisely, bibliographic evidence suggests the prevalence rate of so-called direct replications in the published psychology literature pre-2011 is most likely less than 1% (Makel et al, 2012;M.…”

Open Science: What, Why, and How

“…The vocal dissatisfaction intensified after the publication of Bem's (2011) precognition paper in the most prestigious social psychology journal. Subsequent postpublication critique revealed numerous troubling flaws, especially related to data analysis (Wagenmakers, Wetzels, Boorsboom, & van der Maas, 2011) and research practices (LeBel & Peters, 2011 Among the important insights during the early discussion were that the concerns mentioned earlier were not independent of each other and that many, if not all, result from the problem of misaligned incentives: that what is good for being a successful scientist is not necessarily what is good for science itself (Nosek, Spies, & Motyl, 2012). Scientific practices might therefore be difficult to change if institutions don't change reward structures.…”

Enhancing Transparency of the Research Process to Increase Accuracy of Findings: A Guide for Relationship Researchers

“…One can then estimate the minimum effect size that is reliably detectable (e.g., at a 95% power level) for those sample sizes. From this perspective, observed null results become much more interpretable because one can nonetheless conclude (assuming integrity of manipulations and measurement instruments; LeBel & Peters, 2011) that the effect under study may exist, but would require substantially larger sample sizes to reliably detect (Simonsohn, 2014;LeBel, 2014). Ultimately, the fundamental point here is that specific factors used to determine the target sample size are disclosed in detail, thus making fully transparent the researchers' decision-making process.…”