Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project

Abstract

The low reproducibility rate in social sciences has produced hesitation among researchers in accepting published findings at their face value. Despite the advent of initiatives to increase transparency in research reporting, the field is still lacking tools to verify the credibility of research reports. In the present paper, we describe methodologies that let researchers craft highly credible research and allow their peers to verify this credibility. We demonstrate the application of these methods in a multi-laboratory replication of Bem's Experiment 1 (Bem 2011 J. Pers. Soc. Psychol. 100 , 407–425. ( doi:10.1037/a0021524 )) on extrasensory perception (ESP), which was co-designed by a consensus panel including both proponents and opponents of Bem's original hypothesis. In the study we applied direct data deposition in combination with born-open data and real-time research reports to extend transparency to protocol delivery and data collection. We also used piloting, checklists, laboratory logs and video-documented trial sessions to ascertain as-intended protocol delivery, and external research auditors to monitor research integrity. We found 49.89% successful guesses, while Bem reported 53.07% success rate, with the chance level being 50%. Thus, Bem's findings were not replicated in our study. In the paper, we discuss the implementation, feasibility and perceived usefulness of the credibility-enhancing methodologies used throughout the project.

Prior beliefs and the interpretation of scientific results

“…10: 231613 4. Kekecs et al [4] replication, and the quantification of prior beliefs Arguably the most comprehensive replication attempt to-date is the Royal Society Open Science multilaboratory effort by Kekecs et al [4]. The authors of this paper had replicated Bem's experiment 1 across 10 different sites (laboratories), with 2115 participants yielding a total of 37 836 trials.…”

“…In the next section, I present two scenarios in experimental psychology that help illustrate the challenges to scientific interpretation posed by prior beliefs. The first example concerns competing models of decision making; the second example concerns Bem's [ 3 ] experiment on extrasensory perception and the multi-laboratory replication attempt recently published in the Royal Society [ 4 ]. These examples propel a broader discussion about the way our prior beliefs and expectations affect our interpretation of scientific results, and possibly the amount of the evidence we require to update our beliefs.…”

Championing “Exchange and Cooperation” Efforts in Frontier Science: Epilogue to the Special Issue

“…• Cross-disciplinary team science (e.g., Parnia et al, 2022) • Multi-lab studies (e.g., Maier et al, 2020) • Adversarial collaborations (e.g., Kekecs et al, 2023) Networks Tight A hub and wheel configuration with a local firm at the hub organizing the interdependencies of a complex array of firms. When a director or executive of one firm sits on the board of a second firm or when two firms have directors who also serve on the board of a second firm.…”

Correction to: ‘Raising the value of research studies in psychological science by increasing the credibility of research reports: The Transparent Psi Project’ (2023) by Kekecs et al.

“…This document provides corrections for the previously published paper titled 'Raising the value of research studies in psychological science by increasing the credibility of research reports: The Transparent Psi Project' [1] available at https://royalsocietypublishing.org/doi/10.1098/rsos.191375. The errors in the paper were uncovered by James E. Kennedy, who undertook a comprehensive audit of the project after its publication.…”

“…increasing the credibility of research reports: The Transparent Psi Project' [1]. The corrections are in response to the protocol deviations uncovered by a new research audit conducted on this project by James E. Kennedy [2].…”