Response: Commentary: False-Positive Effect in the Radin Double-Slit Experiment on Observer Consciousness as Determined With the Advanced Meta-Experimental Protocol

Walleczek, Jan; Stillfried, Nikolaus von

doi:10.3389/fpsyg.2020.596125

Cited by 2 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The present work has clarified that the misrepresentations by Radin et al (2019Radin et al ( , 2020 -of the results and advanced protocol in Walleczek and von Stillfried (2019) -are entirely based on the detrimental practice of (undisclosed) HARKing that is used by Radin et al (2019Radin et al ( , 2020. The malpractice of HARKing by Radin is also the topic of the recent commentary by Walleczek and von Stillfried (2020) with a focus on two specific points: First, the so-called "true-positive outcome" was -contrary to the claim by Radin et al (2019Radin et al ( , 2020 -not the result of planned hypothesis testing, but was developed post-hoc, i.e., after unblinding the original results (see Fig. 1 in Sect.…”

Section: Discussionmentioning

confidence: 71%

“…HARKing increases greatly the risk of mistaking a false discovery for a true discovery or vice versa. Please note that a general commentary article by Walleczek and von Stillfried (2020) was recently published in Frontiers in Psychology based upon the main arguments and research record as will be described in the present work.…”

Section: Introductionmentioning

confidence: 99%

“…

A general commentary by Walleczek and von Stillfried (2020) was recently published in Frontiers in Psychology. The present work provides an account of (i) the detailed research record and (ii) the main arguments behind the commentary for the purpose of full transparency and disclosure.

…”

mentioning

confidence: 99%

See 2 more Smart Citations

False-positive Effect in the Radin Double-slit Experiment: HARKing is used by Radin et al. to Misrepresent the Advanced Meta-experimental Protocol used in Walleczek and von Stillfried (2019)

Walleczek¹,

Stillfried²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Walleczek and von Stillfried (2019) reported (i) the absence of any true-positive effects and (ii) the presence of one false-positive effect in a commissioned replication study of the Radin double-slit (DS) experiment on observer consciousness. In their recent misrepresentation, Radin et al. (2019, 2020) regrettably used the malpractice of HARKing, i.e., undisclosed hypothesizing after the results are known. HARKing can increase greatly the risk of false-negative or false-positive conclusions. Specifically, Radin et al. (2019, 2020) deviated in two major ways from the pre-specified protocol for this commissioned study, which (i) was agreed to by Radin before data collection was started (Radin, 2011) and (ii) included data encryption to prevent the use of p-hacking and HARKing. First, Radin et al. (2019) violate the original research design by reporting a so-called “true-positive outcome of a secondary planned hypothesis”. Contrary to the claim by Radin et al. (2019, 2020), that hypothesis was not, however, part of the planned test strategy, but, instead, the associated statistical analysis – a chi-square test – was chosen by Radin sometime after the planned statistical analysis had been completed and the data unblinded. Second, Radin et al. (2019, 2020) violate the funder-approved research design in an additional way by falsely claiming that the newly developed protocol, i.e., the advanced meta-experimental protocol (AMP), implements a non-predictive test strategy when – in fact – the AMP-based test strategy is strictly predictive. Put simply, Radin et al. (2019, 2020) are mistaken that the funder-approved hypotheses posited the random occurrence of effects for the test categories in this replication experiment; instead, a different specific prediction was tested in each of the eight planned test categories, and true-positive effects were predicted to occur for only two (12.5%) of the 16 possible measurement outcomes of the eight planned single-test categories. Therefore, in the predictive single-testing regime, a statistical correction for non-predictive, i.e., random, multiple testing would not be appropriate and would thus violate the AMP-based strategy, which was implemented in the commissioned study based upon the planned outcome predictions as pre-specified in Radin (2011). Neither of these post-hoc changes by Radin et al. (on the basis of HARKing) were disclosed in Radin et al. (2019, 2020) and both these changes violate the funder-approved, original methodology agreed upon in Radin (2011) and pre-specified in the research contract. In summary, the present work reconfirms that – exactly as reported in Walleczek and von Stillfried (2019) – “the false-positive effect, which would be indistinguishable from the predicted true-positive effect, was significant at p = 0.021 (σ = −2.02; N = 1,250 test trials)” and “no statistically significant effects could be identified” in those two groups for which true-positives were predicted to occur. These observations are consistent also with an independent statistical reanalysis of the Radin DS-experiment by Tremblay (2019) and a replication attempt by Guerrer (2019). Tremblay reported significant false-positives in control groups and Guerrer found significant effects with post-hoc analyses only, but null results only when using the planned confirmatory analysis. As a general recommendation, the authors call for the implementation of advanced control-test strategies, including novel approaches from the metascience reform movement, for empirically detecting and preventing uncontrolled false-positive effects in parapsychological research.

show abstract