Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations

Abstract

Publication bias occurs when studies with statistically significant results and large effects are more likely to be published than similarly rigorous studies with null and mixed findings. Results-blind peer review requires peer reviewers to consider only the “Introduction” and “Method” sections of submitted manuscripts prior to making editorial recommendations. This process ensures recommendations for publication focus on methodological rigor and not the direction, significance, or magnitude of the reported effects. The current investigation experimentally tested whether reviewers’ editorial recommendations and perceptions of manuscript importance, quality, and rigor varied as a function of type of review (i.e., results-blind or results-included) among 44 reviewers. Results indicated reviewer recommendations did not vary as a function of review type. However, reviewers found results-blind manuscripts less rigorous than results-included and reported less confidence in their recommendations on result-blinded manuscripts. Descriptive findings of results-blind reviewing were mixed with some support for the method, but a lack of confidence in its overall effectiveness. We discuss findings in relation to the conceptual benefits of results-blind reviewing and the increased focus on open and transparent science within special education and preview the papers included in the special section.

A Priori Justification for Effect Measures in Single-Case Experimental Designs

“…In the context of N-of-1 trials (which could be understood as a specific kind of SCED more common in medical research; Nikles & Mitchell, 2015, Tate & Perdices, 2019, it has been recommended that "all statistical methods planned-from visual representation to meta-analysis-should be described in the protocol" (Porcino et al, 2020, p. 10), including effect sizes, statistical significance, ways of performing sensitivity analyses, and how heterogeneity between participants will be assessed. In the SCED context, the importance of explicitly describing the expected data pattern and the expected effect of the intervention has also been recently emphasized (Maggin et al, 2020).…”

“…In the context of randomization tests, the test statistic is to be chosen before gathering the data, according to the type of effect expected (Heyvaert & Onghena, 2014a, 2014bLevin et al, 2017). Here, this idea is extended to other analytical options for SCED data, in line with current recommendations (Maggin et al, 2020). Deciding the analytical plan prior to gathering the data on the basis of the expected data pattern is possible when there is sufficient previous evidence on the specific kind of dependent variable or outcome score and the intervention.…”

Open Science and Single-Case Design Research

“…Similarly, norms and guidelines do not yet exist for writing and reviewing Stage 1 SCD manuscripts for registered reports. SCD researchers will have to grapple with how to provide reasonable evidence of internal validity without graphed data (see Maggin et al, 2020, for ideas on doing this, including providing graphs of hypothetical data). In addition, registered reports may not be a viable option for SCD researchers intervening on severe behavior (e.g., aggression to others), as the intervention would be delayed by the Stage 1 review process.…”

School Psychology vision for the future.

“…APA offers a range of open science practices for consideration among its journals and provides open science badges as incentives for engaging in such practices. Intriguing research has also begun to experimentally examine the impact of editorial practices designed to address publication bias (e.g., Maggin et al, 2020).…”