Results-blind review: a masked crusader for science

Woznyj, Haley M.; Grenier, Kelcie; Ross, Roxanne; Banks, George C.; Rogelberg, Steven G.

doi:10.1080/1359432x.2018.1496081

Cited by 13 publications

(6 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, only a minority of the participants found results-blind reviewing effective, and most did not recommend the practice for other journals or were unsure. Responses on the results-blind reviewing process are consistent with the previous research that has found openness to the process, but some reticence for widespread adoption (Woznyj et al, 2018).…”

Section: Discussionsupporting

confidence: 85%

“…In results-blind peer review, reviewers evaluate the quality of the study based on its theoretical and methodological rigor alone; the assumption being that research quality relates to the study’s theoretical underpinnings and methodological rigor, not the direction or magnitude of the findings. Research journals from a number of fields, including political science (Findley et al, 2016), organizational psychology (Woznyj et al, 2018), medicine (Button et al, 2016), and neuroscience (Chambers et al, 2014), have piloted or instituted results-blind peer review as a method of counteracting publication bias. Evaluations of these pilots have generally focused on author and reviewer perceptions after participating in results-blind peer review and yielded mixed results.…”

mentioning

confidence: 99%

“…Evaluations of these pilots have generally focused on author and reviewer perceptions after participating in results-blind peer review and yielded mixed results. For example, both Findley et al (2016) and Woznyj et al (2018) found that researchers supported the potential for results-blind review in theory, but had varying concerns about logistics and unintended consequences of implementation.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations

Maggin

Robertson

Cook

2020

Behavioral Disorders

View full text Add to dashboard Cite

Publication bias occurs when studies with statistically significant results and large effects are more likely to be published than similarly rigorous studies with null and mixed findings. Results-blind peer review requires peer reviewers to consider only the “Introduction” and “Method” sections of submitted manuscripts prior to making editorial recommendations. This process ensures recommendations for publication focus on methodological rigor and not the direction, significance, or magnitude of the reported effects. The current investigation experimentally tested whether reviewers’ editorial recommendations and perceptions of manuscript importance, quality, and rigor varied as a function of type of review (i.e., results-blind or results-included) among 44 reviewers. Results indicated reviewer recommendations did not vary as a function of review type. However, reviewers found results-blind manuscripts less rigorous than results-included and reported less confidence in their recommendations on result-blinded manuscripts. Descriptive findings of results-blind reviewing were mixed with some support for the method, but a lack of confidence in its overall effectiveness. We discuss findings in relation to the conceptual benefits of results-blind reviewing and the increased focus on open and transparent science within special education and preview the papers included in the special section.

show abstract

Section: Discussionsupporting

confidence: 85%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations

Maggin

Robertson

Cook

2020

Behavioral Disorders

View full text Add to dashboard Cite

show abstract

“…The RBR process generally eliminates implicit or explicit incentives for researchers to engage in QRPs, instead encouraging scholars to submit manuscripts regardless of whether they found significant results (Button et al, 2016). Woznyj, Grenier, Ross, Banks, and Rogelberg (2018) explored the field's reactions (including practitioners) to the RBR initiative and found that a majority of stakeholders in the organizational sciences (roughly 75%) believes the field would benefit from blind reviewing. Seventy-two percent of stakeholders believe the initiative would help advance cumulative knowledge, while 65% believe RBR would combat against questionable research practices.…”

Section: Results Blind Reviewsmentioning

confidence: 99%

Using results-blind reviewing to support the peer review competency framework

Kreamer

Rogelberg

2020

Ind. Organ. Psychol.

Self Cite

View full text Add to dashboard Cite

The focal article by Köhler et al. (2020) develops a highly compelling competency framework for excellent reviewing. It is certainly a key piece of a robust and open science ecosystem (Banks et al., 2019). The framework can and should be used as a guide for novice and seasoned reviewers "in defining what they should pay attention to, upon what their review should touch, how they should word their review, what kind of advice they need to provide, and how they should overall go about peer reviewing" (Köhler et al., 2020, p. 17). Creating specific reviewer training around the competencies takes this framework to the next level by actively teaching reviewers how to implement the proposed guidelines. Developing these skills is a critical component for bettering the publication process by enabling reviewers to provide feedback to colleagues that is more organized, reliable, and helpful. The focal article, by design, has an individual focus. Thus, this commentary will serve as an extension to the focal article, pushing the call for better reviewing further by making the case for results-blind reviews (RBR) as an environmental mechanism that better supports the framework and training of competencies discussed in the focal article. A conducive environment Literature on transfer-of-training explores under what conditions knowledge and skills learned in training are applied in action (Baldwin & Ford, 1988). This area of research emphasizes the importance of situational cues and environmental factors, noting that these largely determine whether or not learned competencies are actually applied in the workplace (Blume, Ford, Baldwin, & Huang, 2010; Colquitt, Lepine, & Noe, 2000). It is our contention that the training of competencies alone may not be enough for successful transfer of reviewing skills as situation/context matters to what behaviors are exhibited by individuals (Dalal, Alaybek, Sheng, Holland, & Tomassetti, in press). Furthermore, the training focuses on explicit behaviors of reviewers, but does not target the implicit or unconscious biases that may exist in the system itself. Bias in the dominant peer-review system The current dominant peer review system in the fields of industrial and organizational psychology (I-O), organizational behavior (OB), and human resources (HR) requires authors to submit full manuscripts, including results, to a journal. There is good reason to believe that manuscript results can bias reviewers' judgments toward the rest of the article. This, in turn, can undermine the efficacy of the focal article's framework. Research suggests that favoritism toward significant and novel findings in the organizational sciences is widespread (

show abstract

“…PsycINFO, PubMed Central, Web of Science, Scopus). Restricting electronic searches to specific journals (e.g., defined by an impact factor threshold or one of the many lists that rank the quality of journals) (also see next section) does run a risk of bias because of the problems identified in more selective journals favoring novelty over replication and bias towards significant results (for a discussion and a proposed solution, see Woznyj, Grenier, Ross, Banks & Rogelberg, 2018).…”

Section: Searching the Literaturementioning

confidence: 99%

Guidance on conducting and reviewing systematic reviews (and meta-analyses) in work and organizational psychology

Daniels

2018

European Journal of Work and Organizational Psychology

View full text Add to dashboard Cite

Systematic reviews and meta-analyses are means of summarizing and synthesizing research evidence in a given topic area. They can be used: to define the current state of knowledge and how confident we can be in that knowledge; to identify evidence gaps; and to provide recommendations for policy and practice based on the best available evidence. At EJWOP, our editorial stance is explicitly to encourage the conduct of systematic reviews and metaanalyses. The purpose of this editorial is to provide some guidance to authors and journal referees on the (technical) features of good systematic reviews.

show abstract

Results-blind review: a masked crusader for science

Cited by 13 publications

References 58 publications

Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations

Introduction to the Special Series on Results-Blind Peer Review: An Experimental Analysis on Editorial Recommendations and Manuscript Evaluations

Using results-blind reviewing to support the peer review competency framework

Guidance on conducting and reviewing systematic reviews (and meta-analyses) in work and organizational psychology

Contact Info

Product

Resources

About