Using results-blind reviewing to support the peer review competency framework

Abstract

The focal article by Köhler et al. (2020) develops a highly compelling competency framework for excellent reviewing. It is certainly a key piece of a robust and open science ecosystem (Banks et al., 2019). The framework can and should be used as a guide for novice and seasoned reviewers "in defining what they should pay attention to, upon what their review should touch, how they should word their review, what kind of advice they need to provide, and how they should overall go about peer reviewing" (Köhler et al., 2020, p. 17). Creating specific reviewer training around the competencies takes this framework to the next level by actively teaching reviewers how to implement the proposed guidelines. Developing these skills is a critical component for bettering the publication process by enabling reviewers to provide feedback to colleagues that is more organized, reliable, and helpful. The focal article, by design, has an individual focus. Thus, this commentary will serve as an extension to the focal article, pushing the call for better reviewing further by making the case for results-blind reviews (RBR) as an environmental mechanism that better supports the framework and training of competencies discussed in the focal article. A conducive environment Literature on transfer-of-training explores under what conditions knowledge and skills learned in training are applied in action (Baldwin & Ford, 1988). This area of research emphasizes the importance of situational cues and environmental factors, noting that these largely determine whether or not learned competencies are actually applied in the workplace (Blume, Ford, Baldwin, & Huang, 2010; Colquitt, Lepine, & Noe, 2000). It is our contention that the training of competencies alone may not be enough for successful transfer of reviewing skills as situation/context matters to what behaviors are exhibited by individuals (Dalal, Alaybek, Sheng, Holland, & Tomassetti, in press). Furthermore, the training focuses on explicit behaviors of reviewers, but does not target the implicit or unconscious biases that may exist in the system itself. Bias in the dominant peer-review system The current dominant peer review system in the fields of industrial and organizational psychology (I-O), organizational behavior (OB), and human resources (HR) requires authors to submit full manuscripts, including results, to a journal. There is good reason to believe that manuscript results can bias reviewers' judgments toward the rest of the article. This, in turn, can undermine the efficacy of the focal article's framework. Research suggests that favoritism toward significant and novel findings in the organizational sciences is widespread (

An urgent call for I-O psychologists to produce timelier technology research

“…Beyond conferences, we urge I-O psychologists to make use of recent pushes for increased transparency in the publication process to disseminate technology research in a timely fashion. Outlets such as the Journal of Business and Psychology now offer results-blind submission processes (Kreamer & Rogelberg, 2020), which supports technology research in that the review process is initiated before data collection is completed. Other peer-reviewed outlets such as PLOS ONE are open access and have a significantly shorter acceptance-to-publication time than traditional journals (Björk & Solomon, 2013).…”