Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency framework for peer review

Köhler, Tine; González-Morales, M. Glória; Banks, George C.; O’Boyle, Ernest H.; Allen, Joseph A.; Sinha, Ruchi; Woo, Sang Eun; Gulick, Lisa

doi:10.1017/iop.2019.121

Cited by 44 publications

(36 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In so doing, we address one of the many issues that are raised in the focal article. Although much narrower in scope than the competency framework proposed by Köhler et al (2020), our focus on methodological reporting is consistent with several of the defining principles for a robust science of I-O psychology as proposed by Grand et al (2018): rigor, reproducibility, replication, and transparency/openness. Ideally, a methodological checklist would serve as a guide when preparing (in the "guide to authors" section of journal websites), reviewing (sent to reviewers alongside the manuscript), and accepting (used by editors when processing conditionally accepted manuscripts) empirical manuscripts.…”

supporting

confidence: 65%

Methodological checklists for improving research quality and reporting consistency

Eby¹,

Shockley²,

Bauer

et al. 2020

Ind. Organ. Psychol.

View full text Add to dashboard Cite

The focal article by Köhler et al. (2020) provides a useful framework for promoting robust, rigorous, and reliable reviewing by developing a competency framework for reviewers. We applaud important efforts such as this, aimed at enhancing the peer review process and creating common peer review expectations. It is also useful that Köhler et al. tackled this complex issue from the vantage point of multiple stakeholders (e.g., academia, organizations, granting agencies). In this commentary, we focus specifically on the peer review process in academic journals, given our reviewing and editorial experience. The author team's experience includes 157 collective years reviewing for peer-reviewed academic journals, membership on 33 different editorial boards, serving as associate editors for 15 different journals, and leading 7 journals as editor-in-chief, guest editor, or special issue editor. As outlined by Köhler et al. (2020), clear standards for what constitutes an effective review have potential benefits for authors (e.g., more developmental and actionable feedback), reviewers (e.g., clearer reviewing expectations, improved knowledge and skills for reviews), and journal editors and associate editors (henceforth referred to as "editors"; e.g., greater efficiency in processing reviews and reaching decisions), as well as the field as a whole (e.g., improved scientific rigor, reliability, and reproducibility). One of the approaches proposed in the focal article was training on the competency framework (1) in academic classes that discuss peer reviewing, (2) when providing feedback to PhD students who prepare reviews, and (3) at conferences and through the Consortium for the Advancement of Research Methods and Analysis (CARMA). As important and useful as training-based approaches are, such efforts tend to have a relatively limited reach, relative to the number of authors and reviewers (and editors) for which such training would be beneficial. More specifically, although there is overlap in membership, The Society for Industrial and Organizational Psychology has over 10,000 members, the Human Resource Management and Organizational Behavior Divisions of the Academy of Management have over 9,000 members, and the European Association of Work and Organizational Psychology has approximately 2,000 members. Thus, providing formal training for industrial and organizational (I-O) psychologist reviewers on the large-and-growing scale seems formidable or perhaps even prohibitive. Reviewing is also a volunteer activity, so it may be unrealistic to expect individuals to invest a great deal of time on reviewer training given many superseding obligations (e.g., for faculty, this may include teaching, conducting research, mentoring students, applying for grants, The third through ninth authors contributed equally and are listed alphabetically. The ideas presented in this commentary do not reflect formal policy at any journal (e.g., Journal of Applied Psychology) or professional association (e.g., American Psychological Association).

show abstract

supporting

confidence: 65%

Methodological checklists for improving research quality and reporting consistency

Eby¹,

Shockley²,

Bauer

et al. 2020

Ind. Organ. Psychol.

View full text Add to dashboard Cite

show abstract

“…Practically, however, role conflicts may frustrate the review process when they are not recognized and addressed either by guidelines or by formal review processes. Increasing awareness and ideally an explicit focus on the reduction of role conflicts may further enhance the merit of the already useful guidelines set forth by Köhler et al (2020).…”

Section: Reducing Role Conflictmentioning

confidence: 99%

Peer review and role conflict

Voogt

Runge

2020

Ind. Organ. Psychol.

View full text Add to dashboard Cite

In their focal article, Köhler et al. (2020, p. 16) point out that reviewers need to balance their "dual roles as gatekeepers : : : with the generative and developmental role of peers : : : ." The proposed framework also includes the implicit assumption that all proposed competencies are ultimately compatible with each other. Although the framework features a thoughtful and thorough list of necessary competencies, it may also be regarded as too optimistic. Each of the behaviors that is in line with these competencies can be seen as part of a role that needs to be fulfilled concurrently with other roles and ideally by the same person during the reviewing task. We argue that such roles include inherent role conflicts and role ambiguity between different goals and tasks the reviewer is assigned. For instance, it is not clear if the combination of scouting for statistical quality and at the same time for novel findings is always realistic; or, if socialization into the field can always be reconciled with advancement to the field. One may differ about which roles should be paramount, and the selection put forward in this framework is arguably a suitable compromise. To some degree, peer review has roles that could be in line with each other, in which case one could use role theory (Biddle, 1986; Ebbers & Wijnberg, 2017) to develop a job performance scale to measure the reviewers' performance (also see Welbourne, Johnson, & Erez, 1998), an element that would make the suggested framework perhaps more rigorous. However, there is inherent conflict and ambiguity between the different roles in peer review, and tensions around peer review are likely the result from this inherent role conflict. Extensive research has shown that role conflict and ambiguity are negatively related to job performance (Rizzo, House, & Lirtzman, 1970; Tubre & Collins, 2000), which likely extends to peer review. In an effort to make reviews more reliable, Köhler et al. (2020) suggest that reviewers should not alternate between roles and fill all roles simultaneously. While this approach sounds appealing at first glance, it may be difficult to implement in practice. For instance, a reviewer of an article may find only one role particularly suitable for his/her background and expertise. He/she may be encouraging the authors of the article in a certain direction that would potentially lead to a theoretically more rigorous contribution, while in the case of a different article, he/she may point out the many problems in the theoretical approach and thereby disqualify, for instance, the publication of an article. In what is named "mechanical skills," Köhler et al. (2020, p. 13) state in relation to commentary provided by reviewers: "So, we need to communicate realistically what needs to be improved and what can be improved in future work or in a next step." And in terms of "professionalism," the reviewer needs to be constructive. All these elements would suggest that reviewers should encourage authors in a certain direction. As a potential concern, Köhler ...

show abstract

mentioning

confidence: 99%

“…Peer review may be one of the most important duties in our profession (Grand et al, 2018;Köhler et al, 2020). Many of the suggestions made in the focal article by Köhler et al (2020) would indeed benefit the peer review process and the science and practice of industrial and organizational psychology. However, if reviewers were to act on these suggestions, it is possible that they would need to expend a great degree of additional time, effort, and resources to successfully complete many of them (i.e., seeking out additional information when expertise is lacking, giving actionable and developmental feedback).…”

mentioning

confidence: 99%

“…Indeed, they must often make tradeoffs between writing thorough reviews and completing other highly rewarded work (i.e., publishing one's own research). To encourage others to adopt the suggestions presented by Köhler et al (2020), I posit that incentives are a necessary consideration when soliciting high quality reviews using the competency framework. To support this assertion, I draw from previous research on motivation and provide suggestions for potential incentives.…”

mentioning

confidence: 99%

See 1 more Smart Citation