Learning to be confident and capable journal reviewers: an Australian perspective

Abstract

Journal peer review has been the subject of much research. However, the learning process through which reviewers acquire their reviewing ability, and reviewers' own perceptions of their capability have rarely been a focus. This interview study asked three questions about reviewer capability and training. At what stage did you gain confidence in reviewing? How did you learn how to review? Is formal training necessary? The interview is part of a mixed‐method project studying experienced Australian reviewers. The respondents indicate that learning to review is a continuous cycle in which formal training will not work. Following a mostly self‐guided initiation, new reviewers establish personal reviewing patterns. By trial and error, the patterns are consolidated and the reviewers eventually feel ‘confident’. ‘Decisiveness' is a good sign of becoming confident. Most respondents emphasized that journals could play a crucial role in producing good reviewers, e.g. by specifying unambiguously their expectations of ‘good reviews’.

Editors’ Afterword

“…Despite the fact that a significant proportion of academic faculty members formally review journal manuscripts, and despite the increasing body of literature describing aspects of the peer-review process, it is surprising that few standards and criteria have emerged that characterize an optimal review. Consistent with our assertion here, Lu (2012) made the following observation:…”

“…However, we know only snippets about quality judgements; for example, there is high agreement between reviewers when it comes to rejection and in identifying a paper of high quality. Lu's (2012) mixed methods research study of 44 experienced reviewers was that these participants emphasized the important role that journal editors can play in motivating good reviewing by specifying explicitly what a good review is. Thus, the primary goal of the current editorial has been to begin the conversation among members of the mixed methods research community regarding what makes a quality review by providing an evidence-based framework for comprehensively reviewing mixed methods research manuscripts.…”

“…Learning to review journal manuscripts represents a self-guided journey, typically involving minimal mentoring and support, with little or no explicit guidelines (Lu, 2012). Despite the fact that a significant proportion of academic faculty members formally review journal manuscripts, and despite the increasing body of literature describing aspects of the peer-review process, it is surprising that few standards and criteria have emerged that characterize an optimal review.…”

A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology

“…Evidence also suggests that the quality of one's peer reviewing deteriorates over time [47,48] and that peer reviewers are susceptible to positive-outcome bias [54]. The limited training available means that most reviewers are being guided mainly by journals' instructions to peer reviewers or being forced to learn by trial and error [55]. For most medical residents, the only exposure to appraisal of research manuscripts comes from their participation in journal clubs [43].…”

Systematic review of the effectiveness of training programs in writing for scholarly publication, journal editing, and manuscript peer review (protocol)

“…Eastwood also points out that ‘few of the programs developed to meet the National Institutes of Health requirement for training in responsible research practices devote time to the practice and ethics of biomedical reporting [17]’. Similarly, for peer reviewers, there is little to no formal training available, with most reviewers being guided by journals’ instructions to reviewers sections and being forced to learn by trial-and-error [28]. …”