Tools used to assess the quality of peer review reports: a methodological systematic review

Abstract

Background A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research. Methods We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis. Results We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of ‘quality’. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer’s comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18). Conclusion Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research. Electronic supplementary material The online version of this article (10.1186/s12874-019-0688-x) contains supplementary material, which is available to authorized users.

Journal editors’ perspectives on the roles and tasks of peer reviewers in biomedical journals: a qualitative study

“…In a recent study (2019) by Superchi et al that systematically reviewed tools used by journal editors to assess the quality of peer-review reports, the authors identified nine quality domains pertaining to peer-reviewer skills, of which five (ie, relevance and originality of the study, interpretation of study results, strength and weaknesses, manuscript presentation and organisation) arguably overlap with the skillset of authors. The remainder are directly concerned with skills related to structure and delivery of the peer-review report,27 which we believe may not automatically follow from being a prolific author. Therefore, we propose that the following four domains can, and in principle should, be taught to prospective reviewers: (1) structure of the reviewer’s comments; (2) characteristics of reviewers’ comments, including concepts such as clarity, constructiveness, detail/thoroughness, fairness, knowledgeability and tone; (3) timeliness of the review report; and (4) usefulness of the review report to editorial decision making and manuscript improvement.…”

The limitations to our understanding of peer review

“…This could then also be used to create standardised peer review 'check-lists' to help guide reviewers through the process. Research and development of tools for measuring the quality of peer review are only in their relative infancy [82], and even then focused mostly on disciplines such as biomedicine [88].…”

Quantifying professionalism in peer review

“…Editors must be vigilant and if allowed by their journal, screen such comments immediately. Finally, a variety of tools have been created to assess the quality of peer reviews, refer to Superchi, González [ 12 ] for a detailed review. Our coding structure offers one such method to evaluate reviewer behaviour.…”