Effects of training on quality of peer review: randomised controlled trial

Schroter, Sara; Black, Nick; Evans, Stephen; Carpenter, James; Godlee, Fiona; Smith, Richard

doi:10.1136/bmj.38023.700775.ae

Cited by 221 publications

(208 citation statements)

References 12 publications

Supporting

Mentioning

196

Contrasting

Unclassified

Order By: Relevance

“…The objective of this paper is to report the frequency with which the nine major and five minor errors were detected and the impact that training had on each of the 14 errors studied. As the methods of the trial and primary results have previously been reported, 10 they are described only briefly in this paper. In this paper the data from the RCT is used as observational data.…”

Section: Researchmentioning

confidence: 99%

“…9 We conducted a single blind randomized controlled trial (RCT) on the effect of training on the performance of peer reviewers of a general medical journal, the BMJ. 10 Reviewers were randomized to one of three groups (control, face-to face training and self-taught) and invited to review three manuscripts during the study period. The training package focused on what editors want from reviewers and how to critically appraise RCTs.…”

Section: Researchmentioning

confidence: 99%

See 1 more Smart Citation

What errors do peer reviewers detect, and does training improve their ability to detect them?

Schroter¹,

et al. 2008

Self Cite

View full text Add to dashboard Cite

Objective To analyse data from a trial and report the frequencies with which major and minor errors are detected at a general medical journal, the types of errors missed and the impact of training on error detection.Design 607 peer reviewers at the BMJ were randomized to two intervention groups receiving different types of training (face-to-face training or a self-taught package) and a control group. Each reviewer was sent the same three test papers over the study period, each of which had nine major and five minor methodological errors inserted.Setting BMJ peer reviewers. Main outcome measuresThe quality of review, assessed using a validated instrument, and the number and type of errors detected before and after training. ResultsThe number of major errors detected varied over the three papers.The interventions had small effects. At baseline (Paper 1) reviewers found an average of 2.58 of the nine major errors, with no notable difference between the groups.The mean number of errors reported was similar for the second and third papers, 2.71 and 3.0, respectively. Biased randomization was the error detected most frequently in all three papers, with over 60% of reviewers rejecting the papers identifying this error. Reviewers who did not reject the papers found fewer errors and the proportion finding biased randomization was less than 40% for each paper.Conclusions Editors should not assume that reviewers will detect most major errors, particularly those concerned with the context of study. Short training packages have only a slight impact on improving error detection.

show abstract

Section: Researchmentioning

confidence: 99%

Section: Researchmentioning

confidence: 99%

What errors do peer reviewers detect, and does training improve their ability to detect them?

Schroter¹,

et al. 2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…Issues resulting from peer review failure range from simple gate-keeping errors, based on differences in opinion of the perceived impact of research, to failing to detect fraudulent or incorrect work, which then enters the scientific record ( Baxt et al , 1998; Gøtzsche, 1989; Haug, 2015; Moore et al , 2017; Pocock et al , 1987; Schroter et al , 2004; Smith, 2006). A final issue regards peer review by and for non-native English speaking authors, which can lead to cases of linguistic inequality and language-oriented research segregation, in a world where research is increasingly becoming more globally competitive ( Salager-Meyer, 2008, Salager-Meyer, 2014).…”

Section: 01 Methodsmentioning

confidence: 99%

A multi-disciplinary perspective on emergent and future innovations in peer review

et al. 2017

View full text Add to dashboard Cite

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

show abstract

“…BMJ actually tested this hypothesis by randomizing reviewers into three groups: one group had no training, one received personal training in addition to a CD-ROM, and the third group received just the CD-ROM. After using a standardized instrument to measure the quality of reviews and counting the number of intentionally placed errors detected by reviewers, the results showed that training made no overall difference [8].…”

Section: Reviewmentioning

confidence: 99%

A review of the review process

Dye¹

2007

J. Med. Toxicol.

View full text Add to dashboard Cite

Effects of training on quality of peer review: randomised controlled trial

Cited by 221 publications

References 12 publications

What errors do peer reviewers detect, and does training improve their ability to detect them?

What errors do peer reviewers detect, and does training improve their ability to detect them?

A multi-disciplinary perspective on emergent and future innovations in peer review

A review of the review process

Contact Info

Product

Resources

About