Mentored peer review of standardized manuscripts as a teaching tool for residents: a pilot randomized controlled multi-center study

Abstract

BackgroundThere is increasing need for peer reviewers as the scientific literature grows. Formal education in biostatistics and research methodology during residency training is lacking. In this pilot study, we addressed these issues by evaluating a novel method of teaching residents about biostatistics and research methodology using peer review of standardized manuscripts. We hypothesized that mentored peer review would improve resident knowledge and perception of these concepts more than non-mentored peer review, while improving review quality.MethodsA partially blinded, randomized, controlled multi-center study was performed. Seventy-eight neurology residents from nine US neurology programs were randomized to receive mentoring from a local faculty member or not. Within a year, residents reviewed a baseline manuscript and four subsequent manuscripts, all with introduced errors designed to teach fundamental review concepts. In the mentored group, mentors discussed completed reviews with residents. Primary outcome measure was change in knowledge score between pre- and post-tests, measuring epidemiology and biostatistics knowledge. Secondary outcome measures included level of confidence in the use and interpretation of statistical concepts before and after intervention, and RQI score for baseline and final manuscripts.ResultsSixty-four residents (82%) completed initial review with gradual decline in completion on subsequent reviews. Change in primary outcome, the difference between pre- and post-test knowledge scores, did not differ between mentored (−8.5%) and non-mentored (−13.9%) residents (p = 0.48). Significant differences in secondary outcomes (using 5-point Likert scale, 5 = strongly agree) included mentored residents reporting enhanced understanding of research methodology (3.69 vs 2.61; p = 0.001), understanding of manuscripts (3.73 vs 2.87; p = 0.006), and application of study results to clinical practice (3.65 vs 2.78; p = 0.005) compared to non-mentored residents. There was no difference between groups in level of interest in peer review (3.00 vs 3.09; p = 0.72) or the quality of manuscript review assessed by the Review Quality Instrument (RQI) (3.25 vs 3.06; p = 0.50).ConclusionsWe used mentored peer review of standardized manuscripts to teach biostatistics and research methodology and introduce the peer review process to residents. Though knowledge level did not change, mentored residents had enhanced perception in their abilities to understand research methodology and manuscripts and apply study results to clinical practice.Electronic supplementary materialThe online version of this article (doi:10.1186/s41073-017-0032-0) contains supplementary material, which is available to authorized users.

Research Integrity and Peer Review—past highlights and future directions

“…In the peer review section, several articles published have focused on diverse aspects of the peer review process. These span a deeper dive into how grant review panels work [ 33 ] and the need for support for peer reviewers, particular in areas of training [ 34 , 35 ] and mentoring [ 36 ]. Other topics include a study of reviewer recruitment in the field of ecology [ 37 , 38 ] and research into the views of junior hospital doctors on their understanding of models of peer review [ 39 ].…”

Individual versus general structured feedback to improve agreement in grant peer review: A randomized controlled trial

“…Based on the ve quali ed studies, the review concluded that "training did not improve the quality of the peer review report". Most studies identi ed in this review and in other training trials [8,9] test the effect of structured training or mentoring sessions, and the outcomes seldom include interrater reliability or agreement as the outcome measure. In addition, most of the studies focus on journal peer review rather than grant peer review.…”

Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial

“…Based on the five qualified studies, the review concluded that "training did not improve the quality of the peer review report". Most studies identified in this review and in other training trials [12,13] test the effect of structured training or mentoring sessions, and the outcomes seldom include interrater reliability or agreement as the outcome measure. In addition, most of the studies focus on journal peer review rather than grant peer review.…”