A structured mentored review program improves the quality of peer review: a pre-post intervention study

Abstract

Background: Peer review is essential to the advancement of knowledge. However, training on how to conduct peer review is limited, unorganized, and not well studied. Thus, we sought to determine if a structured mentored peer-review program improves the quality of peer review as measured by quantitative assessment of peer review of a standardized manuscript. Methods: This pre-post intervention study enrolled 55 mentees across 5 cohorts from 2020 to 2023. Each cohort completed pre-program evaluations, participated in two mentored reviews, and completed post-program evaluations over 6 months. The primary outcome measured was total score on the modified Review Quality Index (RQI). Secondary outcomes included participant-reported comfort with and understanding of the review process as well RQI subscores. Pre- and post-program measures were compared using the Wilcoxon test. The a priori hypothesis was that mentored review would improve peer review quality. Results: 42 mentees completed both pre- and post-program reviews of a standardized manuscript. For these participants, post-program total modified RQI score (median (IQR) = 31 (26.25-35.75)) was significantly higher than pre-program total scores (26.59 (19.67-29.46)). 53 mentees completed both pre- and post-program surveys which showed improved participant perceptions of the review (median (IQR) scores pre = 4 (3-4) , post =5 (4-5)) and editorial processes (pre = 3 (2-4), post = 4 (4-5)). In addition, confidence in completing an independent review of both scientific (median (IQR) scores pre = 2 (2-3), post = 4 (4-4)) and non-scientific (pre = 3 (2-4) , post = 4 (4-5)) manuscripts significantly increased following program participation. p < 0.0001 for all scores noted above. Conclusions: A six-month structured mentored-review program including 2 mentored reviews improves peer review quality as measured by the modified RQI as well as participant understanding of publication science.