What makes an effective grants peer reviewer? An exploratory study of the necessary skills

Davis, Miriam L E Steiner; Conner, Tiffani; Miller-Bains, Kate; Shapard, Leslie

doi:10.1371/journal.pone.0232327

Cited by 7 publications

(2 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the literature, subject matter expertise is considered a key attribute of reviewers (e.g. Chubin, 1994;Gallo, Sullivan and Glisson, 2016;Steiner Davis et al, 2020). Less widespread, for example, is the view that peers are defined by reputation, standing, and eliteness (e.g.…”

Section: Reasons For the Basic Assumptionmentioning

confidence: 99%

Do peers share the same criteria for assessing grant applications?

Hug

Ochsner

2021

Research Evaluation

View full text Add to dashboard Cite

This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (10 criteria related to knowledge gaps; feasibility; rigour; comprehensibility and argumentation; academic relevance; competence and experience of the applicant) and a broad consensus that include the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increase from the consensus far to the consensus close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.

show abstract

Section: Reasons For the Basic Assumptionmentioning

confidence: 99%

Do peers share the same criteria for assessing grant applications?

Hug

Ochsner

2021

Research Evaluation

View full text Add to dashboard Cite

show abstract

“…[5] High quality peer review is one way to ensure integrity but requires that reviewers have (1) su cient content knowledge, (2) the ability to critically appraise scienti c study, (3) effective written communication skills, and (4) an understanding of the editorial process and purpose. [6][7][8] Despite its importance, the process of learning how to review a manuscript is largely informal and is not required in medical training. [9] Journals often call upon reviewers based on their content expertise; however, there are limited resources to train reviewers in critical appraisal, effective communication, and journalology.…”

Section: Introductionmentioning

confidence: 99%

A structured mentored review program improves the quality of peer review: a pre-post intervention study

Lyons-Warren

Aamodt

Pieper

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Peer review is essential to the advancement of knowledge. However, training on how to conduct peer review is limited, unorganized, and not well studied. Thus, we sought to determine if a structured mentored peer-review program improves the quality of peer review as measured by quantitative assessment of peer review of a standardized manuscript. Methods: This pre-post intervention study enrolled 55 mentees across 5 cohorts from 2020 to 2023. Each cohort completed pre-program evaluations, participated in two mentored reviews, and completed post-program evaluations over 6 months. The primary outcome measured was total score on the modified Review Quality Index (RQI). Secondary outcomes included participant-reported comfort with and understanding of the review process as well RQI subscores. Pre- and post-program measures were compared using the Wilcoxon test. The a priori hypothesis was that mentored review would improve peer review quality. Results: 42 mentees completed both pre- and post-program reviews of a standardized manuscript. For these participants, post-program total modified RQI score (median (IQR) = 31 (26.25-35.75)) was significantly higher than pre-program total scores (26.59 (19.67-29.46)). 53 mentees completed both pre- and post-program surveys which showed improved participant perceptions of the review (median (IQR) scores pre = 4 (3-4) , post =5 (4-5)) and editorial processes (pre = 3 (2-4), post = 4 (4-5)). In addition, confidence in completing an independent review of both scientific (median (IQR) scores pre = 2 (2-3), post = 4 (4-4)) and non-scientific (pre = 3 (2-4) , post = 4 (4-5)) manuscripts significantly increased following program participation. p < 0.0001 for all scores noted above. Conclusions: A six-month structured mentored-review program including 2 mentored reviews improves peer review quality as measured by the modified RQI as well as participant understanding of publication science.

show abstract

Tips and guidelines for being a good peer reviewer

Gisbert¹,

Chaparro²

2023

Gastroenterología y Hepatología (English Edition)

View full text Add to dashboard Cite

What makes an effective grants peer reviewer? An exploratory study of the necessary skills

Cited by 7 publications

References 43 publications

Do peers share the same criteria for assessing grant applications?

Do peers share the same criteria for assessing grant applications?

A structured mentored review program improves the quality of peer review: a pre-post intervention study

Tips and guidelines for being a good peer reviewer

Contact Info

Product

Resources

About