Targeted, actionable and fair: reviewer reports as feedback and its effect on ECR career choices

Abstract

Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the ‘best’ decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the function of peer review by examining how it can be used as a participatory research governance tool by focusing on the function feedback plays in assisting in the development of ECR applicants. Using a combination of survey, interviews and linguistic-based coding of reviewer reports, this study explores how reviewer reports provided to unsuccessful applicants as an artefact of the peer-review decision making process, can be considered as a method of feedback. Specifically, it examines which components of this feedback underpinned their decisions to re-submit their grant applications following first-failure; change their research topics or withdraw from academia entirely. Peer review feedback, we argue, sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The results also highlight a function of peer review overlooked by current research which is not concentrated solely on the development of an outcome, to one that can be used effectively to support the development of individuals and their future research plans.

Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices

“…An initial survey ( Derrick et al 2021 ) of both successful and unsuccessful applicants to the Wellcome Trust ( n = 233), identified a sample of unsuccessful applicants at T 0 who were then invited to an interview ( n = 19) to explore their career decisions and feedback received at T 0 .…”

“…The second round of coding of interviews took into account the participants reaction to feedback that contained one- or more characteristics, and participants association of the feedback as a whole (for survey results see Derrick et al 2021 ). In summary, when the feedback received all three of these components (targeted, actionable and fair, Table 1 ), applicants considered the feedback as ‘good’ and ‘positive’.…”

What Factors are Important to the Success of Resubmitted Grant Applications in Health Research? A Retrospective Study of Over 20,000 Applications to the Canadian Institutes of Health Research

“…Unlike journal peer review, providing applicant feedback is a lower priority of grant peer review committees [1,22] . However, applicants are recommended to, and do, use feedback to help with grant resubmission [7,10,23] . High-quality feedback helps researchers decide whether to submit their application [7] ; low-quality feedback can confuse applicants [24] and may lead to multiple unsuccessful resubmissions [8] .…”

“…However, applicants are recommended to, and do, use feedback to help with grant resubmission [7,10,23] . High-quality feedback helps researchers decide whether to submit their application [7] ; low-quality feedback can confuse applicants [24] and may lead to multiple unsuccessful resubmissions [8] . CIHR has clear guidance for reviewers to promote high-quality reviews [25] , which may have contributed to the improved success of resubmissions seen here.…”

“…In the U.S., resubmitted applications to the National Institute of Health (NIH) programs are more successful than new applications [2,6] , though success rates remain modest [2] . Researchers require clear actionable information to help them decide whether to resubmit a rejected grant application [7] , without which they may face multiple rejections from the same competition [8] . For NIH competitions, quantitative results (e.g.…”