Individual versus general structured feedback to improve agreement in grant peer review: A randomized controlled trial

Abstract

BackgroundVast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving a short individual feedback report compared to a short general feedback report on the agreement between reviewers.MethodsA total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same application.ResultsA total of 2398 paired reviews were included in the analysis. There was no difference between the two groups in terms of the average difference. There was a significant difference between the two groups in the proportion of absolute agreement on whether the application was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. However, the overall levels of agreement remained critically low in both groups and for both outcomes.ConclusionsIndividual feedback did not improve agreement between reviewers. This finding is in line with related studies of journal peer review. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.Trial registrationThe study was preregistered at OSF.io/n4fq3.