Streamlined research funding using short proposals and accelerated peer review: an observational study

Barnett, Adrian; Herbert, Danielle L.; Campbell, Megan; Daly, Naomi; Roberts, Jason A.; Mudge, Alison; Graves, Nicholas

doi:10.1186/s12913-015-0721-7

Cited by 27 publications

(24 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recent work by Graves et al (2011) used a small survey of NHMRC researchers to estimate that the burden fell even more heavily on the applicants, assigning a split of 85 per cent for application production, 9 per cent for reviewing and 5 per cent for administration. Barnett et al (2015) reinforced this conclusion with a larger survey of 285 applicants who had submitted 632 proposals to four health services research funding rounds from May 2012 to November 2013, at the Australian Centre for Health Services Innovation. A review by the New Zealand Royal Society made a similar estimate of the burden shouldered by the applicants – pegging it at 80 per cent ( Gluckman, 2012 ).…”

Section: Resultsmentioning

confidence: 91%

What do we know about grant peer review in the health sciences?

2018

View full text Add to dashboard Cite

Background: Peer review decisions award an estimated >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members. Conclusions: We suggest funders should acknowledge, assess and analyse the uncertainty around peer review, even using reviewers’ uncertainty as an input to funding decisions. Funders could consider a lottery element in some parts of their funding allocation process, to reduce both burden and bias, and allow better evaluation of decision processes. Alternatively, the distribution of scores from different reviewers could be better utilised as a possible way to identify novel, innovative research. Above all, there is a need for open, transparent experimentation and evaluation of different ways to fund research. This also requires more openness across the wider scientific community to support such investigations, acknowledging the lack of evidence about the primacy of the current system and the impossibility of achieving perfection.

show abstract

Section: Resultsmentioning

confidence: 91%

What do we know about grant peer review in the health sciences?

2018

View full text Add to dashboard Cite

show abstract

“…There is scope to improve the amount and quality of such feedback given the amount of time and effort that goes into reviewing each proposal. Such feedback can be very helpful for applicants ( Barnett et al , 2015 ) and grant-making bodies may want to consider how they could improve the quality of feedback for example, by providing audio transcripts of panel meetings.…”

Section: Discussionmentioning

confidence: 99%

The peer review process for awarding funds to international science research consortia: a qualitative developmental evaluation

et al. 2018

View full text Add to dashboard Cite

Evaluating applications for multi-national, multi-disciplinary, Background: dual-purpose research consortia is highly complex. There has been little research on the peer review process for evaluating grant applications and almost none on how applications for multi-national consortia are reviewed. Overseas development investments are increasingly being channelled into international science consortia to generate high-quality research while simultaneously strengthening multi-disciplinary research capacity. We need a better understanding of how such decisions are made and their effectiveness.An award-making institution planned to fund 10 UK-Africa research Methods: consortia. Over two annual rounds, 34 out of 78 eligible applications were shortlisted and reviewed by at least five external reviewers before final selections were made by a face-to-face panel. We used an innovative approach involving structured, overt observations of award-making panel meetings and semi-structured interviews with panel members to explore how assessment criteria concerning research quality and capacity strengthening were applied during the peer review process. Data were coded and analysed using pre-designed matrices which incorporated categories relating to the assessment criteria.In general the process was rigorous and well-managed. However, Results: lack of clarity about differential weighting of criteria and variations in the panel's understanding of research capacity strengthening resulted in some inconsistencies in use of the assessment criteria. Using the same panel for both rounds had advantages, in that during the second round consensus was achieved more quickly and the panel had increased focus on development aspects.Grant assessment panels for such complex research applications Conclusion: need to have topic-and context-specific expertise. They must also understand research capacity issues and have a flexible but equitable and transparent approach. This study has developed and tested an approach for evaluating the operation of such panels and has generated lessons that can promote coherence and transparency among grant-makers and ultimately make the award-making process more effective.

show abstract

“…We did not systematically collect feedback on the change to a two-stage process however, anecdotally, applicants' feedback on the change has been positive. This research adds to the existing evidence base on peer review processes [3][4][5][6][7]11] that shows there are ways to manage funding processes in a more efficient manner that is not detrimental to the quality of the research funded. Research funders should regularly review their funding processes to ensure that they are fit for purpose and reduce burden, and waste, to those involved in the processes.…”

Section: Plos Onementioning

confidence: 93%

“…The National Health and Medical Research Council (NHMRC) of Australia has conducted a number of studies looking at different ways to assess research proposals including simplified processes with the potential to save time and money without detracting from the rigour of the process. The NHMRC also showed that a more streamlined application process with accelerated peer review saved time for applicants and peer reviewers [5]. The National Science Foundation (NSF) adopted a twostage application process which increased the total number of applications.…”

Section: Introductionmentioning

confidence: 99%

Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process

Morgan¹,

Yu²,

Solomon³

et al. 2020

PLoS ONE

View full text Add to dashboard Cite

Background Research funders use a wide variety of application assessment processes yet there is little evidence on their relative advantages and disadvantages. A broad distinction can be made between processes with a single stage assessment of full proposals and those that first invite an outline, with full proposals invited at a second stage only for those which are shortlisted. This paper examines the effects of changing from a one-stage to a two-stage process within the UK's National Institute for Health Research's (NIHR) Research for Patient Benefit (RfPB) Programme which made this change in 2015. Methods A retrospective comparative design was used to compare eight one-stage funding competitions (912 applications) with eight two-stage funding competitions (1090 applications). Comparisons were made between the number of applications submitted, number of peer and lay reviews required, the duration of the funding round, average external peer review scores, and the total costs involved. Results There was a mean number of 114 applications per funding round for the one-stage process and 136 for the two-stage process. The one-stage process took a mean of 274 days and the two-stage process 348 days to complete, although those who were not funded (i.e. the majority) were informed at a mean of 195 days (mean 79 days earlier) under the two-stage process. The mean peer review score for full applications using the one-stage process was 6.46 and for the two-stage process 6.82 (5.6% difference using a 1-10 scale (with 10 being the highest), but there was no significant difference between the lay reviewer scores. The one-stage process required a mean of 423 peer reviews and 102 lay reviewers and the twostage process required a mean of 208 peer reviews and 50 lay reviews (mean difference of

show abstract

Streamlined research funding using short proposals and accelerated peer review: an observational study

Cited by 27 publications

References 4 publications

What do we know about grant peer review in the health sciences?

What do we know about grant peer review in the health sciences?

The peer review process for awarding funds to international science research consortia: a qualitative developmental evaluation

Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process

Contact Info

Product

Resources

About