How to best evaluate applications for junior fellowships? Remote evaluation and face-to-face panel meetings compared

Bieri, Marco; Roser, Katharina; Heyard, Rachel; Egger, Matthias

doi:10.1101/2020.11.26.400028

Cited by 1 publication

(2 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of note, in the context of the Explorer Grant scheme of the New Zealand Health Research Council, Liu et al (2020) recently reported that most applicants agreed with random selection. So far, the SNSF received no negative or positive reactions to the use of random selection from applicants (Bieri et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

“…The Health Research Council of New Zealand (Liu et al, 2020), the Volkswagen Foundation (2017) and recently the Austrian Research Fund (2020) are funders that have applied lotteries, with a focus on transformative research or unconventional research ideas. The SNSF introduced a modified lottery for its junior fellowship scheme, focusing on proposals near the funding line (Adam, 2019;Bieri et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Rethinking the Funding Line at the Swiss National Science Foundation: Bayesian Ranking and Lottery

Heyard¹,

Ott²,

Salanti³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Funding agencies rely on peer review and expert panels to select the research deserving funding. Peer review has limitations, including bias against risky proposals or interdisciplinary research. The inter-rater reliability between reviewers and panels is low, particularly for proposals near the funding line. Funding agencies are increasingly acknowledging the role of chance. The Swiss National Science Foundation (SNSF) introduced a lottery for proposals in the middle group of good but not excellent proposals. In this article, we introduce a Bayesian hierarchical model for the evaluation process. To rank the proposals, we estimate their expected ranks (ER), which incorporates both the magnitude and uncertainty of the estimated differences between proposals. A provisional funding line is defined based on ER and budget. The ER and its credible interval are used to identify proposals with similar quality and credible intervals that overlap with the funding line. These proposals are entered into a lottery. We illustrate the approach for two SNSF grant schemes in career and project funding. We argue that the method could reduce bias in the evaluation process. R code, data and other materials for this article are available online.

show abstract

Section: Discussionmentioning

confidence: 99%