An efficient system to fund science: from proposal review to peer-to-peer distributions

Bollen, Johan; Crandall, David J.; Junk, Damion; Ding, Ying; Börner, Katy

doi:10.1007/s11192-016-2110-3

Cited by 42 publications

(29 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Yaqub found that it is possible to classify serendipity into four basic types [66] and that there may be important factors affecting its occurrence. His conclusions seem to agree with ideas developed in earlier works [67,68,69,70,71,72] which argues that the commonly adopted -apparently meritocratic -strategies, which pursuit excellence and drive out variety, seem destined to be loosing and inefficient. The reason is that they cut out a priori researches that initially appear less promising but that, thanks also to serendipity, could be extremely innovative a posteriori.…”

Section: Serendipity Innovation and Efficient Funding Strategiessupporting

confidence: 84%

Talent Versus Luck: The Role of Randomness in Success and Failure

Pluchino

Biondo

Rapisarda

2018

Advs. Complex Syst.

101

View full text Add to dashboard Cite

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, smartness, efforts, willfulness, hard work or risk taking. Sometimes, we are willing to admit that a certain degree of luck could also play a role in achieving significant material success. But, as a matter of fact, it is rather common to underestimate the importance of external forces in individual successful stories. It is very well known that intelligence (or, more in general, talent and personal qualities) exhibits a Gaussian distribution among the population, whereas the distribution of wealth -often considered a proxy of success -follows typically a power law (Pareto law), with a large majority of poor people and a very small number of billionaires. Such a discrepancy between a Normal distribution of inputs, with a typical scale (the average talent or intelligence), and the scale invariant distribution of outputs, suggests that some hidden ingredient is at work behind the scenes. In this paper, with the help of a very simple agent-based toy model, we suggest that such an ingredient is just randomness. In particular, we show that, if it is true that some degree of talent is necessary to be successful in life, almost never the most talented people reach the highest peaks of success, being overtaken by mediocre but sensibly luckier individuals. As to our knowledge, this counterintuitive result -although implicitly suggested between the lines in a vast literature -is quantified here for the first time. It sheds new light on the effectiveness of assessing merit on the basis of the reached level of success and underlines the risks of distributing excessive honors or resources to people who, at the end of the day, could have been simply luckier than others. With the help of this model, several policy hypotheses are also addressed and compared to show the most efficient strategies for public funding of research in order to improve meritocracy, diversity and innovation.

show abstract

Section: Serendipity Innovation and Efficient Funding Strategiessupporting

confidence: 84%

Talent Versus Luck: The Role of Randomness in Success and Failure

Pluchino

Biondo

Rapisarda

2018

Advs. Complex Syst.

101

View full text Add to dashboard Cite

show abstract

“…The current peer review system is subject to biases and inconsistencies (96). Several alternatives have been proposed, such as the random distribution of funding (97), person-directed funding that does not involve proposal preparation and review (31), opening the proposal review process to the entire online population (98), removing human reviewers altogether by allocating funds through a performance measure (99), and scientist crowd-funding (100). …”

Section: Discussionmentioning

confidence: 99%

Science of science

Fortunato

Bergstrom

Börner

et al. 2018

Science

Self Cite

1,000

709

View full text Add to dashboard Cite

BACKGROUND The increasing availability of digital data on scholarly inputs and outputs—from research funding, productivity, and collaboration to paper citations and scientist mobility—offers unprecedented opportunities to explore the structure and evolution of science. The science of science (SciSci) offers a quantitative understanding of the interactions among scientific agents across diverse geographic and temporal scales: It provides insights into the conditions underlying creativity and the genesis of scientific discovery, with the ultimate goal of developing tools and policies that have the potential to accelerate science. In the past decade, SciSci has benefited from an influx of natural, computational, and social scientists who together have developed big data–based capabilities for empirical analysis and generative modeling that capture the unfolding of science, its institutions, and its workforce. The value proposition of SciSci is that with a deeper understanding of the factors that drive successful science, we can more effectively address environmental, societal, and technological problems. ADVANCES Science can be described as a complex, self-organizing, and evolving network of scholars, projects, papers, and ideas. This representation has unveiled patterns characterizing the emergence of new scientific fields through the study of collaboration networks and the path of impactful discoveries through the study of citation networks. Microscopic models have traced the dynamics of citation accumulation, allowing us to predict the future impact of individual papers. SciSci has revealed choices and trade-offs that scientists face as they advance both their own careers and the scientific horizon. For example, measurements indicate that scholars are risk-averse, preferring to study topics related to their current expertise, which constrains the potential of future discoveries. Those willing to break this pattern engage in riskier careers but become more likely to make major breakthroughs. Overall, the highest-impact science is grounded in conventional combinations of prior work but features unusual combinations. Last, as the locus of research is shifting into teams, SciSci is increasingly focused on the impact of team research, finding that small teams tend to disrupt science and technology with new ideas drawing on older and less prevalent ones. In contrast, large teams tend to develop recent, popular ideas, obtaining high, but often short-lived, impact. OUTLOOK SciSci offers a deep quantitative understanding of the relational structure between scientists, institutions, and ideas because it facilitates the identification of fundamental mechanisms responsible for scientific discovery. These interdisciplinary data-driven efforts complement contributions from related fields such as sciento-metrics and the economics and sociology of science. Although SciSci seeks long-standing universal laws and mechanisms that apply across various fields of science, a fundamental challenge going forward is accounting for un...

show abstract

“…I was surprised that some of the more innovative solutions to funding peer review were not included, specifically using prediction markets 1 and using the “wisdom of the crowd” 2 .…”

mentioning

confidence: 99%

What do we know about grant peer review in the health sciences?

2017

View full text Add to dashboard Cite

Background: Peer review decisions award >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 relevant papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the overall efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members. Conclusions: We suggest funders should acknowledge, assess and analyse the uncertainty around peer review, even using reviewers' uncertainty as an input to funding decisions. Funders could consider a lottery element in some parts of their funding allocation process, to reduce both burden and bias, and allow better evaluation of decision processes. Alternatively, the distribution of scores from different reviewers could be better utilised as a possible way to identify novel, innovative research. Above all, there is a need for open, transparent experimentation and evaluation of different ways to fund research. This also requires more openness across the wider scientific community to support such investigations, acknowledging the lack of evidence about the primacy of the current system and the impossibility of achieving perfection.

show abstract

An efficient system to fund science: from proposal review to peer-to-peer distributions

Cited by 42 publications

References 28 publications

Talent Versus Luck: The Role of Randomness in Success and Failure

Talent Versus Luck: The Role of Randomness in Success and Failure

Science of science

What do we know about grant peer review in the health sciences?

Contact Info

Product

Resources

About