Seeking the Wisdom of Crowds Through Challenge-Based Competitions in Biomedical Research

Costello, James C.; Stolovitzky, Gustavo

doi:10.1038/clpt.2013.36

Cited by 46 publications

(33 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…15) won a third place prize of $10,000. As in previous installments of the DREAM challenges 16 , the aggregation of predictions across teams 1 and 2 further reduced the prediction error. Bootstrapping further allowed us to estimate that a statistically significant improvement over the baseline algorithm would correspond to an r.m.s.…”

Section: Methods Performance and Assessmentmentioning

confidence: 70%

Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression

et al. 2014

View full text Add to dashboard Cite

Amyotrophic lateral sclerosis (ALS) is a fatal neurodegenerative disease with substantial heterogeneity in its clinical presentation. This makes diagnosis and effective treatment difficult, so better tools for estimating disease progression are needed. Here, we report results from the DREAM-Phil Bowen ALS Prediction Prize4Life challenge. In this crowdsourcing competition, competitors developed algorithms for the prediction of disease progression of 1,822 ALS patients from standardized, anonymized phase 2/3 clinical trials. The two best algorithms outperformed a method designed by the challenge organizers as well as predictions by ALS clinicians. We estimate that using both winning algorithms in future trial designs could reduce the required number of patients by at least 20%. The DREAM-Phil Bowen ALS Prediction Prize4Life challenge also identified several potential nonstandard predictors of disease progression including uric acid, creatinine and surprisingly, blood pressure, shedding light on ALS pathobiology. This analysis reveals the potential of a crowdsourcing competition that uses clinical trial data for accelerating ALS research and development

show abstract

Section: Methods Performance and Assessmentmentioning

confidence: 70%

Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Although the idea is not new 11 , the current practice of crowdsourcing is truly a product of our times in that it leverages the prompt feedback, ease of access, communication and a participatory culture that is fuelled by the Internet. Over the past 20 years, crowdsourcing has developed rapidly across many academic 12 and commercial 13 initiatives. In the context of biomedical research, many initiatives have developed in areas ranging from protein structure prediction to disease prognosis 12 .…”

mentioning

confidence: 99%

“…Over the past 20 years, crowdsourcing has developed rapidly across many academic 12 and commercial 13 initiatives. In the context of biomedical research, many initiatives have developed in areas ranging from protein structure prediction to disease prognosis 12 .…”

mentioning

confidence: 99%

Crowdsourcing biomedical research: leveraging communities as innovation engines

et al. 2016

View full text Add to dashboard Cite

The generation of large-scale biomedical data is creating unprecedented opportunities for basic and translational science. Typically, the data producers perform initial analyses, but it is very likely that the most informative methods may reside with other groups. Crowdsourcing the analysis of complex and massive data has emerged as a framework to find robust methodologies. When the crowdsourcing is done in the form of collaborative scientific competitions, known as Challenges, the validation of the methods is inherently addressed. Challenges also encourage open innovation, create collaborative communities to solve diverse and important biomedical problems, and foster the creation and dissemination of well-curated data repositories.

show abstract

“…6 Projects in the biomedical domain include protein structure prediction (CASP) 12 18 and Kaggle (www.kaggle.com) have evolved to harness the power of crowdsourcing. In this subsection we review the scoring approaches of crowdsourcing efforts pre-dating IMPROVER, focusing largely on CAPRI, CASP, and DREAM, as their scoring methodology and rationale have been previously documented.…”

Section: Scoring Strategies In Collaborative-competitionsmentioning

confidence: 99%