Validating Candidate Gene-Mutation Relations in MEDLINE Abstracts via Crowdsourcing

Burger, John D.; Doughty, Emily; Bayer, Sam; Tresner-Kirsch, David; Wellner, Ben; Aberdeen, John S.; Lee, Kyungjoon; Kann, Maricel G.; Hirschman, Lynette

doi:10.1007/978-3-642-31040-9_8

Cited by 8 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The average F score of workers that completed a hundred or more abstracts was 0.791 with a standard deviation of 0.066. This result generally reiterates the findings in [10], which observed a clear increase in performance for workers that completed more than 100 of their gene-mutation validation tasks. However, the correlation between the number of tasks performed and the average quality of the work was not statistically significant in this experiment.…”

Section: Resultssupporting

confidence: 89%

“…Two early studies demonstrated that a gold standard corpus of annotated clinical trials documents could be assembled through microtask crowdsourcing [8, 9]. Recently, others have successfully applied microtasking to validate predicted gene-mutation relations in PubMed abstracts [10] and for medical relation extraction [11]. Here we extend these efforts by implementing and testing a disease mention annotation task on PubMed abstracts using the AMT platform.…”

Section: Introductionmentioning

confidence: 94%

See 1 more Smart Citation

Microtask Crowdsourcing for Disease Mention Annotation in Pubmed Abstracts

Good

Nanis

et al. 2014

Biocomputing 2015

View full text Add to dashboard Cite

Identifying concepts and relationships in biomedical text enables knowledge to be applied in computational analyses. Many biological natural language process (BioNLP) projects attempt to address this challenge, but the state of the art in BioNLP still leaves much room for improvement. Progress in BioNLP research depends on large, annotated corpora for evaluating information extraction systems and training machine learning models. Traditionally, such corpora are created by small numbers of expert annotators often working over extended periods of time. Recent studies have shown that workers on microtask crowdsourcing platforms such as Amazon's Mechanical Turk (AMT) can, in aggregate, generate high-quality annotations of biomedical text. Here, we investigated the use of the AMT in capturing disease mentions in PubMed abstracts. We used the NCBI Disease corpus as a gold standard for refining and benchmarking our crowdsourcing protocol. After several iterations, we arrived at a protocol that reproduced the annotations of the 593 documents in the 'training set' of this gold standard with an overall F measure of 0.872 (precision 0.862, recall 0.883). The output can also be tuned to optimize for precision (max = 0.984 when recall = 0.269) or recall (max = 0.980 when precision = 0.436). Each document was examined by 15 workers, and their annotations were merged based on a simple voting method. In total 145 workers combined to complete all 593 documents in the span of 1 week at a cost of $.06 per abstract per worker. The quality of the annotations, as judged with the F measure, increases with the number of workers assigned to each task such that the system can be tuned to balance cost against quality. These results demonstrate that microtask crowdsourcing can be a valuable tool for generating well-annotated corpora in BioNLP. Data produced for this analysis are available at

show abstract

Section: Resultssupporting

confidence: 89%

Section: Introductionmentioning

confidence: 94%

Microtask Crowdsourcing for Disease Mention Annotation in Pubmed Abstracts

Good

Nanis

et al. 2014

Biocomputing 2015

View full text Add to dashboard Cite

show abstract

“…However, in contrast to the general NLP domain, there are only a few studies involving crowdsourcing in biomedical NLP and almost none for clinical NLP. Most recently, Burger et al [19] performed a task of extracting the gene-mutation relations in Medical Literature Analysis and Retrieval System Online (MEDLINE) abstracts on AMT. In their work, candidate mutations were extracted from 250 MEDLINE abstracts using the Extractor of Mutations (EMU) presented together with the curated gene lists from the National Center for Biotechnology Information (NCBI).…”

Section: Introductionmentioning

confidence: 99%

Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language Processing

Zhai¹,

Lingren²,

Deléger³

et al. 2013

J Med Internet Res

View full text Add to dashboard Cite

BackgroundA high-quality gold standard is vital for supervised, machine learning-based, clinical natural language processing (NLP) systems. In clinical NLP projects, expert annotators traditionally create the gold standard. However, traditional annotation is expensive and time-consuming. To reduce the cost of annotation, general NLP projects have turned to crowdsourcing based on Web 2.0 technology, which involves submitting smaller subtasks to a coordinated marketplace of workers on the Internet. Many studies have been conducted in the area of crowdsourcing, but only a few have focused on tasks in the general NLP field and only a handful in the biomedical domain, usually based upon very small pilot sample sizes. In addition, the quality of the crowdsourced biomedical NLP corpora were never exceptional when compared to traditionally-developed gold standards. The previously reported results on medical named entity annotation task showed a 0.68 F-measure based agreement between crowdsourced and traditionally-developed corpora.ObjectiveBuilding upon previous work from the general crowdsourcing research, this study investigated the usability of crowdsourcing in the clinical NLP domain with special emphasis on achieving high agreement between crowdsourced and traditionally-developed corpora.MethodsTo build the gold standard for evaluating the crowdsourcing workers’ performance, 1042 clinical trial announcements (CTAs) from the ClinicalTrials.gov website were randomly selected and double annotated for medication names, medication types, and linked attributes. For the experiments, we used CrowdFlower, an Amazon Mechanical Turk-based crowdsourcing platform. We calculated sensitivity, precision, and F-measure to evaluate the quality of the crowd’s work and tested the statistical significance (P<.001, chi-square test) to detect differences between the crowdsourced and traditionally-developed annotations.ResultsThe agreement between the crowd’s annotations and the traditionally-generated corpora was high for: (1) annotations (0.87, F-measure for medication names; 0.73, medication types), (2) correction of previous annotations (0.90, medication names; 0.76, medication types), and excellent for (3) linking medications with their attributes (0.96). Simple voting provided the best judgment aggregation approach. There was no statistically significant difference between the crowd and traditionally-generated corpora. Our results showed a 27.9% improvement over previously reported results on medication named entity annotation task.ConclusionsThis study offers three contributions. First, we proved that crowdsourcing is a feasible, inexpensive, fast, and practical approach to collect high-quality annotations for clinical text (when protected health information was excluded). We believe that well-designed user interfaces and rigorous quality control strategy for entity annotation and linking were critical to the success of this work. Second, as a further contribution to the Internet-based crowdsourcing field, we will publicly release t...

show abstract

“…Building on results from an earlier experiment (22), we refined the framework and enriched the gold standard for easier evaluation.…”

Section: Methodsmentioning

confidence: 99%

Hybrid curation of gene–mutation relations combining automated extraction and crowdsourcing

et al. 2014

Self Cite

View full text Add to dashboard Cite

Background: This article describes capture of biological information using a hybrid approach that combines natural language processing to extract biological entities and crowdsourcing with annotators recruited via Amazon Mechanical Turk to judge correctness of candidate biological relations. These techniques were applied to extract gene– mutation relations from biomedical abstracts with the goal of supporting production scale capture of gene–mutation–disease findings as an open source resource for personalized medicine. Results: The hybrid system could be configured to provide good performance for gene–mutation extraction (precision ∼82%; recall ∼70% against an expert-generated gold standard) at a cost of $0.76 per abstract. This demonstrates that crowd labor platforms such as Amazon Mechanical Turk can be used to recruit quality annotators, even in an application requiring subject matter expertise; aggregated Turker judgments for gene–mutation relations exceeded 90% accuracy. Over half of the precision errors were due to mismatches against the gold standard hidden from annotator view (e.g. incorrect EntrezGene identifier or incorrect mutation position extracted), or incomplete task instructions (e.g. the need to exclude nonhuman mutations). Conclusions: The hybrid curation model provides a readily scalable cost-effective approach to curation, particularly if coupled with expert human review to filter precision errors. We plan to generalize the framework and make it available as open source software.Database URL: http://www.mitre.org/publications/technical-papers/hybrid-curation-of-gene-mutation-relations-combining-automated

show abstract

Validating Candidate Gene-Mutation Relations in MEDLINE Abstracts via Crowdsourcing

Cited by 8 publications

References 16 publications

Microtask Crowdsourcing for Disease Mention Annotation in Pubmed Abstracts

Microtask Crowdsourcing for Disease Mention Annotation in Pubmed Abstracts

Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language Processing

Hybrid curation of gene–mutation relations combining automated extraction and crowdsourcing

Contact Info

Product

Resources

About