Hybrid curation of gene–mutation relations combining automated extraction and crowdsourcing

Burger, John D.; Doughty, Emily; Khare, Ritu; Wei, Chih-Hsuan; Mishra, Rajashree; Aberdeen, John S.; Tresner-Kirsch, David; Wellner, Ben; Kann, Maricel G.; Lu, Zhiyong; Hirschman, Lynette

doi:10.1093/database/bau094

Cited by 39 publications

(38 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…National and international Research Infrastructures should play a central role to promote optimal data stewardship practices across the databases they support. Similarly, innovative curation models should emerge by combining the quality and richness of curation workfl ows, more cost-effective crowd-based triage, and the scalability of text mining instruments [ 58 ].…”

Section: Resultsmentioning

confidence: 99%

Text Mining to Support Gene Ontology Curation and Vice Versa

Ruch

2016

Methods in Molecular Biology

View full text Add to dashboard Cite

In this chapter, we explain how text mining can support the curation of molecular biology databases dealing with protein functions. We also show how curated data can play a disruptive role in the developments of text mining methods. We review a decade of efforts to improve the automatic assignment of Gene Ontology (GO) descriptors, the reference ontology for the characterization of genes and gene products. To illustrate the high potential of this approach, we compare the performances of an automatic text categorizer and show a large improvement of +225 % in both precision and recall on benchmarked data. We argue that automatic text categorization functions can ultimately be embedded into a Question-Answering (QA) system to answer questions related to protein functions. Because GO descriptors can be relatively long and specific, traditional QA systems cannot answer such questions. A new type of QA system, so-called Deep QA which uses machine learning methods trained with curated contents, is thus emerging. Finally, future advances of text mining instruments are directly dependent on the availability of high-quality annotated contents at every curation step. Databases workflows must start recording explicitly all the data they curate and ideally also some of the data they do not curate.

show abstract

Section: Resultsmentioning

confidence: 99%

Text Mining to Support Gene Ontology Curation and Vice Versa

Ruch

2016

Methods in Molecular Biology

View full text Add to dashboard Cite

show abstract

“…Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce utilize the 'power of the crowd' to perform discrete tasks [8], create content frameworks like Wikipedia [9], produce repositories of public data [10]. Khatib, et al, used crowdsourcing as a method to create an online game in which participants created an accurate model of a protein structure that had previously eluded scientists [11].…”

Section: Research Articlementioning

confidence: 99%

Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce

Kirsch¹,

Comstock²,

Harder³

et al. 2016

Arch Nurs Pract Care

View full text Add to dashboard Cite

Background: In the current healthcare environment, education for technical skills focuses on quality improvement that demands ongoing skill assessment. Objectively assessing competency is a complex task that, when done effectively, improves patient care. Current methods are time-consuming, expensive, and subjective. Crowdsourcing is the practice of obtaining services from a large group of people, typically the general public on an online community. CSATS (Crowd Sourced Assessment of Technical Skills) uses crowdsourcing as an innovative way to rapidly, objectively, and comprehensively assess technical skills. We hypothesized that CSATS could accurately evaluate the technical skill proficiency of nurses. Methods:An interface displaying one of 34 video-recorded nurses performing a glucometer skills test and a corresponding survey listing each required step were uploaded to an Amazon.com hosted crowdsourcing site, Mechanical Turk™. The crowd evaluated completion and sequence of the glucometer steps in each video. Results:In under 4 hours, we obtained 1,300 crowd ratings, approximately 38 per video that evaluated the user's performance based on completion and correct order of steps. The crowd identified individual performance variance, specific steps frequently missed by users, and provided feedback tailored to each user. CSATS identified 15% of nurses who would benefit from additional training. Conclusion:Our study showed that healthcare-naïve crowd workers can assess technical skill proficiency rapidly and accurately at nominal cost. CSATS may be a valuable tool to assist educators in creating targeted training curricula for nurses in need of follow up while rapidly identifying nurses whose technical skills meet expectations, thus, dramatically reducing the resource burden for training. myriad of challenges [2]. Assessing competency is time-intensive and expensive, and requires at least one certified nurse educator to teach/assess relatively simple nursing skills. In our study, over 1,400 nurses being trained at one hospital for a glucometer device received 15 minutes of didactic instruction and 5 minutes of hands on demonstration as part of seven 4-hour training sessions over the course of two months. That training period involved paying 1,400 nurses for their time and paying experts and evaluators to lead the training, all while mandating all nurses spend time in an education session rather than at the bedside. In addition to the costs and time, successful competency assessment in and of itself is challenging due to biased evaluators, inconsistent assessment tools, and no objective way to guarantee that those individuals who need more help receive it [3,4,6]. Some innovations have attempted to address the challenges of competency assessment, such as using video-based self-assessments to give participants and their evaluators accurate views of their performance [6]. The field of nursing and healthcare in general is in need of a competency assessment methodology that is reliable, fast, and affordable.Crowd ...

show abstract

“…Application of crowdsourcing in biology, particularly for data curation, is not new and comprises a variety of areas including: name entity recognition (NER) for drug and diseases (25-27), identification of medically-relevant terms from patient online posts (28), annotation of diseases described in PubMed (29), systematic examination of databases and other resources for drug indications, biomedical ontologies and gene-disease interactions (26,30-33), identification of the relationship between genes and mutations (34), and annotation of medical data for electronic health records (35). Interestingly, in most of the studies, crowdsourcing has proven to be as effective as expert curation (26,36).…”

Section: Introductionmentioning

confidence: 99%

ARGminer: A web platform for crowdsourcing-based curation of antibiotic resistance genes

Argoty

Guron

Garner

et al. 2018

Preprint

View full text Add to dashboard Cite

15Curation of antibiotic resistance gene (ARG) databases is a labor-intensive process that 16 requires expert knowledge to manually collect, correct, and/or annotate individual 17 genes. Correspondingly, updates to existing databases tend to be infrequent, commonly 18 requiring years for completion and often containing inconsistences. Further, because of 19 limitations of manual curation, most existing ARG databases contain only a small 20 proportion of known ARGs (~5k genes). A new approach is needed to achieve a truly 21 comprehensive ARG database, while also maintaining a high level of accuracy. Here we 22propose a new web-based curation system, ARG-miner, which supports annotation of 23ARGs at multiple levels, including: gene name, antibiotic category, resistance 24NC 4.0 International license peer-reviewed) is the author/funder. It is made available under aThe copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/274282 doi: bioRxiv preprint first posted online Mar. 1, 2018; 2 mechanism, and evidence for mobility and occurrence in clinically-important bacterial 25 strains. To overcome limitations of manual curation, we employ crowdsourcing as a 26 novel strategy for expanding curation capacity towards achieving a truly comprehensive, 27 up-to-date database.We develop and validate the approach by comparing 28 performance of multiple cohorts of curators with varying levels of expertise, 29 demonstrating that ARG-miner is more cost effective and less time-consuming relative 30 to traditional expert curation. We further demonstrate the reliability of a trust validation 31 filter for rejecting confounding input generated by spammers. Crowdsourcing was found 32 to be as accurate as expert annotation, with an accuracy >90% for the annotation of a 33 diverse test set of ARGs. ARG-miner provides a public API and database available at 34

show abstract

Hybrid curation of gene–mutation relations combining automated extraction and crowdsourcing

Cited by 39 publications

References 38 publications

Text Mining to Support Gene Ontology Curation and Vice Versa

Text Mining to Support Gene Ontology Curation and Vice Versa

Crowd Sourced Assessment of Technical Skills (CSATS): A Scalable Assessment Tool for the Nursing Workforce

ARGminer: A web platform for crowdsourcing-based curation of antibiotic resistance genes

Contact Info

Product

Resources

About