Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT

Mortensen, Jonathan; Minty, Evan; Januszyk, Michael; Sweeney, Timothy E.; Rector, Alan L.; Noy, Natalya F.; Musen, Mark A.

doi:10.1136/amiajnl-2014-002901

Cited by 50 publications

(50 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Crowdsourcing data annotation and curation in bioinformatics can be handled well with this approach. This scheme has also been applied to provide pathway resources 68,69 , reconstruct the human metabolic network 70 , annotate molecular interactions in Mycobacterium tuberculosis 71 and identify crucial errors in ontologies 72 .…”

Section: Figurementioning

confidence: 99%

Crowdsourcing biomedical research: leveraging communities as innovation engines

et al. 2016

View full text Add to dashboard Cite

The generation of large-scale biomedical data is creating unprecedented opportunities for basic and translational science. Typically, the data producers perform initial analyses, but it is very likely that the most informative methods may reside with other groups. Crowdsourcing the analysis of complex and massive data has emerged as a framework to find robust methodologies. When the crowdsourcing is done in the form of collaborative scientific competitions, known as Challenges, the validation of the methods is inherently addressed. Challenges also encourage open innovation, create collaborative communities to solve diverse and important biomedical problems, and foster the creation and dissemination of well-curated data repositories.

show abstract

Section: Figurementioning

confidence: 99%

Crowdsourcing biomedical research: leveraging communities as innovation engines

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Recent research has shown that AMT and other crowdsourcing platforms can be used to generate corpora for clinical natural language processing and disease mention annotation [41,42]. AMT was used to detect errors in a medical ontology, and it was found that the crowd was as effective as the domain experts [43]. In addition, AMT workers were engaged in identifying disease mentions in PubMed abstracts [42] and rank adverse drug reactions in order of severity [44] with good results.…”

Section: Methodsmentioning

confidence: 99%

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation

Lalor¹,

Wu²,

Chen³

et al. 2018

J Med Internet Res

View full text Add to dashboard Cite

BackgroundPatient portals are widely adopted in the United States and allow millions of patients access to their electronic health records (EHRs), including their EHR clinical notes. A patient’s ability to understand the information in the EHR is dependent on their overall health literacy. Although many tests of health literacy exist, none specifically focuses on EHR note comprehension.ObjectiveThe aim of this paper was to develop an instrument to assess patients’ EHR note comprehension.MethodsWe identified 6 common diseases or conditions (heart failure, diabetes, cancer, hypertension, chronic obstructive pulmonary disease, and liver failure) and selected 5 representative EHR notes for each disease or condition. One note that did not contain natural language text was removed. Questions were generated from these notes using Sentence Verification Technique and were analyzed using item response theory (IRT) to identify a set of questions that represent a good test of ability for EHR note comprehension.ResultsUsing Sentence Verification Technique, 154 questions were generated from the 29 EHR notes initially obtained. Of these, 83 were manually selected for inclusion in the Amazon Mechanical Turk crowdsourcing tasks and 55 were ultimately retained following IRT analysis. A follow-up validation with a second Amazon Mechanical Turk task and IRT analysis confirmed that the 55 questions test a latent ability dimension for EHR note comprehension. A short test of 14 items was created along with the 55-item test.ConclusionsWe developed ComprehENotes, an instrument for assessing EHR note comprehension from existing EHR notes, gathered responses using crowdsourcing, and used IRT to analyze those responses, thus resulting in a set of questions to measure EHR note comprehension. Crowdsourced responses from Amazon Mechanical Turk can be used to estimate item parameters and select a subset of items for inclusion in the test set using IRT. The final set of questions is the first test of EHR note comprehension.

show abstract

“…These are large scale medical taxonomies which have been exploited in modern clinical systems showing significant good results in the targeted clinical systems. In Mortensen et al (2014) it shows that the clinicians using healthcare systems equipped with SNOMED outperformed clinicians using conventional systems without SNOMED CT capabilities.…”

Section: Ontology Driven Clinical Decision Support Frameworkmentioning

confidence: 98%

A novel ontology and machine learning driven hybrid cardiovascular clinical prognosis as a complex adaptive clinical system

Farooq

Hussain

2016

Complex Adapt Syst Model

View full text Add to dashboard Cite

Purpose: This multidisciplinary industrial research project sets out to develop a hybrid clinical decision support mechanism (inspired by ontology and machine learning driven techniques) by combining evidence, extrapolated through legacy patient data to facilitate cardiovascular preventative care. Methods:The proposed cardiovascular clinical decision support framework comprises of two novel key components:(1) Ontology driven clinical risk assessment and recommendation system (ODCRARS) (2) Machine learning driven prognostic system (MLDPS). State of the art machine learning and feature selection methods are utilised for the prognostic modelling purposes. The ODCRARS is a knowledge-based system which is based on clinical expert's knowledge, encoded in the form of clinical rules engine to carry out cardiac risk assessment for various cardiovascular diseases. The MLDPS is a non knowledge-based/data driven system which is developed using state of the art machine learning and feature selection techniques applied on real patient datasets. Clinical case studies in the RACPC, heart disease and breast cancer domains are considered for the development and clinical validation purposes. For the purpose of this paper, clinical case study in the RACPC/chest pain domain will be discussed in detail from the development and validation perspective. Results:The proposed clinical decision support framework is validated through clinical case studies in the cardiovascular domain. This paper demonstrates an effective cardiovascular decision support mechanism for handling inaccuracies in the clinical risk assessment of chest pain patients and help clinicians effectively distinguish acute angina/cardiac chest pain patients from those with other causes of chest pain. Conclusion:The new clinical models, having been evaluated in clinical practice, resulted in very good predictive power, demonstrating general performance improvement over benchmark multivariate statistical classifiers. Various chest pain risk assessment prototypes have been developed and deployed online for further clinical trials.

show abstract

Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT

Abstract: We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.

Cited by 50 publications

References 18 publications

Crowdsourcing biomedical research: leveraging communities as innovation engines

Crowdsourcing biomedical research: leveraging communities as innovation engines

ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation

A novel ontology and machine learning driven hybrid cardiovascular clinical prognosis as a complex adaptive clinical system

Contact Info

Product

Resources

About