Automated and flexible identification of complex disease: building a model for systemic lupus erythematosus using noisy labeling

Murray, Sara G.; Avati, Anand; Schmajuk, Gabriela; Yazdany, Jinoos

doi:10.1093/jamia/ocy154

Cited by 44 publications

(27 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, computational methods that allow more granular phenotype extraction from EHRs are advancing rapidly. For example, in a recent study, we applied an artificial intelligence algorithm to recognize patterns of lupus and assign probabilities of disease . Work is ongoing to scale such algorithms in repositories like the RISE Registry to understand the full spectrum of phenotypes across a population, to track outcomes, and to conduct discovery research.…”

Section: Generating Real‐world Evidencementioning

confidence: 99%

Reimagining Rheumatology: Big Data and the Future of Clinical Practice and Research

Yazdany

Hannan²

2020

Arthritis Care & Research

Self Cite

View full text Add to dashboard Cite

In this issue of Arthritis Care & Research, we present another set of themed articles that are relevant to rheumatology clinical practice and research. Our themed issues are designed to spark interest and influence knowledge growth in rheumatology. The topic for this themed issue is the pertinent use of rheumatology registries, Big Data, and very large patient or administrative data sources to inform patient or individual aspects of key outcomes in the rheumatic diseases. Manuscripts representing a broad range of topics across the lifespan were considered, including treat-totarget, disease burden, costs, as well as other topics, using a wide variety of Big Data sources. Manuscripts submitted for the themed issues of Arthritis Care & Research undergo the same peer-review procedures as other scientific manuscripts in our journal, and therefore meet the same rigorous standards as articles in this or any other issue.The call for articles for this theme issue on Big Data resulted in 76 submissions. From these papers, we are proud to publish 12 articles covering important topics and issues in the rheumatic diseases, including osteoarthritis, rheumatoid arthritis (RA), lupus, juvenile dermatomyositis, and patient care costs. The tremendous response and the wide array of topics addressed by registries, administrative data, and large patient data sources furthered our thoughts on using Big Data in rheumatology and the expected exponential growth in our utilization of these data to increase our understanding of rheumatic diseases. Big Data is an exciting resource that captures nuance and detail, highlighting the evidence in evidence-based medicine for many aspects of disease management that have not been fully appreciated by past study designs or data collection.The volume of Big Data in health care is growing exponentially, outstripping data growth in other sectors and creating both new opportunities and significant challenges for medicine and research. Billions of dollars have been invested in health information technology (IT) infrastructure. Electronic health records (EHRs) have fundamentally changed the work of physicians and health systems. To be a rheumatologist today means spending as many or more hours interfacing with EHRs as with patients. These seismic shifts in medicine have been accompanied by promises of safer, better, and even more cost-efficient care, but to a large extent these promises remain unrealized. Yet, there is reason to be hopeful. Here we discuss the ways the field of rheumatology can harness Big Data to advance our specialty and improve health outcomes for people with rheumatic diseases.What is Big Data? Big Data typically refers to databases or registries with high volume, rapid velocity, much variety, and high veracity. These 4 "Vs" characterize the vast amounts of data that can be aggregated from often disparate sources for analysis. The American College of Rheumatology's RISE (Rheumatology Informatics System for Effectiveness) Registry, a focus for one of the themed articles, is a great exa...

show abstract

Section: Generating Real‐world Evidencementioning

confidence: 99%

Reimagining Rheumatology: Big Data and the Future of Clinical Practice and Research

Yazdany

Hannan²

2020

Arthritis Care & Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…To address the scarcity of labeled training data, Chen et al used active learning to intelligently select training samples for labeling, demonstrating that classifier performance could be preserved with fewer samples [13]. Another trend is the use of "silver standard training sets", a semi-supervised approach where training samples are labeled using an imperfect heuristic rather than by manual review [14][15][16][17][18][19]. The intuition is that noise-tolerant classifiers trained on imperfectly labeled data will abstract higher order properties of the phenotype beyond the original labeling heuristic (so-called noise-tolerant learning [20]).…”

Section: Background and Significancementioning

confidence: 99%

Development and Validation of Phenotype Classifiers across Multiple Sites in the Observational Health Sciences and Informatics (OHDSI) Network

Kashyap

Seneviratne

Banda

et al. 2019

Preprint

View full text Add to dashboard Cite

Objective: Accurate electronic phenotyping is essential to support collaborative observational research. Supervised machine learning methods can be used to train phenotype classifiers in a high-throughput manner using imperfectly labeled data. We developed ten phenotype classifiers using this approach and evaluated performance across multiple sites within the Observational Health Sciences and Informatics (OHDSI) network. Materials and Methods:We constructed classifiers using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) R-package, an open-source framework for learning phenotype classifiers using datasets in the OMOP CDM.We labeled training data based on the presence of multiple mentions of disease-specific codes.Performance was evaluated on cohorts derived using rule-based definitions and real-world disease prevalence. Classifiers were developed and evaluated across three medical centers, including one international site.Results: Compared to the multiple mentions labeling heuristic, classifiers showed a mean recall boost of 0.43 with a mean precision loss of 0.17. Performance decreased slightly when classifiers were shared across medical centers, with mean recall and precision decreasing by 0.08 and 0.01, respectively, at a site within the USA, and by 0.18 and 0.10, respectively, at an international site. Discussion and Conclusion:We demonstrate a high-throughput pipeline for constructing and sharing phenotype classifiers across multiple sites within the OHDSI network using APHRODITE. Classifiers exhibit good portability between sites within the USA, however limited portability internationally, indicating that classifier generalizability may have geographic limitations, and consequently, sharing the classifier-building recipe, rather than the pre-trained classifiers, may be more useful for facilitating collaborative observational research.

show abstract

“…12,13 Additionally, while case and control phenotyping using EHR data has also relied on a small number of expert curated cohorts, recent studies have demonstrated that ML approaches can expand upon and identify such cohorts using automated feature selection and imperfect case de nitions in a high-throughput manner. [14][15][16][17][18] Studies have also shown that case and control selection with diagnosis codes can signi cantly affect model performance, the hierarchical organization of structured medical data can be utilized for feature reduction and model performance improvement, and calibration is essential for understanding the clinical utility of a phenotyping model. [19][20][21][22] Stroke phenotyping algorithms have also used machine learning to enhance the classi cation performance of a diagnosis-code based AIS phenotyping algorithm.…”

Section: Introductionmentioning

confidence: 99%

Comparative analysis, applications, and interpretation of electronic health record-based stroke phenotyping methods

Thangaraj

Kummer

Lorberbaum

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Accurate identification of acute ischemic stroke (AIS) patient cohorts is essential for a wide range of clinical investigations. Automated phenotyping methods that leverage electronic health records (EHRs) represent a fundamentally new approach cohort identification without current laborious and ungeneralizable generation of phenotyping algorithms. We systematically compared and evaluated the ability of machine learning algorithms and case-control combinations to phenotype acute ischemic stroke patients using data from an EHR.Materials and Methods: Using structured patient data from the EHR at a tertiary-care hospital system, we built and evaluated machine learning models to identify patients with AIS based on 75 different case-control and classifier combinations. We then estimated the prevalence of AIS patients across the EHR. Finally, we externally validated the ability of the models to detect AIS patients without AIS diagnosis codes using the UK Biobank.Results: Across all models, we found that the mean AUROC for detecting AIS was 0.963±0.0520 and average precision score 0.790±0.196 with minimal feature processing. Classifiers trained with cases with AIS diagnosis codes and controls with no cerebrovascular disease codes had the best average F1 score (0.832±0.0383). In the external validation, we found that the top probabilities from a model-predicted AIS cohort were significantly enriched for AIS patients without AIS diagnosis codes (60-150 fold over expected). Conclusions: Our findings support machine learning algorithms as a generalizable way to accurately identify AIS patients without using process-intensive manual feature curation. When a set of AIS patients is unavailable, diagnosis codes may be used to train classifier models.

show abstract

Automated and flexible identification of complex disease: building a model for systemic lupus erythematosus using noisy labeling

Cited by 44 publications

References 9 publications

Reimagining Rheumatology: Big Data and the Future of Clinical Practice and Research

Reimagining Rheumatology: Big Data and the Future of Clinical Practice and Research

Development and Validation of Phenotype Classifiers across Multiple Sites in the Observational Health Sciences and Informatics (OHDSI) Network

Comparative analysis, applications, and interpretation of electronic health record-based stroke phenotyping methods

Contact Info

Product

Resources

About