Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records

Adamson, Blythe; Waskom, Michael; Blarre, Auriane; Kelly, Jonathan; Krismer, Konstantin; Nemeth, Sheila; Gippetti, James; Ritten, John; Harrison, Katherine; Ho, George; Linzmayer, Robin; Bansal, Tarun; Wilkinson, Samuel; Amster, Guy; Estola, Evan; Benedum, Corey M.; Fidyk, Erin; Estévez, Melissa; Shapiro, Will; Cohen, Aaron B.

doi:10.1101/2023.03.02.23286522

Cited by 8 publications

(8 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each of the 18 variables has been extracted through NLP of clinical notes, followed by an advanced ML or deep learning model, including LSTM and XGBoost, after undergoing a rigorous development, validation, and testing process that aligns with the data and the model’s objectives. Model details, such as how they were developed, have been previously described [ 17 ]. Briefly, models are trained on the data labeled by expert abstraction to recognize, interpret, and curate free text into structured variable values in order to mimic the abstraction process.…”

Section: Methodsmentioning

confidence: 99%

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

Benedum

Sondhi

Fidyk

et al. 2023

Cancers

Self Cite

View full text Add to dashboard Cite

Meaningful real-world evidence (RWE) generation requires unstructured data found in electronic health records (EHRs) which are often missing from administrative claims; however, obtaining relevant data from unstructured EHR sources is resource-intensive. In response, researchers are using natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction) to extract real-world data (RWD) at scale. This study assessed the quality and fitness-for-use of EHR-derived oncology data curated using NLP with ML as compared to the reference standard of expert abstraction. Using a sample of 186,313 patients with lung cancer from a nationwide EHR-derived de-identified database, we performed a series of replication analyses demonstrating some common analyses conducted in retrospective observational research with complex EHR-derived data to generate evidence. Eligible patients were selected into biomarker- and treatment-defined cohorts, first with expert-abstracted then with ML-extracted data. We utilized the biomarker- and treatment-defined cohorts to perform analyses related to biomarker-associated survival and treatment comparative effectiveness, respectively. Across all analyses, the results differed by less than 8% between the data curation methods, and similar conclusions were reached. These results highlight that high-performance ML-extracted variables trained on expert-abstracted data can achieve similar results as when using abstracted data, unlocking the ability to perform oncology research at scale.

show abstract

Section: Methodsmentioning

confidence: 99%

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

Benedum

Sondhi

Fidyk

et al. 2023

Cancers

Self Cite

View full text Add to dashboard Cite

show abstract

“…to output a set of structured variables required for real-world data analysis. The authors concluded that NLP enabled the extraction of retrospective clinical data from the EMR faster and more efficiently [27]. The authors did not however directly compare their model to human data extractors - their purpose was to show success in building an AI model that extracts accurate information from the EMR.…”

Section: Discussionmentioning

confidence: 99%

“…To our knowledge, there are no other studies that directly compared free-text data extraction from the EMR between AI chatbots and human data extractors. Adamson et al [27] applied NLP to train, validate, and test the extraction of information from unstructured documents (e.g., clinician notes, radiology reports, lab reports, etc.) to output a set of structured variables required for real-world data analysis.…”

Section: Discussionmentioning

confidence: 99%

Using Artificial Intelligence To Label Free-Text Operative And Ultrasound Reports For Grading Pediatric Appendicitis

Abu-Ashour,

Emil,

Poenaru

2023

Preprint

View full text Add to dashboard Cite

Purpose Data science approaches personalizing pediatric appendicitis management are hampered by small datasets and unstructured electronic medical records (EMR). Artificial intelligence (AI) chatbots based on large language models (LLMs) can structure free-text EMR data. Here we compare data extraction quality between ChatGPT-4 and human data collectors. Methods To train AI models to grade pediatric appendicitis preoperatively, several data collectors (medical students and research assistants) extracted detailed preoperative and operative data from 2100 children operated for acute appendicitis between 2014-2021. Collectors were trained and certified for the task based on satisfactory Kappa scores. ChatGPT-4 was prompted to structure free text from 103 random anonymized ultrasound and operative records in the dataset using the set variables and coding options, and to estimate the Pediatric Appendicitis Grade (PAG) from the operative report. A pediatric surgeon then adjudicated all data, identifying errors in each method. Results Within the 44 ultrasound (42.7%) and 32 operative reports (31.1%) discordant in at least one field, 98% of the errors were found in the manual data extraction. The PAG was erroneously assigned manually in 29 patients (28.2%), and by ChatGPT-4 in 3 (2.9%). Across datasets, the use of the AI chatbot was able to avoid misclassification in 59.2% of the records including both reports and extracted data approx. 100x faster than manually. Conclusion An AI chatbot significantly outperformed manual data extraction in accuracy for ultrasound and operative reports, and correctly assigned the PAG score. While wider validation is required and data safety concerns must be fully addressed, these novel AI tools show significant promise in improving the accuracy and efficiency of research data collection.

show abstract

“…Retrospective longitudinal clinical data were derived from electronic health record data, comprising patient-level structured and unstructured data, curated via technology-enabled abstraction, and were linked to genomic data derived from FMI CGP tests in the FH-FMI CGDB by deidentified, deterministic matching . Patient smoking status was extracted by natural language processing of electronic health record documents . In this cohort, overall survival (OS) with routine clinical treatment was calculated from start of treatment in the metastatic setting to death from any cause, and patients without a record of mortality were right censored at the date of their last clinic visit or structured activity.…”

Section: Methodsmentioning

confidence: 99%

Genomic Profiles and Clinical Outcomes of Penile Squamous Cell Carcinoma With Elevated Tumor Mutational Burden

Necchi,

Spiess,

Costa de Padua

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceTumor mutational burden (TMB) is a putative biomarker of efficacy for immune checkpoint inhibitor (ICI) therapies of solid tumors, but not specifically for penile squamous cell carcinoma (PSCC).ObjectiveTo characterize biomarker features and ICI therapy outcomes associated with high TMB in PSCC in the routine clinical practice setting.Design, Setting, and ParticipantsIn this cohort study, 397 PSCC cases were analyzed to identify genomic alterations in more than 300 cancer-associated genes and genomic signatures, including TMB, using a hybrid capture–based comprehensive genomic profiling assay. Tumor mutational burden was categorized as low (&lt;10 mutations per megabase [mut/Mb]), high (10-19 mut/Mb), or very high (≥20 mut/Mb). Germline status of genetic alterations was predicted using a validated somatic-germline computational method. Clinical outcomes of patients with metastatic PSCC receiving first-line ICI were abstracted using the deidentified nationwide Clinico-Genomic Database (CGDB) from January 1, 2011, through December 31, 2022.ExposureComprehensive genomic profiling was performed using FoundationOne and FoundationOne CDx assays from Foundation Medicine Inc.Main outcomes and measuresThe spectrum of genetic alterations by TMB level in PSCC, the percentage of germline genetic alterations, and the outcome (overall survival with routine clinical treatment) by TMB of chemotherapy-naive patients with PSCC who received ICI treatment up front were assessed in this descriptive study.ResultsAmong 397 patients (median [IQR] age, 65 [54-73] years; 266 [67.0%] of European, 83 [20.9%] of admixed American, and 34 [8.5%] of African or other genomic ancestry), the median (IQR) age (eg, 65 [53-73] years for low TMB vs 68 [61-78] years for TMB ≥10 mut/Mb) and genomic ancestry distribution (eg, European 228 of 339 [67.3%] for low TMB vs 38 of 58 [65.5%] for TMB ≥10 mut/Mb) were similar between TMB subgroups. There were 339 PSCC cases (85.4%) with low TMB, 40 cases (10.1%) with high TMB, and 18 cases (4.5%) with very high TMB. Comparisons of TMB of 10 mut/Mb or higher vs low TMB showed an enrichment of genetic alterations in PIK3CA (48.3% vs 18.3%; P &lt; .001) and KMT2D (29.3% vs 7.7%; P &lt; .001) and less frequent genetic alterations in CDKN2A (25.9% vs 45.7%; P = .05). Most genetic alterations did not co-occur. Human papillomavirus identification was more frequent as TMB increased: 28.3% for low TMB, 50.0% for high, and 72.2% for very high. In total, 95 of 1377 genetic alterations (6.9%) were germline. Of 10 patients identified from the CGDB receiving frontline ICIs, median (IQR) follow-up was 9.9 months. Four patients had overall survival with clinical treatment of more than 12 months, including 2 of 3 patients with TMB of 10 mut/Mb or higher.Conclusions and RelevanceIn this cohort study of advanced metastatic PSCC based on TMB levels, significant differences were observed for biomarkers in nearly 15% of patients with a TMB of 10 mut/Mb or higher. Germline testing and ICI-based therapy should be integrated into the management of selected PSCC cases.

show abstract

Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records

Cited by 8 publications

References 29 publications

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

Using Artificial Intelligence To Label Free-Text Operative And Ultrasound Reports For Grading Pediatric Appendicitis

Genomic Profiles and Clinical Outcomes of Penile Squamous Cell Carcinoma With Elevated Tumor Mutational Burden

Contact Info

Product

Resources

About