P57 Machine Learning-Accelerated Outcomes Research: A Real-World Case Study of Biomarker-Associated Overall Survival in Oncology

Benedum, Corey M.; Adamson, Blythe; Cohen, AB; Estévez, Melissa; Sondhi, Arjun; Fidyk, Erin; Nemeth, Sheila; Bozkurt, Selen

doi:10.1016/j.jval.2022.09.069

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Detailed performance metrics are out of scope for this paper. Beyond satisfactory ML metrics, we found that in some cases ML-extraction can achieve similar error rates as manual abstraction by clinical experts ( Waskom et al, 2023 ), and replication studies suggest that research analysis relying on multiple variables can reach similar results and conclusions when using variables curated by ML-extraction compared with human experts ( Benedum et al, 2022 ; Sondhi et al, 2022 ; Benedum et al, 2023 ).…”

Section: Resultsmentioning

confidence: 70%

“…Using a performance evaluation framework ( Devlin et al, 2018 ) for variables curated using the approach of ML extraction we affirmed high quality and fitness-for-use in RWE generation. We have shown that validations using the combination of multiple ML-extracted variables in one RWD analysis demonstrated no meaningful difference in RWE findings based on replications with the Flatiron Health variables curated by ML extraction compared with expert human abstraction ( Forsyth et al, 2018 ; Zeng et al, 2018 ; Jorge et al, 2019 ; Karimi et al, 2021 ; Maarseveen et al, 2021 ; Benedum et al, 2022 ; Sondhi et al, 2022 ; Yang et al, 2022 ; Benedum et al, 2023 ).…”

Section: Discussionmentioning

confidence: 91%

See 1 more Smart Citation

Approach to machine learning for extraction of real-world data variables from electronic health records

Adamson,

Waskom,

Blarre

et al. 2023

Front. Pharmacol.

View full text Add to dashboard Cite

Background: As artificial intelligence (AI) continues to advance with breakthroughs in natural language processing (NLP) and machine learning (ML), such as the development of models like OpenAI’s ChatGPT, new opportunities are emerging for efficient curation of electronic health records (EHR) into real-world data (RWD) for evidence generation in oncology. Our objective is to describe the research and development of industry methods to promote transparency and explainability.Methods: We applied NLP with ML techniques to train, validate, and test the extraction of information from unstructured documents (e.g., clinician notes, radiology reports, lab reports, etc.) to output a set of structured variables required for RWD analysis. This research used a nationwide electronic health record (EHR)-derived database. Models were selected based on performance. Variables curated with an approach using ML extraction are those where the value is determined solely based on an ML model (i.e. not confirmed by abstraction), which identifies key information from visit notes and documents. These models do not predict future events or infer missing information.Results: We developed an approach using NLP and ML for extraction of clinically meaningful information from unstructured EHR documents and found high performance of output variables compared with variables curated by manually abstracted data. These extraction methods resulted in research-ready variables including initial cancer diagnosis with date, advanced/metastatic diagnosis with date, disease stage, histology, smoking status, surgery status with date, biomarker test results with dates, and oral treatments with dates.Conclusion: NLP and ML enable the extraction of retrospective clinical data in EHR with speed and scalability to help researchers learn from the experience of every person with cancer.

show abstract

Section: Resultsmentioning

confidence: 70%

Section: Discussionmentioning

confidence: 91%

Approach to machine learning for extraction of real-world data variables from electronic health records

Adamson,

Waskom,

Blarre

et al. 2023

Front. Pharmacol.

View full text Add to dashboard Cite

show abstract

“…Beyond satisfactory ML metrics, we found that in some cases ML-extraction can achieve similar error rates as manual abstraction by clinical experts (Waskom et al, in press, 2023), and replication studies suggest that research analysis relying on multiple variables can reach similar results and conclusions when using variables curated by ML-extraction compared with human experts (Benedum et al, in press, 2023). 35 36…”

Section: Resultsmentioning

confidence: 99%

“…We have shown that validations using the combination of multiple ML-extracted variables in one RWD analysis demonstrated no meaningful difference in RWE findings based on replications with the Flatiron Health variables curated by ML extraction compared with expert human abstraction (Benedum et al, in press, 2023). 36 35…”

Section: Discussionmentioning

confidence: 99%

Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records

Adamson

Waskom

Blarre

et al. 2023

Preprint

View full text Add to dashboard Cite

BackgroundAs artificial intelligence (AI) continues to advance with breakthroughs in natural language processing (NLP) and machine learning (ML), such as the development of models like OpenAI’s ChatGPT, new opportunities are emerging for efficient curation of electronic health records (EHR) into real-world data (RWD) for evidence generation in oncology. Our objective is to describe the research and development of industry methods to promote transparency and explainability.MethodsWe applied NLP with ML techniques to train, validate, and test the extraction of information from unstructured documents (eg, clinician notes, radiology reports, lab reports, etc.) to output a set of structured variables required for RWD analysis. This research used a nationwide electronic health record (EHR)-derived database. Models were selected based on performance. Variables curated with an approach using ML extraction are those where the value is determined solely based on an ML model (ie, not confirmed by abstraction), which identifies key information from visit notes and documents. These models do not predict future events or infer missing information.ResultsWe developed an approach using NLP and ML for extraction of clinically meaningful information from unstructured EHR documents and found high performance of output variables compared with variables curated by manually abstracted data. These extraction methods resulted in research-ready variables including initial cancer diagnosis with date, advanced/metastatic diagnosis with date, disease stage, histology, smoking status, surgery status with date, biomarker test results with dates, and oral treatments with dates.ConclusionsNLP and ML enable the extraction of retrospective clinical data in EHR with speed and scalability to help researchers learn from the experience of every person with cancer.

show abstract

P57 Machine Learning-Accelerated Outcomes Research: A Real-World Case Study of Biomarker-Associated Overall Survival in Oncology

Cited by 2 publications

References 0 publications

Approach to machine learning for extraction of real-world data variables from electronic health records

Approach to machine learning for extraction of real-world data variables from electronic health records

Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records

Contact Info

Product

Resources

About