Considerations for the Use of Machine Learning Extracted Real-World Data to Support Evidence Generation: A Research-Centric Evaluation Framework

Abstract

A vast amount of real-world data, such as pathology reports and clinical notes, are captured as unstructured text in electronic health records (EHRs). However, this information is both difficult and costly to extract through human abstraction, especially when scaling to large datasets is needed. Fortunately, Natural Language Processing (NLP) and Machine Learning (ML) techniques provide promising solutions for a variety of information extraction tasks such as identifying a group of patients who have a specific diagnosis, share common characteristics, or show progression of a disease. However, using these ML-extracted data for research still introduces unique challenges in assessing validity and generalizability to different cohorts of interest. In order to enable effective and accurate use of ML-extracted real-world data (RWD) to support research and real-world evidence generation, we propose a research-centric evaluation framework for model developers, ML-extracted data users and other RWD stakeholders. This framework covers the fundamentals of evaluating RWD produced using ML methods to maximize the use of EHR data for research purposes.

Replication of Real-World Evidence in Oncology Using Electronic Health Record Data Extracted by Machine Learning

“…Additionally, there is a need for model transparency and explainability such that model predictions can be trusted by stakeholders and therefore be more readily accepted [ 36 ]. Finally, proper model evaluation is needed to ensure that models are fair and generalizable, which requires an adequate volume of high-quality labeled test data that is not used during model training and validation [ 6 , 37 ].…”

“…These sentences are then transformed into a mathematical representation that the model can interpret. Individual models used in this study were evaluated with the research-centric evaluation framework developed by Estevez et al [ 6 ]. Each model’s performance was evaluated using a test set of over 3000 unique lung cancer patients.…”

“…How to access and analyze this information at scale for RWE generation is a massive challenge. The standard method of data curation through expert human abstraction is resource-intensive and time-consuming, limiting the number of patients available for research purposes [ 5 , 6 , 7 ]. In response, natural language processing (NLP) with machine learning (ML) techniques (i.e., ML extraction ) is increasingly being applied to EHR data for more efficient and scalable generation of RWD ( Box 1 ).…”

“…In response to this gap, we previously developed a research-centric evaluation framework to evaluate ML-extracted RWD and provide insights on model performance, strengths and limitations, and fitness-for-use [ 6 ]. This framework primarily focuses on evaluating a single ML-extracted variable, independent of the output of other ML extraction models.…”

Approach to Machine Learning for Extraction of Real-World Data Variables from Electronic Health Records

“…Measuring performance is a complex challenge because even a model with good overall performance might systematically underperform on a particular subcohort of interest, and because while conventional metrics apply to individual models, dozens of ML extracted variables may be combined to answer a specific research question. We use a research-centric evaluation framework 34 to assess the quality of variables curated with ML. Evaluations include one or more of the following strategies: (1) overall performance assessment, (2) stratified performance assessment, and (3) quantitative error analysis, and (4) replication analysis.…”