Assessment of Natural Language Processing of Electronic Health Records to Measure Goals-of-Care Discussions as a Clinical Trial Outcome

Abstract

ImportanceMany clinical trial outcomes are documented in free-text electronic health records (EHRs), making manual data collection costly and infeasible at scale. Natural language processing (NLP) is a promising approach for measuring such outcomes efficiently, but ignoring NLP-related misclassification may lead to underpowered studies.ObjectiveTo evaluate the performance, feasibility, and power implications of using NLP to measure the primary outcome of EHR-documented goals-of-care discussions in a pragmatic randomized clinical trial of a communication intervention.Design, Setting, and ParticipantsThis diagnostic study compared the performance, feasibility, and power implications of measuring EHR-documented goals-of-care discussions using 3 approaches: (1) deep-learning NLP, (2) NLP-screened human abstraction (manual verification of NLP-positive records), and (3) conventional manual abstraction. The study included hospitalized patients aged 55 years or older with serious illness enrolled between April 23, 2020, and March 26, 2021, in a pragmatic randomized clinical trial of a communication intervention in a multihospital US academic health system.Main Outcomes and MeasuresMain outcomes were natural language processing performance characteristics, human abstractor-hours, and misclassification-adjusted statistical power of methods of measuring clinician-documented goals-of-care discussions. Performance of NLP was evaluated with receiver operating characteristic (ROC) curves and precision-recall (PR) analyses and examined the effects of misclassification on power using mathematical substitution and Monte Carlo simulation.ResultsA total of 2512 trial participants (mean [SD] age, 71.7 [10.8] years; 1456 [58%] female) amassed 44 324 clinical notes during 30-day follow-up. In a validation sample of 159 participants, deep-learning NLP trained on a separate training data set from identified patients with documented goals-of-care discussions with moderate accuracy (maximal F1 score, 0.82; area under the ROC curve, 0.924; area under the PR curve, 0.879). Manual abstraction of the outcome from the trial data set would require an estimated 2000 abstractor-hours and would power the trial to detect a risk difference of 5.4% (assuming 33.5% control-arm prevalence, 80% power, and 2-sided α = .05). Measuring the outcome by NLP alone would power the trial to detect a risk difference of 7.6%. Measuring the outcome by NLP-screened human abstraction would require 34.3 abstractor-hours to achieve estimated sensitivity of 92.6% and would power the trial to detect a risk difference of 5.7%. Monte Carlo simulations corroborated misclassification-adjusted power calculations.Conclusions and RelevanceIn this diagnostic study, deep-learning NLP and NLP-screened human abstraction had favorable characteristics for measuring an EHR outcome at scale. Adjusted power calculations accurately quantified power loss from NLP-related misclassification, suggesting that incorporation of this approach into the design of studies using NLP would be beneficial.

Intervention to Promote Communication About Goals of Care for Hospitalized Patients With Serious Illness

“…Fourth, the construct of goals-of-care discussions represents a spectrum of important discussions that are multifaceted and vary over dimensions of context, timing, depth, content, and execution—all of which influence the quality of goals-of-care discussions . Efforts are ongoing to refine the criteria by which goals-of-care discussions are defined and measured …”

“…The primary outcome was the proportion of patients with EHR-documented goals-of-care discussions within 30 days of randomization. Documented goals-of-care discussions were identified by natural language processing (NLP)–screened human abstraction and this process is separately described . Goals-of-care discussions were defined as those that were about a patient’s overarching goals for medical care .…”

“…Passages that screened positive by the NLP model were then reviewed by research coordinators to determine the presence or absence of documented goals-of-care discussions. The screening threshold was set to achieve a patient-level sensitivity of 92.6% for NLP-screened human abstraction in a 159-patient internal validation sample; this discrimination threshold corresponded with a patient-level sensitivity of 99.2% and specificity of 33.2% for the NLP model functioning as a standalone classifier within the same validation sample . Randomization was concealed from abstractors, and instances of disagreement were resolved by consensus.…”

“…In addition, these 2 trials involved manual EHR review to assess documentation of goals-of-care discussions that is time intensive. These limitations highlight the need to evaluate the effectiveness of a more scalable implementation of the intervention as well as more efficient methods to measure documented goals-of-care discussions for patients with serious illness …”

“…These limitations highlight the need to evaluate the effectiveness of a more scalable implementation of the intervention as well as more efficient methods to measure documented goals-of-care discussions for patients with serious illness. 12,13 This large pragmatic RCT tested whether a clinicianfacing communication-priming intervention (Jumpstart Guide) containing patient-specific EHR data and communication prompts increased documented goals-of-care discussions for hospitalized older patients with serious illness compared with usual care. The differential effects of the intervention were examined by age, sex, history of dementia, minoritized race or ethnicity, and study site.…”