Machine learning optimization of an electronic health record audit for heart failure in primary care

Abstract

Aims The diagnosis of heart failure (HF) is an important problem in primary care. We previously demonstrated a 74% increase in registered HF diagnoses in primary care electronic health records (EHRs) following an extended audit procedure. What remains unclear is the accuracy of registered HF pre‐audit and which EHR variables are most important in the extended audit strategy. This study aims to describe the diagnostic HF classification sequence at different stages, assess general practitioner (GP) HF misclassification, and test the predictive performance of an optimized audit. Methods and results This is a secondary analysis of the OSCAR‐HF study, a prospective observational trial including 51 participating GPs. OSCAR used an extended audit based on typical HF risk factors, signs, symptoms, and medications in GPs' EHR. This resulted in a list of possible HF patients, which participating GPs had to classify as HF or non‐HF. We compared registered HF diagnoses before and after GPs' assessment. For our analysis of audit performance, we used GPs' assessment of HF as primary outcome and audit queries as dichotomous predictor variables for a gradient boosted machine (GBM) decision tree algorithm and logistic regression model. Of the 18 011 patients eligible for the audit intervention, 4678 (26.0%) were identified as possible HF patients and submitted for GPs' assessment in the audit stage. There were 310 patients with registered HF before GP assessment, of whom 146 (47.1%) were judged not to have HF by their GP (over‐registration). There were 538 patients with registered HF after GP assessment, of whom 374 (69.5%) did not have registered HF before GP assessment (under‐registration). The GBM and logistic regression model had a comparable predictive performance (area under the curve of 0.70 [95% confidence interval 0.65–0.77] and 0.69 [95% confidence interval 0.64–0.75], respectively). This was not significantly impacted by reducing the set of predictor variables to the 10 most important variables identified in the GBM model (free‐text and coded cardiomyopathy, ischaemic heart disease and atrial fibrillation, digoxin, mineralocorticoid receptor antagonists, and combinations of renin‐angiotensin system inhibitors and beta‐blockers with diuretics). This optimized query set was enough to identify 86% (n = 461/538) of GPs' self‐assessed HF population with a 33% reduction (n = 1537/4678) in screening caseload. Conclusions Diagnostic coding of HF in primary care health records is inaccurate with a high degree of under‐registration and over‐registration. An optimized query set enabled identification of more than 80% of GPs' self‐assessed HF population.

Mixed‐methods evaluation of a multifaceted heart failure intervention in general practice: the OSCAR‐HF pilot study

“…Audit and feedback were used to overcome the problem of under‐registration, under‐diagnosis, and overdiagnosis and the lack of uptitration of GDMT in general practice. 13 , 14 , 23 , 24 The use of computerized audit and feedback for quality improvement is not new; however, it is more frequently used to target HF treatment than to diagnose HF. 11 , 25 , 26 Our study showed that improving the registration and validity of general practice HF diagnoses is necessary and feasible, in line with a recent British study.…”

“…We approached the concern of GPs with respect to the time investment by refining the audit strategy to optimize its specificity (thus reducing the number‐needed‐to‐screen). 23 In addition, these audits should be integrated intuitively into the GPs' existing EHR architecture (‘simply press the button’). This should be relatively easy because the bulk of HF patients in primary care can be identified with a small set of EHR queries.…”

“…This should be relatively easy because the bulk of HF patients in primary care can be identified with a small set of EHR queries. 23 NP testing, together with specialist collaboration, is an essential complement to the audit process because the audit procedure cannot exist without feedback on the validity of the HF diagnosis. 27 , 32 Audit and feedback were not sufficient to improve the uptitration of GDMT.…”

“…First, audit and feedback should be considered an essential part of HF population management in primary care. We approached the concern of GPs with respect to the time investment by refining the audit strategy to optimize its specificity (thus reducing the number‐needed‐to‐screen) 23 . In addition, these audits should be integrated intuitively into the GPs' existing EHR architecture (‘simply press the button’).…”

Reply to the letter regarding the article ‘Dulaglutide and cardiovascular and heart failure outcomes in patients with and without heart failure: a post‐hoc analysis from the REWIND randomized trial’

“…This stratification is supported by our data which show that patients identified as such were a higher risk group than those who were not. Expansion of the HF definition to those with any history of HF is likely to include many patients without a true history of HF 9,10 and may bias our findings of no significant change in HF outcomes and major adverse cardiovascular events with dulaglutide in those with HF at baseline.…”