Impact of a Machine Learning–Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices

Abstract

Background There is increasing attention on machine learning (ML)-based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated in clinical practice. We implemented a CDSS to aid general practitioners (GPs) in treating patients with urinary tract infections (UTIs), which are a significant health burden worldwide. Objective This study aims to prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. In doing so, we hope to identify drivers and obstacles that positively impact the quality of health care practice with ML. Methods The CDSS was developed by Pacmed, Nivel, and Leiden University Medical Center (LUMC). The CDSS presents the expected outcomes of treatments, using interpretable decision trees as ML classifiers. Treatment success was defined as a subsequent period of 28 days during which no new antibiotic treatment for UTI was needed. In this prospective observational study, 36 primary care practices used the software for 4 months. Furthermore, 29 control practices were identified using propensity score-matching. All analyses were performed using electronic health records from the Nivel Primary Care Database. Patients for whom the software was used were identified in the Nivel database by sequential matching using CDSS use data. We compared the proportion of successful treatments before and during the study within the treatment arm. The same analysis was performed for the control practices and the patient subgroup the software was definitely used for. All analyses, including that of physicians’ prescription behavior, were statistically tested using 2-sided z tests with an α level of .05. Results In the treatment practices, 4998 observations were included before and 3422 observations (of 2423 unique patients) were included during the implementation period. In the control practices, 5044 observations were included before and 3360 observations were included during the implementation period. The proportion of successful treatments increased significantly from 75% to 80% in treatment practices (z=5.47, P<.001). No significant difference was detected in control practices (76% before and 76% during the pilot, z=0.02; P=.98). Of the 2423 patients, we identified 734 (30.29%) in the CDSS use database in the Nivel database. For these patients, the proportion of successful treatments during the study was 83%—a statistically significant difference, with 75% of successful treatments before the study in the treatment practices (z=4.95; P<.001). Conclusions The introduction of the CDSS as an intervention in the 36 treatment practices was associated with a statistically significant improvement in treatment success. We excluded temporal effects and validated the results with the subgroup analysis in patients for whom we were certain that the software was used. This study shows important strengths and points of attention for the development and implementation of an ML-based CDSS in clinical practice. Trial Registration ClinicalTrials.gov NCT04408976; https://clinicaltrials.gov/ct2/show/NCT04408976

Artificial-Intelligence-Based Clinical Decision Support Systems in Primary Care: A Scoping Review of Current Clinical Implementations

“…Of the six included studies, three are from the United States [ 27 , 28 , 29 ], while the rest are from the Netherlands [ 30 ], Spain [ 31 ], and China [ 32 ]. The two most common study designs were randomized clinical trials (two) [ 27 , 29 ], one of which was single-blinded [ 27 ], and observational cross-sectional studies (two) [ 28 , 32 ].…”

“…All six studies were performed in primary care clinics. The number of clinicians using the CDSSs was reported in four studies [ 28 , 29 , 31 , 32 ], and the number of patients assessed was reported in three studies [ 27 , 29 , 30 ].…”

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

“…Accuracy was most commonly measured, although a few studies examined safety and clinician time (e.g., [44,46,53]). Effects on care-delivery were assessed using a variety of measures including time to treatment (e.g., [29,39,67,68]). Patient outcomes were assessed using measures like length of stay and mortality (e.g., [61,62,66]), but no studies examined adverse events due to AI errors.…”

“…Other studies reported strategies to incorporate well-known considerations for the use of digital health technologies such as ensuring that AI advice was actionable (e.g., when algorithms were designed to operationalise national guidelines [48]), and integrated into clinical workflow and existing CISs including EHRs (e.g., [22,29,77]). In one study where the AI was not integrated into the EHRs, general practitioners needed to enter patient characteristics into a web-based version of the AI [68]. A key challenge for system implementation is to build upon general considerations for digital health and identify specific measures required for the safe and effective use of ML algorithms.…”

“…In primary care, Herter et al [68] examined an AI that provided general practitioners with expected outcomes and support information about antibiotics for urinary tract infections based on the Dutch College of General Practitioners' guidelines. A pro-spective observational study in 36 primary care practices over a 4-month period was associated with an increase in proportion of successful treatments from 75% to 80% in intervention practices while there was no difference in the matched controls.…”