Usability of a Machine-Learning Clinical Order Recommender System Interface for Clinical Decision Support and Physician Workflow

Abstract

Objective: To determine whether clinicians will use machine learned clinical order recommender systems for electronic order entry for simulated inpatient cases, and whether such recommendations impact the clinical appropriateness of the orders being placed. Materials and Methods: 43 physicians used a clinical order entry interface for five simulated medical cases, with each physician-case randomized whether to have access to a previously-developed clinical order recommendation system. A panel of clinicians scored determined whether orders placed were clinically appropriate. Outcomes assessed included The primary outcome was the differences in clinical appropriateness scores, of orders for cases randomized to the recommender system. Secondary outcomes included usage metrics, and physician opinions. Results: Clinical appropriateness scores for orders were comparable for cases randomized to the recommender system (mean difference -0.1 order per score, 95% CI:[-0.4, 0.2]). Physicians using the recommender placed more orders (mean 17.3 vs. 15.7 orders; incidence ratio 1.09, 95% CI:[1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Approximately 95% of participants agreed the system would be useful for their workflows. Discussion: Machine-learned clinical order options can meet physician needs better than standard manual search systems. This may increase the number of clinical orders placed per case, while still resulting in similar overall clinically appropriate choices. Conclusions: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

Fallacy of Median Door‐to‐ECG Time: Hidden Opportunities for STEMI Screening Improvement

“… 14 , 15 , 16 , 17 Predictive models using statistical inference and machine learning are finding opportunities to enhance diagnosis. 16 , 17 , 18 , 19 There is prior research identifying risk factors for STEMI as a disease. 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 However, developing a predictive model to enhance timely diagnosis requires an understanding of risk factors for STEMI diagnostic delay.…”

Artificial intelligence for clinical oncology

“…Beyond validation of clinically meaningful endpoints, demonstrating clinical usability involves study of the AI model in a real-world setting, where it interfaces with clinical practitioners and patients. Evaluation of effects of the model on time task, user satisfaction, and acceptance of AI recommendations should be performed (Kumar et al, 2020). A mechanism of feedback should be integrated into the design of the platform to identify weak points and opportunities for improved interface (Cutillo et al, 2020).…”