Pharmacists’ perceptions of a machine learning model for the identification of atypical medication orders

Abstract

Objectives The study sought to assess the clinical performance of a machine learning model aiming to identify unusual medication orders. Materials and Methods This prospective study was conducted at CHU Sainte-Justine, Canada, from April to August 2020. An unsupervised machine learning model based on GANomaly and 2 baselines were trained to learn medication order patterns from 10 years of data. Clinical pharmacists dichotomously (typical or atypical) labeled orders and pharmacological profiles (patients’ medication lists). Confusion matrices, areas under the precision-recall curve (AUPRs), and F1 scores were calculated. Results A total of 12 471 medication orders and 1356 profiles were labeled by 25 pharmacists. Medication order predictions showed a precision of 35%, recall (sensitivity) of 26%, and specificity of 97% as compared with pharmacist labels, with an AUPR of 0.25 and an F1 score of 0.30. Profile predictions showed a precision of 49%, recall of 75%, and specificity of 82%, with an AUPR of 0.60, and an F1 score of 0.59. The model performed better than the baselines. According to the pharmacists, the model was a useful screening tool, and 9 of 15 participants preferred predictions by medication, rather than by profile. Discussion Predictions for profiles had higher F1 scores and recall compared with medication order predictions. Although the performance was much better for profile predictions, pharmacists generally preferred medication order predictions. Conclusions Based on the AUPR, this model showed better performance for the identification of atypical pharmacological profiles than for medication orders. Pharmacists considered the model a useful screening tool. Improving these predictions should be prioritized in future research to maximize clinical impact.

Role of artificial intelligence in pharmacy practice: A narrative review

“…The most common reason for exclusion on the second screen was that the article did not evaluate human subjects (37.8%), followed by not including medication use (37.1%). Of these, a total of 22 articles were selected to be discussed in our review 11–32 . Of the included studies, data were primarily derived from an academic hospital (43.5%), were located within the United States (43.5%), were retrospective cohort studies (91.3%), and were published in 2020 or later (65.2%).…”

“…They found that certain predictors such as level of prescribed experience, type of medication (e.g., antimicrobials), and transitions of care as important areas of focus, which may help manage workload in understaffed pharmacies. Another study evaluated atypical prescription orders, defined as those that deviate from usual prescription patterns, which may identify orders that may require higher scrutiny 19 . The authors evaluated three different ML models, finding that the accuracy of their model was variable by medical specialty, with surgery and neonatal intensive care having the highest and lowest accuracy, respectively.…”

“…Recent examples include drug repurposing for identifying effective medications for coronavirus disease 2019 (COVID- 19), such as baricitinib and dexamethasone. 1,2 Within pharmacy practice, the use of AI can enhance various aspects such as medication safety, pharmacy operations, and patient-specific recommendations that are supported by AI models.…”

Development and validation of a machine learning-based detection system to improve precision screening for medication errors in the neonatal intensive care unit

“…In a study in which each ME in NICU patients (n = 410) was classified and predicted as appropriate-inappropriate, the F 1 score indicating model performance was found to be 0.13 (Hogue et al, 2021). In our study, in which a total of 5,954 medication orders specifically for the NICU were examined, the prediction performance of the model obtained with patients categorized as ME detected or not detected was found to be much higher (F 1 score: 0.931).…”

Evaluating the performance of ChatGPT in clinical pharmacy

“…In clinical pharmacy practice, AI-powered systems have demonstrated the potential to aid clinical pharmacists, as previously discussed 18 . The development of AI-powered apps and tools is primarily focused on prescription review 19,20 . Furthermore, the AI platform assists clinical pharmacists in drug counseling, thus reducing costs and medical utilization 21 .…”