Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Chaparro, Juan D.; Beus, Jonathan M.; Dziorny, Adam; Hagedorn, Philip; Hernandez, Sean; Kandaswamy, Swaminathan; Kirkendall, Eric S.; McCoy, Allison B.; Muthu, Naveen; Orenstein, Evan

doi:10.1055/s-0042-1748856

Cited by 19 publications

(12 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior studies have shown that interruptive alerts are frequently disabled 37,38 due to inappropriate alert triggering, poor usability, or poor fit with the workflow, 39 with recommendation to avoid using interruptive alerts for nonemergent CDSS. 40,41 There is some evidence that interruptive alerts do cause more change in provider behavior than noninterruptive alerts, 39 giving credence to the idea that a noninterruptive IADx would commonly be ignored. 42 Provider preference may partly be explained by the fact that users were only offered the options of either an interruptive or noninterruptive BPA.…”

Section: Discussionmentioning

confidence: 99%

Human-Centered Design of a Clinical Decision Support for Anemia Screening in Children with Inflammatory Bowel Disease

et al. 2023

View full text Add to dashboard Cite

Background Inflammatory bowel disease (IBD) commonly leads to iron deficiency anemia (IDA). Rates of screening and treatment of IDA are often low. A clinical decision support system (CDSS) embedded in an electronic health record could improve adherence to evidence-based care. Rates of CDSS adoption are often low due to poor usability and fit with work processes. One solution is to use human centered design (HCD), which designs CDSS based on identified user needs and context of use and evaluates prototypes for usefulness and usability. Objectives Use HCD to design a CDSS tool called the IBD Anemia Diagnosis Tool, IADx. Methods Interviews with IBD practitioners informed creation of a process map of anemia care that was used by an interdisciplinary team that used HCD principles to create a prototype CDSS. The prototype was iteratively tested with “Think Aloud” usability evaluation with clinicians as well as semi-structured interviews, a survey, and observations. Feedback was coded and informed redesign. Results Process mapping showed that IADx should function at in-person encounters and asynchronous lab review. Clinicians desired full automation of clinical information acquisition such as lab trends and analysis such as calculation of iron deficit, less automation of clinical decision selection such as lab ordering and no automation of action implementation such as signing medication orders. Providers preferred an interruptive alert over a non-interruptive reminder. Discussion Providers preferred an interruptive alert, perhaps due to the low likelihood of noticing a non-interruptive advisory. High levels of desire for automation of information acquisition and analysis with less automation of decision selection and action may be generalizable to other CDSSs designed for chronic disease management. This underlines the ways in which CDSSs have the potential to augment rather than replace provider cognitive work.

show abstract

Section: Discussionmentioning

confidence: 99%

Human-Centered Design of a Clinical Decision Support for Anemia Screening in Children with Inflammatory Bowel Disease

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Much of the progress in NLP has focused on information retrieval and extraction [32]; however, the application of these methods at scale with a combination of software developers and operations remains challenging at health care institutions. The role of NLP in BPAs has been limited to date, and prior BPAs have used existing technologies embedded into the EHR [33]. Similar to prior motivations for BPAs delivered to bedside clinicians [34], our intention was to support and enhance decision-making at the beside with a recommendation for an Addiction Medicine consult in patients who may otherwise not receive it or have it delayed, similar to another NLP-driven BPA [17].…”

Section: Principal Findingsmentioning

confidence: 99%

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

Afshar¹,

Adelaine²,

Resnik³

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

Background The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. Objective We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. Methods The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. Results The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. Conclusions The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence–driven CDS. Trial Registration ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480

show abstract

“…In this study, psuedo-randomized testing is used to compare an experimental alert to the control of no alert. We see great potential for this methodology to support clinical decision support governance 29,30 -such trials can be used in future quality improvement projects to ensure that ineffective alerts are not perpetuated in the EHR by testing proposed interventions against a control 31 , particularly when multiple interventions are being trialed simultaneously Limitations of the EHR software make it difficult to conduct a truly randomized study without customization or use of a third-party application. In quality improvement or more operationally focused projects, it may be infeasible to dedicate the resources required to set up such an evaluation.…”

Section: The Use Of Pseudo-randomized Testing In Quality Improvementmentioning

confidence: 99%

Pseudorandomized Testing of a Discharge Medication Alert to Reduce Free-Text Prescribing

Rabbani

Dash

et al. 2023

Appl Clin Inform

View full text Add to dashboard Cite

Background: Pseudo-randomized testing can be applied to perform rigorous yet practical evaluations of clinical decision support tools. We apply this methodology to an interruptive alert aimed at reducing free-text prescriptions. Using free-text instead of structured computerized provider order entry elements can cause medication errors and inequity in care by bypassing medication-based clinical decision support tools and hindering automated translation of prescription instructions. Objective: Evaluate the effectiveness of an interruptive alert at reducing free-text prescriptions via pseudo-randomized testing using native electronic health records (EHR) functionality. Methods: Two versions of an EHR alert triggered when a provider attempted to sign a discharge free-text prescription. The visible version displayed an interruptive alert to the user, and a silent version triggered in the background, serving as a control. Providers were assigned to the visible and silent arms based on even/odd EHR provider IDs. The proportion of encounters with a free-text prescription was calculated across the groups. Alert trigger rates were compared in process control charts. Free-text prescriptions were analyzed to identify prescribing patterns. Results: Over the 28 week study period, 143 providers triggered 695 alerts (345 visible and 350 silent). The proportions of encounters with free-text prescriptions were 83% (266/320) and 90% (273/303) in the intervention and control groups respectively (p-value = 0.01). For the active alert, median time to action was 31 seconds. Alert trigger rates between groups were similar over time. Ibuprofen, oxycodone, steroid tapers, and oncology-related prescriptions accounted for most free-text prescriptions. A majority of these prescriptions originated from user preference lists. Discussion: An interruptive alert was associated with a modest reduction in free-text prescriptions. Furthermore, the majority of these prescriptions could have been reproduced using structured order entry fields. Targeting user preference lists shows promise for future intervention.

show abstract

Clinical Decision Support Stewardship: Best Practices and Techniques to Monitor and Improve Interruptive Alerts

Cited by 19 publications

References 66 publications

Human-Centered Design of a Clinical Decision Support for Anemia Screening in Children with Inflammatory Bowel Disease

Human-Centered Design of a Clinical Decision Support for Anemia Screening in Children with Inflammatory Bowel Disease

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

Pseudorandomized Testing of a Discharge Medication Alert to Reduce Free-Text Prescribing

Contact Info

Product

Resources

About