Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study

Wright, Adam; Ash, Joan S.; Aaron, Skye; Ai, Angela; Hickman, Thu Trang T.; Wiesen, Jane; Galanter, William L.; McCoy, Allison B.; Schreiber, Richard; Longhurst, Chris; Sittig, Dean F.

doi:10.1016/j.ijmedinf.2018.08.001

Cited by 28 publications

(21 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As EHR developers continue to develop more and varied ways to incorporated CDS alerts into their EHRs and other systems, many healthcare organizations are interested in setting up sustainable governance structures to manage their use with organizational controls (Wright et al, 2018). Many of the ideas investigated in this research were influenced by the expectations of a governance group comprised of clinical, administrative, and information technology professionals.…”

Section: Discussionmentioning

confidence: 99%

Comparative Effectiveness of Best Practice Alerts with Active and Passive Presentations: A Retrospective Study

Valvona

Rayo

Abdel‐Rasoul

et al. 2020

Proceedings of the International Symposium on Human Factors and

View full text Add to dashboard Cite

We assess the relationship of active or passive presentation of Best Practice Advisories (BPAs) for hospital clinicians with compliance rates of recommended actions. We identify the design characteristics of alerts that can be used to assess the effectiveness of design choices with superior usability. Alerts in Electronic Health Records (EHRs) are frequently overridden by healthcare providers. Identifying characteristics of effective alerts can increase the frequency that actions recommended in evidence-based care guidelines are done, reduce user frustration, and improve interface usability along with the willingness to use alerts. We conducted a retrospective analysis of data for 11 BPAs between June 2014 and May 2015. The outcome measure was the percent correspondence with recommended actions. A repeated measures regression model was used for the correlation of the BPA presentation type with the outcome measure. The BPA presentation type was significant such that the odds are 7.7 times greater that a recommended action would be taken by a provider with an active BPA presentation type after adjusting for whether an action was required. Active presentation alerts achieve higher compliance rates. CDS alerts that actively interrupted the provider’s workflow were associated with a higher compliance rate with recommended actions.

show abstract

Section: Discussionmentioning

confidence: 99%

Comparative Effectiveness of Best Practice Alerts with Active and Passive Presentations: A Retrospective Study

Valvona

Rayo

Abdel‐Rasoul

et al. 2020

Proceedings of the International Symposium on Human Factors and

View full text Add to dashboard Cite

show abstract

“…We aimed to understand if telemedicine as a care setting demonstrated significantly different firing volumes or firing rates than office visits. Regular review of alert firing rates is the best practice for identifying alert display malfunctions [ 15 ]. Many organizations, like ours, have adopted dashboards for ongoing monitoring of alert firing rates and firing volumes [ 16 - 18 ].…”

Section: Methodsmentioning

confidence: 99%

Giving Your Electronic Health Record a Checkup After COVID-19: A Practical Framework for Reviewing Clinical Decision Support in Light of the Telemedicine Expansion

Feldman¹,

Szerencsy²,

Mann³

et al. 2021

JMIR Med Inform

View full text Add to dashboard Cite

Background The transformation of health care during COVID-19, with the rapid expansion of telemedicine visits, presents new challenges to chronic care and preventive health providers. Clinical decision support (CDS) is critically important to chronic care providers, and CDS malfunction is common during times of change. It is essential to regularly reassess an organization's ambulatory CDS program to maintain care quality. This is especially true after an immense change, like the COVID-19 telemedicine expansion. Objective Our objective is to reassess the ambulatory CDS program at a large academic medical center in light of telemedicine's expansion in response to the COVID-19 pandemic. Methods Our clinical informatics team devised a practical framework for an intrapandemic ambulatory CDS assessment focused on the impact of the telemedicine expansion. This assessment began with a quantitative analysis comparing CDS alert performance in the context of in-person and telemedicine visits. Board-certified physician informaticists then completed a formal workflow review of alerts with inferior performance in telemedicine visits. Informaticists then reported on themes and optimization opportunities through the existing CDS governance structure. Results Our assessment revealed that 10 of our top 40 alerts by volume were not firing as expected in telemedicine visits. In 3 of the top 5 alerts, providers were significantly less likely to take action in telemedicine when compared to office visits. Cumulatively, alerts in telemedicine encounters had an action taken rate of 5.3% (3257/64,938) compared to 8.3% (19,427/233,636) for office visits. Observations from a clinical informaticist workflow review included the following: (1) Telemedicine visits have different workflows than office visits. Some alerts developed for the office were not appearing at the optimal time in the telemedicine workflow. (2) Missing clinical data is a common reason for the decreased alert firing seen in telemedicine visits. (3) Remote patient monitoring and patient-reported clinical data entered through the portal could replace data collection usually completed in the office by a medical assistant or registered nurse. Conclusions In a large academic medical center at the pandemic epicenter, an intrapandemic ambulatory CDS assessment revealed clinically significant CDS malfunctions that highlight the importance of reassessing ambulatory CDS performance after the telemedicine expansion.

show abstract

“…[1][2][3][4][5] However, in recent work, our research team identified a large number of implementation errors and technical malfunctions in CDS systems. [6][7][8][9][10] Causes for these errors vary, and include improper definition of value sets, terminology changes, and system upgrades. 7,9,10 Many CDS systems allow for the construction of rules from logical terms, linked through a Boolean logic statement.…”

Section: Background and Significancementioning

confidence: 99%

Algorithmic Detection of Boolean Logic Errors in Clinical Decision Support Statements

et al. 2021

Self Cite

View full text Add to dashboard Cite

Objective Clinical decision support (CDS) can contribute to quality and safety. Prior work has shown that errors in CDS systems are common and can lead to unintended consequences. Many CDS systems use Boolean logic, which can be difficult for CDS analysts to specify accurately. We set out to determine the prevalence of certain types of Boolean logic errors in CDS statements. Methods Nine health care organizations extracted Boolean logic statements from their Epic electronic health record (EHR). We developed an open-source software tool, which implemented the Espresso logic minimization algorithm, to identify three classes of logic errors. Results Participating organizations submitted 260,698 logic statements, of which 44,890 were minimized by Espresso. We found errors in 209 of them. Every participating organization had at least two errors, and all organizations reported that they would act on the feedback. Discussion An automated algorithm can readily detect specific categories of Boolean CDS logic errors. These errors represent a minority of CDS errors, but very likely require correction to avoid patient safety issues. This process found only a few errors at each site, but the problem appears to be widespread, affecting all participating organizations. Conclusion Both CDS implementers and EHR vendors should consider implementing similar algorithms as part of the CDS authoring process to reduce the number of errors in their CDS interventions.

show abstract

Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study

Abstract: These 47 best practices represent an ideal situation. The research identifies the balance between importance and difficulty, highlights the challenges faced by organizations seeking to implement CDS, and describes several opportunities for future research to reduce alert malfunctions.

Cited by 28 publications

References 45 publications

Comparative Effectiveness of Best Practice Alerts with Active and Passive Presentations: A Retrospective Study

Comparative Effectiveness of Best Practice Alerts with Active and Passive Presentations: A Retrospective Study

Giving Your Electronic Health Record a Checkup After COVID-19: A Practical Framework for Reviewing Clinical Decision Support in Light of the Telemedicine Expansion

Algorithmic Detection of Boolean Logic Errors in Clinical Decision Support Statements

Contact Info

Product

Resources

About