Analysis of clinical decision support system malfunctions: a case series and survey

Abstract

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions.Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women’s Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions.Results We identified four CDSS malfunctions at Brigham and Women’s Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually.Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate.Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit

“…Malfunctions of CDS have been studied in the literature, which has identified that they may be widespread and may exist for long time periods due to lack of investigation into this matter 23. A few of the malfunctions that were encountered during this study period were focused on the actual medication ordered compared with what the CDS believed was ordered (eg, simvastatin as previously mentioned).…”

Change-point detection for monitoring clinical decision support systems with a multi-process dynamic linear model

“…We use real data of rule firing counts from a large teaching hospital collected over a period of approximately five years [1]. There are 14 rules containing at least one known change-point with a total of 22 change-points, which are used in the real data experiment.…”

“…Ideally, the system would function optimally all the time. However, in reality the system may be prone to various kinds of malfunctions affecting its performance [1]. Developing tools that are able to detect early and reliably its malfunctions is critical to preserve its intended function and to eliminate potential patient-related risks.…”

Advances in Clinical Decision Support: Highlights of Practice and the Literature 2015-2016

“…Moreover, even systems that are certified under regulatory schemes may lack the capability to implement certain kinds of decision support rules [43]. Indeed, there is increasing recognition that CPOE and related technology can malfunction in ways that cause problems, which poses at least a potential threat to patient safety [44]. Nevertheless, there is recognition that an emphasis on user interface design issues may hold the key to improving the utility of this technology, such as simulation work with potential users that can help identify potential failure points [45].…”