Using statistical anomaly detection models to find clinical decision support malfunctions

Ray, Soumi; McEvoy, Dustin; Aaron, Skye; Hickman, Thu-Trang T.; Wright, Adam

doi:10.1093/jamia/ocy041

Cited by 33 publications

(32 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the documented results of commissioning can be reused at specified time intervals, in order to certify that the CDSS performance has not unduly drifted over time. Multiple statistical anomaly detection models applied to anomaly detection on CDSS over time have been described and compared in the literature, and the most appropriate method will depend on the nature of the CDSS . The nature and frequency of such QA tests depends on the likelihood of unwanted deviation in CDSS performance and its potential consequences.…”

Section: Quality Assurancementioning

confidence: 99%

Artificial intelligence‐based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance

Mahadevaiah

Bermejo

et al. 2020

Medical Physics

View full text Add to dashboard Cite

Background Recent advances in machine and deep learning based on an increased availability of clinical data have fueled renewed interest in computerized clinical decision support systems (CDSSs). CDSSs have shown great potential to improve healthcare, increase patient safety and reduce costs. However, the use of CDSSs is not without pitfalls, as an inadequate or faulty CDSS can potentially deteriorate the quality of healthcare and put patients at risk. In addition, the adoption of a CDSS might fail because its intended users ignore the output of the CDSS due to lack of trust, relevancy or actionability. Aim In this article, we provide guidance based on literature for the different aspects involved in the adoption of a CDSS with a special focus on machine and deep learning based systems: selection, acceptance testing, commissioning, implementation and quality assurance. Results A rigorous selection process will help identify the CDSS that best fits the preferences and requirements of the local site. Acceptance testing will make sure that the selected CDSS fulfills the defined specifications and satisfies the safety requirements. The commissioning process will prepare the CDSS for safe clinical use at the local site. An effective implementation phase should result in an orderly roll out of the CDSS to the well‐trained end‐users whose expectations have been managed. And finally, quality assurance will make sure that the performance of the CDSS is maintained and that any issues are promptly identified and solved. Conclusion We conclude that a systematic approach to the adoption of a CDSS will help avoid pitfalls, improve patient safety and increase the chances of success.

show abstract

Section: Quality Assurancementioning

confidence: 99%

Artificial intelligence‐based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance

Mahadevaiah

Bermejo

et al. 2020

Medical Physics

View full text Add to dashboard Cite

show abstract

“…Outlier detection a rich area of research in data mining [5,6] and has been extensively applied to clinical data for addressing different issues, such as detecting unusual patient-management actions in ICU, [7] deriving workflow consensus from multiple clinical activity logs, [8] characterizing critical conditions in patients undergoing cardiac surgery, [9] discovering unusual patient management, [10] alert firing within Clinical Decision Support Systems, [11] finding clinical decision support malfunctions, [12] identifying high performers in hypoglycemia safety in diabetic patients, [13] and classifying the influence factor in diabetes symptoms. [14] There are many algorithms developed to detect outliers based on different approaches to what constitutes an outlier, for which there is no universally agreed definition.…”

Section: Methodsmentioning

confidence: 99%

A Clustering Approach for Detecting Implausible Observation Values in Electronic Health Records Data

Estiri

Murphy

2019

Preprint

View full text Add to dashboard Cite

BackgroundIdentifying implausible clinical observations (e.g., laboratory test and vital sign values) in Electronic Health Record (EHR) data using rule-based procedures is challenging. Anomaly/outlier detection methods can be applied as an alternative algorithmic approach to flagging such implausible values in EHRs. ObjectiveThe primary objectives of this research were to develop and test an unsupervised clusteringbased anomaly/outlier detection approach for detecting implausible observations in EHR data as an alternative algorithmic solution to the existing procedures. MethodsOur approach is built upon two underlying hypotheses that, (i) when there are large number of observations, implausible records should be sparse, and therefore (ii) if these data are clustered properly, clusters with sparse populations should represent implausible observations. To test these hypotheses, we applied an unsupervised clustering algorithm to EHR observation data on 50 laboratory tests. We tested different specifications of the clustering approach and computed confusion matrix indices against a set of silver-standard plausibility thresholds. We compared the results from the proposed approach with conventional anomaly detection (CAD) approach's, including standard deviation and Mahalanobis distance. ResultsWe found that the clustering approach produced results with exceptional specificity and high sensitivity. Compared with the conventional anomaly detection approaches, our proposed clustering approach resulted in significantly smaller number of false positive cases. ConclusionOur contributions include (i) a clustering approach for identifying implausible EHR observations, (ii) evidence that implausible observations are sparse in EHR laboratory test results, (iii) a parallel implementation of the clustering approach on i2b2 star schema, and (3) a set of silver-standard plausibility thresholds for 50 laboratory tests that can be used in other studies for validation. The proposed algorithmic solution can augment human decisions to improve data quality. Therefore, a workflow is needed to complement the algorithm's job and initiate necessary actions that need to be taken in order to improve the quality of data.

show abstract

“…As part of a larger project on CDS malfunctions, 4–12 we have been monitoring academic and industry sources for reports of malfunctions. We have recently observed three separate reports of malfunctions related to the classification of carvedilol.…”

Section: Case Seriesmentioning

confidence: 99%

Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol

Wright

Aaron

et al. 2018

Journal of the American Medical Informatics Association

Self Cite

View full text Add to dashboard Cite

Clinical vocabularies allow for standard representation of clinical concepts, and can also contain knowledge structures, such as hierarchy, that facilitate the creation of maintainable and accurate clinical decision support (CDS). A key architectural feature of clinical hierarchies is how they handle parent-child relationships — specifically whether hierarchies are strict hierarchies (allowing a single parent per concept) or polyhierarchies (allowing multiple parents per concept). These structures handle subsumption relationships (ie, ancestor and descendant relationships) differently. In this paper, we describe three real-world malfunctions of clinical decision support related to incorrect assumptions about subsumption checking for β-blocker, specifically carvedilol, a non-selective β-blocker that also has α-blocker activity. We recommend that 1) CDS implementers should learn about the limitations of terminologies, hierarchies, and classification, 2) CDS implementers should thoroughly test CDS, with a focus on special or unusual cases, 3) CDS implementers should monitor feedback from users, and 4) electronic health record (EHR) and clinical content developers should offer and support polyhierarchical clinical terminologies, especially for medications.

show abstract

Using statistical anomaly detection models to find clinical decision support malfunctions

Abstract: Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

Cited by 33 publications

References 48 publications

Artificial intelligence‐based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance

Artificial intelligence‐based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance

A Clustering Approach for Detecting Implausible Observation Values in Electronic Health Records Data

Smashing the strict hierarchy: three cases of clinical decision support malfunctions involving carvedilol

Contact Info

Product

Resources

About