A Metrics Taxonomy and Reporting Strategy for Rule-Based Alerts

Krall, Michael A.; Gerace, Alexander

doi:10.7812/tpp/14-227

Cited by 2 publications

(2 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of attempts to build taxonomies of Clinical Decision Support (CDS) systems and rule-based alerts were made [24,25]. However, the authors were focused on types (front end/back end) and major groups as benefit, domain, and type of intervention.…”

Section: Discussionmentioning

confidence: 99%

Clinical calculators in hospital medicine: Availability, classification, and needs

Dziadzko

Gajic

Pickering

et al. 2016

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Clinical calculators in hospital medicine: Availability, classification, and needs

Dziadzko

Gajic

Pickering

et al. 2016

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

“…In particular, there is a growing body of evidence to suggest that CDS systems and alerts are vulnerable to malfunctions 5 , 10 , 11 . CDS alerts can malfunction in a variety of ways: an alert might break; the way an alert is designed might not be helpful to users; or an alert might leave out clinically important inclusions or exclusions, causing it to be triggered for the wrong population of patients or providers 4 , 12–14 . Malfunctioning alerts may pose a safety risk to patients if they encourage providers to take clinical actions that are not appropriate for the patient.…”

Section: Introductionmentioning

confidence: 99%

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons

Aaron

McEvoy

Ray

et al. 2018

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

BackgroundRule-base clinical decision support alerts are known to malfunction, but tools for discovering malfunctions are limited.ObjectiveInvestigate whether user override comments can be used to discover malfunctions.MethodsWe manually classified all rules in our database with at least 10 override comments into 3 categories based on a sample of override comments: “broken,” “not broken, but could be improved,” and “not broken.” We used 3 methods (frequency of comments, cranky word list heuristic, and a Naïve Bayes classifier trained on a sample of comments) to automatically rank rules based on features of their override comments. We evaluated each ranking using the manual classification as truth.ResultsOf the rules investigated, 62 were broken, 13 could be improved, and the remaining 45 were not broken. Frequency of comments performed worse than a random ranking, with precision at 20 of 8 and AUC = 0.487. The cranky comments heuristic performed better with precision at 20 of 16 and AUC = 0.723. The Naïve Bayes classifier had precision at 20 of 17 and AUC = 0.738.DiscussionOverride comments uncovered malfunctions in 26% of all rules active in our system. This is a lower bound on total malfunctions and much higher than expected. Even for low-resource organizations, reviewing comments identified by the cranky word list heuristic may be an effective and feasible way of finding broken alerts.ConclusionOverride comments are a rich data source for finding alerts that are broken or could be improved. If possible, we recommend monitoring all override comments on a regular basis.

show abstract

A Metrics Taxonomy and Reporting Strategy for Rule-Based Alerts

Cited by 2 publications

References 7 publications

Clinical calculators in hospital medicine: Availability, classification, and needs

Clinical calculators in hospital medicine: Availability, classification, and needs

Cranky comments: detecting clinical decision support malfunctions through free-text override reasons

Contact Info

Product

Resources

About