Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned

Abstract

BackgroundAs the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians.ObjectiveThe objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center’s emergency department EHR.MethodsWe conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients’ chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a “think aloud” method and “near-live” clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings.ResultsPhase I: Data from the “think-aloud” phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as “well-organized” and “better than clinical judgment”. Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial.ConclusionsThis study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.

Improving Provider Adoption With Adaptive Clinical Decision Support Surveillance: An Observational Study (Preprint)

“…It is thus crucial to devise computerized tools that are optimally integrated into clinician workflow without causing alert fatigue [3] and that usability may be as important as accuracy and effectiveness in implementing CDS [11]. We speculate that the high adoption rates of 29-42% in our study, as opposed to an average of 10-20% adoption from previous CDS research [9], were a result of the comprehensive user centered development process including workflow assessment of the emergency department, focus groups, usability testing, and collaboration with a multidisciplinary team [23,27,31].…”

“…As our past research study has shown, usability testing prior to integration of the tool can lead to high adoption rates [3]. Thus we implemented a two-phase usability testing approach with emergency physicians prior to the integration of the Wells' CPR into the EHR [31]. Furthermore, the "sensitivity and specificity trigger analysis" (SSTA) was developed to identify optimal trigger locations to avoid underutilization and overtriggering [23].…”

“…After thorough usability testing of the tool in the EHR playground environment within our Usability Lab, [23,26,27] the tool was launched within the emergency rooms at two large academic tertiary centers in a staggered roll out.…”

“…Upon triggering, the provider filled out the Wells' CDS tool as it appears in Figure 1. The completed tool calculated the patient's risk for PE and stratified the patient into low, intermediate, or high categories, each with a bundled order set that the provider could choose for the next step [26]. A trigger rate tracking tool was developed using Tableau to monitor adoption and trigger rates of the Wells' CDS tool implementation.…”

Investigating Clinical Decision Support Systems Success Factors with Usability Testing

“…There are related evidences that the usability of CDSS should be evaluated since it is determined as the successful implementation of CDSS [12], [13]. Authors in [12] evaluate usability of CDSS by conducting two phases of usability testing.…”