Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

Abstract

Background Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups. Objectives We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard. Methods An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data. Results The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models. Conclusion We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

Comprehensive Decision Analysis of Industry 4.0 Virtual Enterprises considering the Personalized Customization Model of Product Life Cycle

“…From the perspectives of exogenous theory and endogenous theory, it explores the relationship between the transformation of entrepreneurial decision-making ability and the growth of enterprises. Literature [18] formulated information literacy standards for corporate managers, from the perspective of information behavior, and proposed ways to improve the decision-making ability of corporate managers.…”

Visualizing Opportunity Index Data Using a Dashboard Application: A Tool to Communicate Infant Mortality-Based Area Deprivation Index Information

“…The usability study involved an evaluation of the OID that focused on the effectiveness of the dashboard and satisfaction from its use, these are primary outcomes that have been similarly employed by other dashboard evaluations. [37][38][39] Wu et al define effectiveness as the accuracy and completeness of achieving goals and satisfaction as subjective opinions of use. Given the use of dashboards to discover insights through exploration, we recognize the challenges with operationalizing an efficiency metric on dashboard use and chose to informally track this metric.…”

The Effect of Electronic Health Record Usability Redesign on Annual Screening Rates in an Ambulatory Setting

“…17 Many studies of EHR usability are limited to descriptions of UCD processes or results of usability evaluations. [18][19][20][21] Other studies of EHR usability focus on poor usability and its association with patient harm and clinician dissatisfaction. [2][3][4]6,22,23 While it is important to continue to describe the impact of poor usability, our understanding of usability is enhanced by careful descriptions of the improvements seen with corrections of usability problems.…”