Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR

Abstract

Objective To comparatively evaluate the effectiveness of three different methods involving end-users for detecting usability problems in an EHR: user testing, semi-structured interviews and surveys. Materials and methods Data were collected at two major urban dental schools from faculty, residents and dental students to assess the usability of a dental EHR for developing a treatment plan. These included user testing (N=32), semi-structured interviews (N=36), and surveys (N=35). Results The three methods together identified a total of 187 usability violations: 54% via user testing, 28% via the semi-structured interview and 18% from the survey method, with modest overlap. These usability problems were classified into 24 problem themes in 3 broad categories. User testing covered the broadest range of themes (83%), followed by the interview (63%) and survey (29%) methods. Discussion Multiple evaluation methods provide a comprehensive approach to identifying EHR usability challenges and specific problems. The three methods were found to be complementary, and thus each can provide unique insights for software enhancement. Interview and survey methods were found not to be sufficient by themselves, but when used in conjunction with the user testing method, they provided a comprehensive evaluation of the EHR. Conclusion We recommend using a multi-method approach when testing the usability of health information technology because it provides a more comprehensive picture of usability challenges.

User-centered design of multi-gene sequencing panel reports for clinicians

“…The SUS has become a standard of usability measurement [34]. While other usability evaluation surveys exist [35,36], the SUS is among the most broadly used and has a number of advantages including its ease of use with only 10 items rated on a 5-point scale; its comprehensibility given a resultant score ranging from 0 to 100; and it is “technology agnostic” [33,37]. Participants are asked to score each item on a scale of one to five that range from Strongly Disagree to Strongly Agree.…”

Cognitive Informatics for Biomedicine

“…As with medical EHRs, a usercentered designed dental EHR facilitates good usability, assuring that the user can efficiently and effectively complete work tasks satisfactorily and successfully (Walji et al 2014). It is also understood that, on the contrary, a poorly designed EHR with poor usability can lead to potential patient safety issues (Horsky et al 2005a, b;).…”

“…Because one does not get many opportunities to overhaul a major module within the EHR, we conducted three complementary usability assessments in order to maximize our ability to capture challenges. We will summarize that work here; we have published full details in the International Journal of Medical Informatics (Walji et al 2013(Walji et al , 2014 Think-Aloud User Testing We created two pre-defined scenarios to assess users' interactions with DDS in the axiUm Treatment Planning module: a simple task of entering one diagnosis and a more complex treatment-planning task. Participants were asked to think aloud (Ericsson and Simon 1993) and verbalize their thoughts as they worked through each scenario.…”

“…Table 9.1 provides an overview of the 24 high-level usability problems that were found through the use of the three methods. The methods together identified a total of 187 usability violations: 54 % via user testing, 28 % via the semi-structured interview and 18 % from the survey method, with modest overlap (Walji et al 2014). Interfacerelated problems included unexpected approaches for displaying diagnosis, lack of visibility, and inconsistent use of user interface widgets.…”

New model of Electronic Health Record: Macedonian case study

“…Some of these systems include simple electronic health records (EHR) or electronic medical records (EMR) that replace paper records for information storage and retrieval [1]. Walji et al [2] recommend the usage of a multi method approach when testing the usability of health information technology.…”