Expert Reviews: How Many Experts is Enough?

Abstract

We asked five usability specialists to review the user interface to a phone-based, interactive voice response system. The experts were instructed to conduct their review independently in three one-hour sessions and to record each usability problem on a Problem Description Sheet along with the elapsed time from the beginning of the hour. Each expert then spent one hour reviewing their problem sheets and making a summary list of problems. Finally, the experts spent two hours together on a conference call discussing their impressions and coming to consensus on a prioritized list of problems and solutions. The results showed that when allocating expert time, it is more effective to have a greater number of experts spend fewer hours than to use fewer experts for more hours. The individual summaries included the majority of the severe problems, but left out many less severe problems and added new problems. The group report did not surface any new problems, but described the problems as being caused by more basic design flaws and proposed solutions that focused on the conceptual model on which the design was based.

Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems

“…Since the developer was interested in specific areas of the system, a taskbased approach was used for both methods rather than including an additional "free-form" exploration. The limited time for the evaluation session is in-line with recommendations provided by Dumas, Sorce, and Virzi (1995), where it appears to be more effective to have a greater number of evaluators work for a shorter time than to have one or two evaluators examine an interface for an extended period of time.…”

“…Had the evaluation been extended for several hours, it is very likely that detection rates would be in the range reported by other similar evaluation studies. Although the participants had limited evaluation time, Dumas et al (1995) have pointed out in practical application it is more effective to have a greater number of evaluators examine an interface for a shorter period of time than just a few evaluators who spend a large amount of time. The result of using several evaluators for short periods of time inspecting an interface generally produces better coverage of detecting important usability problems.…”

Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice?

“…Here I argue that the value of HE differs from that of a user test, and, like Dumas et al (1995), I start with the assumption that inspection methods, including HE, are effective.…”

“…Virzi, 1997), most authors tend to support the use of these methods, some very strongly (Mack and Nielsen, 1995;Nielsen and Molich, 1990), others with some reservation (Dumas et al, 1995;Karat et al, 1992;Virzi et al, 1993). However, authors agree that these methods do have some value especially when performed early in the development cycle (Virzi, 1997).…”

Improved Usability of Aviation Automation Through Direct Manipulation and Graphical User Interface Design

“…Some references offer that approximately 5 expert usability evaluators can identify about 75% of known usability problems in test applications (Dix et al, 1998;Nielsen, 1994). Other research has shown that as few as 3 experts working for a limited time duration (a 1-hr evaluation of an interactive voice-response system) can produce a 70% problem detection rate, on average (Dumas, Sorce, & Virzi, 1995). System complexity should also be considered in determining the number of judges to use because more complex systems, such as the MCDU, may require more experts for effective coverage of usability problems.…”