Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?

Abstract

Attention has been given to making user interface design and testing less costly so that it might be more easily incorporated into the product development life cycle. Three experiments are reported in this paper that relate the proportion of usability problems identified in an evaluation to the number of subjects participating in that study. The basic findings are that (a) 80% of the usability problems are detected with four or five subjects, (b) additional subjects are less and less likely to reveal new information, and (c) the most severe usability problems are likely to have been detected in the first few subjects. Ramifications for the practice of human factors are discussed as they relate to the type of usability test cycle employed and the goals of the usability test.

Case studies of mental models in home heat control: Searching for feedback, valve, timer and switch theories

“…Flyvbjerg (2011) argues that rich data gathered from detailed, real life situations can provide meaningful insights, that could not be gained from context-independent findings. Virzi (1992), R.A (1992), when conducting research into usability, found 80% of problems, including the most severe, are detected with the first 4-5 subjects, illustrating how key insights can be gained with very small samples. Hancock et al (2009) also argue that ideographic case representations are increasingly relevant for the design of humanmachine systems, as advances in technology begin to focus on exploiting individual differences.…”

Beyond the five-user assumption: Benefits of increased sample sizes in usability testing

“…The 5-user assumption arose from two sources: (1) secondary analyses of other testers' data by Nielsen (1993) and (2) "law-of-diminishing-returns" arguments made by Virzi (1992). Both Nielsen (1993) and Virzi sought to demonstrate that statistical rigor can be relaxed considerably in real-world usability testing.…”

“…Building on the foundational work of Nielsen (1993) and Virzi, it is appropriate to revisit the calculations and data on which the assumption was based. Virzi's (1992) It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more.…”

“…Woolrych and Cockton (2001), in their detailed deconstruction of Landauer and Nielsen's (1993) formula, confirmed the potential for overpredicting the reliability of small-sample usability test results, demonstrating the inflated fixed value recommended by Landauer and Nielsen for the probability that any user will find any problem. Nielsen (1993) and Virzi (1992) both made attempts to describe the limitations of their 5-user recommendations. Virzi indicated that "[s]ubjects should be run until the number of new problems uncovered drops to an acceptable level" (p. 467), leaving the practitioner to define "acceptable level."…”

“…Early studies supporting the assumption argued that just 5 participants could reveal about 80% of all usability problems that exist in a product (Nielsen, 1993;Virzi, 1992). This figure indicates a probability of the percentage of problems missed; there is, currently, no way to determine with reasonable certainty that any set of f ive tests matched those percentages, or which particular problems were revealed or missed (Woolrych & Cockton, 2001).…”