Reviewing and Extending the Five-User Assumption

Abstract

The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -which we call the 'Grounded Procedure' -is introduced and its value argued.

Occupational Therapists’ Views on Using a Virtual Reality Interior Design Application Within the Pre-Discharge Home Visit Process

“…The number of participants that took part in this study may be considered to be too small to make generalizations about OT perceptions of the use of VRIDA in the PHV process more generally. However, in accordance with recent research findings in the usability testing research domain, the number of participants that took part in this study exceeds the suggested threshold number of five participants that are necessary to provide useful and effective feedback when using the think-aloud protocol for interactive prototype evaluation [ 40 ]. In relation to the TAM model, it is noted that it has been advocated that the Human Activity Assistive Technology (HAAT) model integrates the social model of disability, concepts from occupational therapy theory, and principles of assistive technology adoption and abandonment.…”

“…A total of seven OTs were recruited and took part in the study. This number of participants is in excess of the recommended threshold of five participants typically required to carry out effective think-aloud interaction and usability testing [ 40 , 41 ]. Five of the participants were female and two were male.…”

Perceived usability of a store and forward telehealth platform for diagnosis and management of oral mucosal lesions: A cross-sectional study

“…Considering that the present study used a convenience sample, its power was calculated according to method proposed by Borsci et al [24]. To estimate the amount of problems reached by the sample in question, the following equation was used: D = 1-(1-p) n , where p represents the raw p-value, n indicate the sample size and D, the percentage of problems reached by the sample.…”

Local Standards for Sample Size at CHI

“…Because of this experience, they are viewed as trusted sources of valuable information about a topic. For example, we conducted a literature search for information about sample size with respect to usability studies and found highly expert recommendations ranging from 4 ± 1 (for think aloud studies [25]) to 10 ± 2 [21] with others recommending a grounded procedure which starts with an estimate, observes the data collected for the estimate and then reevaluates [11]. See [10, pg.…”