Sample Sizes for Usability Studies: Additional Considerations

Lewis, James R.

doi:10.1177/001872089403600215

Cited by 255 publications

(177 citation statements)

References 15 publications

Supporting

Mentioning

160

Contrasting

Unclassified

Order By: Relevance

“…In addition, a sequential heuristic evaluation based on this definition was demonstrated to be an efficient method for evaluating how well the usability of a newly designed shopping mall conforms to universal design. This study substantiated what Lewis (1994) stated, that the likelihood of the discovery of additional problems increases with the iterative sequence approach. With the help of the three sessions of this heuristic evaluation, usability problems were reported that could not have been detected solely through an analysis of the construction drawings.…”

Section: Discussionsupporting

confidence: 88%

An interdisciplinary heuristic evaluation method for universal building design

Afacan

Erbuğ

2009

Applied Ergonomics

View full text Add to dashboard Cite

a b s t r a c tThis study highlights how heuristic evaluation as a usability evaluation method can feed into current building design practice to conform to universal design principles. It provides a definition of universal usability that is applicable to an architectural design context. It takes the seven universal design principles as a set of heuristics and applies an iterative sequence of heuristic evaluation in a shopping mall, aiming to achieve a cost-effective evaluation process. The evaluation was composed of three consecutive sessions. First, five evaluators from different professions were interviewed regarding the construction drawings in terms of universal design principles. Then, each evaluator was asked to perform the predefined task scenarios. In subsequent interviews, the evaluators were asked to reanalyze the construction drawings. The results showed that heuristic evaluation could successfully integrate universal usability into current building design practice in two ways: (i) it promoted an iterative evaluation process combined with multi-sessions rather than relying on one evaluator and on one evaluation session to find the maximum number of usability problems, and (ii) it highlighted the necessity of an interdisciplinary ad hoc committee regarding the heuristic abilities of each profession. A multi-session and interdisciplinary heuristic evaluation method can save both the project budget and the required time, while ensuring a reduced error rate for the universal usage of the built environments.

show abstract

Section: Discussionsupporting

confidence: 88%

An interdisciplinary heuristic evaluation method for universal building design

Afacan

Erbuğ

2009

Applied Ergonomics

View full text Add to dashboard Cite

show abstract

“…Sample size was estimated based on previous studies showing that 5 participants are sufficient to detect 80% of usability problems [13,22,23]. Considering a response rate of 33% [24], the sample size was defined as 8 participants per group -a highly experienced group and an inexperienced group.…”

Section: Sample Size Calculationmentioning

confidence: 99%

Usability of a store and forward telehealth platform for diagnosis and management of oral mucosal lesions: a cross-sectional study (Preprint)

Gon�alves¹,

Martins²,

Martins³

et al. 2018

Preprint

View full text Add to dashboard Cite

Background: Telediagnosis is a promising tool in situations of limited access to specialty care. Considering that, an oral health telediagnosis platform (EstomatoNet) was created in the state of Rio Grande do Sul, Brazil. EstomatoNet provides specialist support over a web-based platform to primary care dentists for diagnosis of oral lesions. Objective: To evaluate the usability of EstomatoNet and to identify user perceptions regarding their expectations and difficulties with the system. Methods: Sixteen dentists were selected for the study: 8 were frequent users of EstomatoNet and 8 were residents who had never used the Platform. To assess usability, participants were required to request telediagnosis support for a fictional case provided by the research team. During the process of uploading the information and sending the request, users were asked to "think out loud," expressing their perceptions. The session was observed by an examiner with remote access to the user's screen (via Skype). After the simulation, users completed the System Usability Scale (SyUS). Results:The mean SyUS score assigned by frequent users was 84.7±6.6, vs. 82.2±9.3 for residents (satisfactory usability: score above 68). The difference between the groups was not statistically significant (Student t test, P = .55). The inexperienced group took longer (347.1s±101.1) to complete the task than frequent users (252.8s±80.3), however the difference between the groups was not statistically significant (Student t test, P = .06). In their subjective evaluation, users suggested the inclusion of a field to add further information on outcomes and resolution of the case and changes in the position of the "Send" button to improve work flow. Conclusions:The present results indicate good usability of EstomatoNet. The Platform seems to meet the needs of users regardless of how experienced they are; nevertheless, a few minor changes in some steps would improve the tool.

show abstract

“…The question of how many users to include in usability testing is one that has been well-debated in HCI [20][21][22][23]. The first studies in this area mostly focused on determining the cost-benefit of web interface analysis by estimating the return on investment to justify the cost of usability assessment [24].…”

Section: Usability Testing Of Medical Devices: Estimating the Sample mentioning

confidence: 99%

“…Lewis [20,41] applied this in conjunction with the GT model procedure and showed that it delivers a conservative and reliable value of p.…”

Section: Usability Testing Of Medical Devices: Estimating the Sample mentioning

confidence: 99%

How many testers are needed to assure the usability of medical devices?

Borsci

Macredie

Martin

2014

Expert Review of Medical Devices

View full text Add to dashboard Cite

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription.  The assessment of usability and its place in the management of product safety is an increasingly important aspect of medical device development. Medical device manufacturers, especially in small companies, do not have sufficient expertise or knowledge about usability and human factors and it is hard for them suitably to address all of the steps required by relevant standards. Despite a rigorous framework for designing and assessing a product, an increasing number of medical devices are recalled each year because of safety concerns. There is substantial evidence that testing a device with mandated sample sizes could lead manufacturers to evaluate a product without having real control over the reliability of the assessment. A new approach to usability data management applied in this paper, called the 'Grounded Procedure', drives manufacturers to estimate the sample size needed to identify a given proportion of interaction problems and to inform critical product decisions. Using the Grounded Procedure could enable manufacturers to increase the usability and the safety of their medical devices and help practitioners to check the representativeness of the evaluation cohorts, to analyse significance of specific usability problems, and to rethink the user selection criteria for validation testing. Summary 3Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing whilst managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product's safety. This paper reviews the FDA's indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90-97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA's own concerns expressed in [1]) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach -illustrated in this paper through a case study -called the 'Grounded Procedure'.4

show abstract

Sample Sizes for Usability Studies: Additional Considerations

Cited by 255 publications

References 15 publications

An interdisciplinary heuristic evaluation method for universal building design

An interdisciplinary heuristic evaluation method for universal building design

Usability of a store and forward telehealth platform for diagnosis and management of oral mucosal lesions: a cross-sectional study (Preprint)

How many testers are needed to assure the usability of medical devices?

Contact Info

Product

Resources

About