The Bootstrap Discovery Behaviour Model

ACM Trans. Comput.-Hum. Interact.

Macredie

Barnett

et al. 2013

Self Cite

The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach -which we call the 'Grounded Procedure' -is introduced and its value argued.

Reviewing and Extending the Five-User Assumption

ACM Trans. Comput.-Hum. Interact.

Macredie

Barnett

et al. 2013

Self Cite

How many testers are needed to assure the usability of medical devices?

Expert Review of Medical Devices

Macredie

Martin

2014

Self Cite

A note on versions:The version presented here may differ from the published version or from the version of record. If you wish to cite this item you are advised to consult the publisher's version. Please see the repository url above for details on accessing the published version and note that access may require a subscription.  The assessment of usability and its place in the management of product safety is an increasingly important aspect of medical device development. Medical device manufacturers, especially in small companies, do not have sufficient expertise or knowledge about usability and human factors and it is hard for them suitably to address all of the steps required by relevant standards. Despite a rigorous framework for designing and assessing a product, an increasing number of medical devices are recalled each year because of safety concerns. There is substantial evidence that testing a device with mandated sample sizes could lead manufacturers to evaluate a product without having real control over the reliability of the assessment. A new approach to usability data management applied in this paper, called the 'Grounded Procedure', drives manufacturers to estimate the sample size needed to identify a given proportion of interaction problems and to inform critical product decisions. Using the Grounded Procedure could enable manufacturers to increase the usability and the safety of their medical devices and help practitioners to check the representativeness of the evaluation cohorts, to analyse significance of specific usability problems, and to rethink the user selection criteria for validation testing. Summary 3Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing whilst managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product's safety. This paper reviews the FDA's indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90-97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA's own concerns expressed in [1]) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach -illustrated in this paper through a case study -called the 'Grounded Procedure'.4

Providing assistive technology in Italy: the perceived delivery process quality as affecting abandonment

Federici

Disability and Rehabilitation: Assistive Technology

2014

Self Cite