With how many users should you test a medical infusion pump? Sampling strategies for usability tests on high-risk systems

Abstract

Usability testing is recognized as an effective means to improve the usability of medical devices and prevent harm for patients and users. Effectiveness of problem discovery in usability testing strongly depends on size and representativeness of the sample. We introduce the late control strategy, which is to continuously monitor effectiveness of a study towards a preset target. A statistical model, the LNB(zt) model, is presented, supporting the late control strategy. We report on a case study, where a prototype medical infusion pump underwent a usability test with 34 users. On the data obtained in this study, the LNB(zt) model is evaluated and compared against earlier prediction models. The LNB(zt) model fits the data much better than previously suggested approaches and improves prediction. We measure the effectiveness of problem identification, and observe that it is lower than is suggested by much of the literature. Larger sample sizes seem to be in order. In addition, the testing process showed high levels of uncertainty and volatility at small to moderate sample sizes, partly due to users' individual differences. In reaction, we propose the idiosyncrasy score as a means to obtain representative samples. Statistical programs are provided to assist practitioners and researchers in applying the late control strategy.

Combining situated Cognitive Engineering with a novel testing method in a case study comparing two infusion pump interfaces

“…This study also showed that following a standardized Human Factors design approach does not automatically result in an exhaustive detection of usability problems (see also Schmettow et al, 2013) and a complete elimination of use errors.…”

Low Cost Command and Control System for Automated Infusion Devices

“…The block diagram of the virtual instrument contains three different operational blocks, each of them having a precise purpose in the functionality of the experimental device: -the pulse measurement and display block (1); -the data processing and conditioning block (2); -the actuation control block (3).…”

“…The development of integrated medical systems used nowadays includes automated syringe pumps, controlled perfusion devices, peristaltic pumps or many other self-operated instruments that can ease the work of medical personnel. [1] [2] A variety of sensors, electronic components, electric drivers and motors are integrant parts of the mentioned equipment. For that reason, in various cases, the real usage of the integrated medical device is reduced and the costs are high in respect with the real necessity of the product.…”

Estimating the number of usability problems affecting medical devices: modelling the discovery matrix

“…Using the zero truncation technique presented in equation 8, Schmettow developed the logit-normal binomial zero truncated (LNBzt) model and applied it to the usability of medical infusion pumps [24]. To the best of our knowledge, this model is the only one that accounts for both heterogeneity and unobserved problems.…”

“…In these five datasets, the number of participants ranged from 15 to 76. Previous studies of these datasets [7,18,22,24] demonstrated that the probability of problem detection was heterogeneous. As suggested by the results of the simulation study, the methods not taking account of heterogeneity considered that the discovery process was complete or very close to being complete for all datasets (except MACERR: see below).…”