Estimating the number of usability problems affecting medical devices: modelling the discovery matrix

Abstract

Background Usability studies of medical devices are mandatory for market access. The studies’ goal is to identify and eliminate usability problems that could cause harm the user or limit the device’s effectiveness. In practice, human factor engineers study participants under actual conditions of use and list the problems encountered. This results in a binary discovery matrix in which each row corresponds to a participant, and each column corresponds to a usability problem. One of the main challenges in usability studies is estimating the total number of problems, in order to assess the completeness of the discovery process. Today’s margin-based methods fit the column sums to a binomial model of problem detection. However, the discovery matrix actually observed is truncated because of undiscovered problems, which corresponds to fitting the marginal sums without the zeros. Margin-based methods fail to overcome the bias related to truncation of the matrix. The objective of the present study was to develop and test a matrix-based method for estimating the total number of usability problems. Methods The matrix-based method models the likelihood of the discovery matrix, and allows one to account for all the available information. It also circumvents a drawback of margin-based methods by simultaneously estimating two unknown parameters: the probability of problem detection and the total number of problems. Furthermore, the matrix-based method takes account of a heterogeneous probability of detection, which better reflects a real-life setting. As suggested in the usability literature, a logit-normal prior for the probability of detection is selected. Results We assessed the matrix-based method’s performance in a range of settings reflecting real-life usability studies and with both homogeneous and heterogeneous probabilities of problem detection. In our simulations, the matrix-based method improved the estimation of the number of problems (in terms of bias, consistency, and coverage probability of the confidence interval) in a wide range of settings. We also applied our method to real data from a usability study of infusion pumps. Conclusions Our method should be applied by regulators and device manufacturers to estimate the number of usability problems using the set of statistical routines provided.