Measurement Uncertainty in Clinical Validation Studies of Sensors

Abstract

Accurate clinical sensors and devices are essential to support optimal medical decision-making, and accuracy can be demonstrated through the conduct of clinical validation studies using validated reference sensors and/or devices for comparison. Typically unmeasurable, the true reference value can be substituted with an accepted physiological measurement with an associated uncertainty. We describe a basic model of measurement uncertainty that specifies the factors that may degrade the accuracy of an observed measurement value from a sensor, and we detail validation study design strategies that may be used to quantify and minimize these uncertainties. In addition, we describe a model that extends the observed measurement uncertainty to the resultant clinical decision and the factors that may impact the uncertainty of this decision. Clinical validation studies should be designed to estimate and minimize uncertainty that is unrelated to the sensor accuracy. The contribution of measurement observation uncertainty to clinical decision-making should be minimized but also acknowledged and incorporated into the clinical decision-making process.

Repeatability of Pulse Oximetry Measurements in Children During Triage in 2 Ugandan Hospitals

“…ICC values have previously been defined as the following: less than 0.2—slight repeatability, between 0.2 and 0.4—low repeatability, between 0.4 and 0.7—moderate repeatability, between 0.7 and 0.9—high repeatability, and greater than 0.9—very high repeatability. 24 Using this scale, this study showed that repeatability increased from low to moderate when using 35 seconds of data versus a shorter time frame. A further increase in high repeatability occurred when only high-quality data were used to calculate the median reported SpO2.…”

“…These factors include the short-term variation within the patient (within-subject variation), the variations in how observers perform the measurement (within- and between-observer variation), and other variations in filtering, averaging, or rounding of measurements. 24 …”

“…Other models of pulse oximeters may have different results, especially if they do not provide adequate feedback to the user about the current quality of the measurement. 24 In addition, the iSpO2 may not be the ideal device for many low-resource settings as it requires a mobile device and costs US$330. Lastly, poor repeatability of pulse oximetry measurements may not necessarily indicate poor accuracy, and good repeatability, on the other hand, does not necessarily equal good accuracy; one should not be taken as a substitute for the other as there are many forms of uncertainty, including user’s technique and fluctuations in vital signs between measurements.…”

“…Lastly, poor repeatability of pulse oximetry measurements may not necessarily indicate poor accuracy, and good repeatability, on the other hand, does not necessarily equal good accuracy; one should not be taken as a substitute for the other as there are many forms of uncertainty, including user’s technique and fluctuations in vital signs between measurements. 24 …”

“…These factors include the short-term variation within the patient (withinsubject variation), the variations in how observers perform the measurement (within-and betweenobserver variation), and other variations in filtering, averaging, or rounding of measurements. 24 It may take time for the measurement to stabilize when applying a pulse oximeter to a child due Abbreviations: L95, lower limit of confidence interval; LoA, limit of agreement; U95, upper limit of confidence interval. a N¼3,823.…”