Toward Harmonized Treadmill-Based Validation of Step-Counting Wearable Technologies: A Scoping Review

Abstract

<jats:p><jats:bold>Background</jats:bold>: The authors conducted a scoping review as a first step toward establishing harmonized (ie, consistent and compatible), empirically based best practices for validating step-counting wearable technologies. <jats:bold>Purpose</jats:bold>: To catalog studies validating step-counting wearable technologies during treadmill ambulation. <jats:bold>Methods</jats:bold>: The authors searched PubMed and SPORTDiscus in August 2019 to identify treadmill-based validation studies that employed the criterion of directly observed (including video recorded) steps and cataloged study sample characteristics, protocol details, and analytical procedures. Where reported, speed- and wear location–specific mean absolute percentage error (MAPE) values were tabulated. Weighted median MAPE values were calculated by wear location and a 0.2-m/s speed increment. <jats:bold>Results</jats:bold>: Seventy-seven eligible studies were identified: most had samples averaging 54% (SD = 5%) female and 27 (5) years of age, treadmill protocols consisting of 3 to 5 bouts at speeds of 0.8 (0.1) to 1.6 (0.2) m/s, and reported measures of bias. Eleven studies provided MAPE values at treadmill speeds of 1.1 to 1.8 m/s; their weighted median MAPE values were 7% to 11% for wrist-worn, 1% to 4% for waist-worn, and ≤1% for thigh-worn devices. <jats:bold>Conclusions</jats:bold>: Despite divergent study methodologies, the authors identified common practices and summarized MAPE values representing device step-count accuracy during treadmill walking. These initial empirical findings should be further refined to ultimately establish harmonized best practices for validating wearable technologies.</jats:p>

Cadence (steps/min) and relative intensity in 21 to 60-year-olds: the CADENCE-adults study

“…Notably, prior research on wearable technologies has reported that wrist-worn devices can significantly over-or underestimate energy expenditure [37] and called for increased accuracy in wrist-worn devices due to low validity findings [38]. Specifically, a review by Moore et al [39] reported that median aggregated values of step count mean absolute percentage error (MAPE) error representing the comparison between direct observation and wearable technologies were higher for wrist-worn devices (MAPE = 7 to 11%) than for waist-worn (MAPE =1 to 4%), or thigh-worn (MAPE ≤1%).…”

A catalog of validity indices for step counting wearable technologies during treadmill walking: the CADENCE-Kids study

“…Also, children (5-9 years of age) walked with significantly different step time, cycle time, cycle frequency and cadence than their adult peers (19-32 years of age) in walking and running tests [16]. It is therefore critical to consider such threats to validity when comparing performance of step counting wearable technologies using standardized methods and validation metrics among a cohort of children [17].…”

“…The performance of step counting wearable technologies should then be compared to this criterion standard [18]. Standardized and harmonized validation indices, specifically accuracy, bias, and precision are also necessary to facilitate comparability of different types of wearable technologies [17,19,20]. The CTA [18], Welk et al [20], Walther et al [19], and a previous scoping review from our research group [17], recommended that accuracy (defined as the overall distance between estimated or observed values and the true value [19]) be determined using mean absolute percentage error (MAPE,), calculated as follows:…”

“…Tested speeds varied and ranged from 0.8 km/h (0.5 mph) to 9.7 km/h (6.0 mph). Also, none of these studies included all three measures of validity (i.e., accuracy, bias, and precision) or assessed the effect of age on wearable technology step counting performance [17].…”

Assessing Heart Rate Using Consumer Technology Association Standards

“…These standards provide guidance for researchers to evaluate research-grade devices as well as commercial devices used by the lay public. Since the CTA released their standards for protocol and validation, researchers have begun to acknowledge them in their designs for step count [6][7][8][9], and HR [10][11][12]. However, only limited studies have actually implemented these standards [6].…”