Recommendations for Assessment of the Reliability, Sensitivity, and Validity of Data Provided by Wearable Sensors Designed for Monitoring Physical Activity

Abstract

Although it is becoming increasingly popular to monitor parameters related to training, recovery, and health with wearable sensor technology (wearables), scientific evaluation of the reliability, sensitivity, and validity of such data is limited and, where available, has involved a wide variety of approaches. To improve the trustworthiness of data collected by wearables and facilitate comparisons, we have outlined recommendations for standardized evaluation. We discuss the wearable devices themselves, as well as experimental and statistical considerations. Adherence to these recommendations should be beneficial not only for the individual, but also for regulatory organizations and insurance companies.

Evaluation of pulse rate measurement with a wrist worn device during different tasks and physical activity

“…Additionally, there are methodological concerns to be addressed as some previous research validated multiple devices that were simultaneously worn on one wrist. This is not recommended for two main reasons; firstly, by fitting more than one device on the wrist, a correct placement according to the manufacturer's instructions is not warranted and, secondly, devices touching each other tap together with every movement (Duking, Fuss, Holmberg, & Sperlich, 2018;Duking, Hotho, Holmberg, Fuss, & Sperlich, 2016;Malone, Lovell, Varley, & Coutts, 2017). Two concerns that might affect the measurement output.…”

“…Furthermore, the majority of the previous validation studies was conducted in a laboratory setting. However, the wrist-worn pulse rate measurement systems are designed for monitoring general activity in various forms of exercise, intensity, positions, and while moving freely (Duking et al, 2018). For all the aforementioned reasons, there is still a lack of knowledge regarding the accuracy of the wrist-worn devices on the market that measure pulse rate using PPG and what exactly the influencing factors regarding accuracy are.…”

Sensor Reliability in Cyber-Physical Systems Using Internet-of-Things Data: A Review and Case Study

“…Certainly, the evaluation of sensor reliability is a highly important task, especially for Internet of Things (IoT) applications, and new performance indices need to be proposed [54]. Some recommendations for the standardized evaluation of sensor reliability are given in [55]. A recent review of analyses of the state of the art time delays, network size, energy efficiency, scalability, and reliability of mobile wireless sensor networks can be found in [56].…”

Evaluation of mobile applications for fitness training and physical activity in healthy low-trained people - A modular interdisciplinary framework

“…In addition, the graphical representation of error assessment also covers a wide range from simple bar charts to regression and Bland-Altman plots (Atkinson & Nevill, 1998). Concurrent criterion, content, and construct validity Düking et al (2018) propose recommendations for standardized evaluation of reliability, sensitivity, and validity of PA-monitoring wearable devices. In their checklist the authors identify 11 factors which are grouped into three categories (see Table 5).…”

“…However, these "aggressive and exaggerated claims" (Düking et al, 2018, p.1) have rarely been sufficiently validated (e.g., Halson et al, 2016;Peake, Kerr, & Sullivan, 2018;Romeo et al, 2019). In current reviews, two aspects are criticized, i.e., lack of user or consumer integration in the development process and lack of adequate validating research (e.g., Düking et al, 2018;Peake, Kerr, & Sullivan, 2018, Warraich, 2016. Halson et al (2016) enumerate numerous issues of mobile ICT regarding SPA and FT, ranging from technical issues like sensor placement and accuracy (see also Wahl et al, 2017) to ethical considerations.…”