Call to action: an open-source pipeline for standardized performance evaluation of sleep-tracking technology

Benedetti, Davide; Menghini, Luca; Vallat, Raphaël; Mallett, Remington; Kiss, Orsolya; Faraguna, Ugo; Baker, Fiona C.; Zambotti, Massimiliano de

doi:10.1093/sleep/zsac304

Cited by 5 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite such promising opportunities, wearable sleep trackers pose several challenges that should be carefully considered. First, their validity is uncertain and can vary across different devices and populations, with method comparison studies being increasingly needed as these technologies evolve (Benedetti et al, 2023; Depner et al, 2020; Menghini, Cellini, et al, 2021). The term ‘performance evaluation’ has been recently proposed instead of ‘validation’ precisely because the continuous update of device features prevents the research community from definitely establishing their validity (de Zambotti et al, 2022).…”

Section: Wearable Sleep Trackers In Applied Psychologymentioning

confidence: 99%

Is it Time to Include Wearable Sleep Trackers in the Applied Psychologists’ Toolbox?

Menghini,

Balducci,

de Zambotti

2024

Span. J. Psychol.

View full text Add to dashboard Cite

Wearable sleep trackers are increasingly used in applied psychology. Particularly, the recent boom in the fitness tracking industry has resulted in a number of relatively inexpensive consumer-oriented devices that further enlarge the potential applications of ambulatory sleep monitoring. While being largely positioned as wellness tools, wearable sleep trackers could be considered useful health devices supported by a growing number of independent peer-reviewed studies evaluating their accuracy. The inclusion of sensors that monitor cardiorespiratory physiology, diurnal activity data, and other environmental signals allows for a comprehensive and multidimensional approach to sleep health and its impact on psychological well-being. Moreover, the increasingly common combination of wearable trackers and experience sampling methods has the potential to uncover within-individual processes linking sleep to daily experiences, behaviors, and other psychosocial factors. Here, we provide a concise overview of the state-of-the-art, challenges, and opportunities of using wearable sleep-tracking technology in applied psychology. Specifically, we review key device profiles, capabilities, and limitations. By providing representative examples, we highlight how scholars and practitioners can fully exploit the potential of wearable sleep trackers while being aware of the most critical pitfalls characterizing these devices. Overall, consumer wearable sleep trackers are increasingly recognized as a valuable method to investigate, assess, and improve sleep health. Incorporating such devices in research and professional practice might significantly improve the quantity and quality of the collected information while opening the possibility of involving large samples over representative time periods. However, a rigorous and informed approach to their use is necessary.

show abstract

Section: Wearable Sleep Trackers In Applied Psychologymentioning

confidence: 99%

Is it Time to Include Wearable Sleep Trackers in the Applied Psychologists’ Toolbox?

Menghini,

Balducci,

de Zambotti

2024

Span. J. Psychol.

View full text Add to dashboard Cite

show abstract

“…When using various devices, there are problems related to differences in operation methods or algorithms [38]; even when using a single device, the measurement process or results may change because of changes in firmware or algorithms. Therefore, further research on standardized performance evaluation systems for sleep-tracking technology is required [39].…”

Section: Introductionmentioning

confidence: 99%

Exploring Variations in Sleep Perception: Comparative Study of Chatbot Sleep Logs and Fitbit Sleep Data

Jang,

Lee,

Son

et al. 2023

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background Patient-generated health data are important in the management of several diseases. Although there are limitations, information can be obtained using a wearable device and time-related information such as exercise time or sleep time can also be obtained. Fitbits can be used to acquire sleep onset, sleep offset, total sleep time (TST), and wakefulness after sleep onset (WASO) data, although there are limitations regarding the depth of sleep and satisfaction; therefore, the patient’s subjective response is still important information that cannot be replaced by wearable devices. Objective To effectively use patient-generated health data related to time such as sleep, it is first necessary to understand the characteristics of the time response recorded by the user. Therefore, the aim of this study was to analyze the characteristics of individuals’ time perception in comparison with wearable data. Methods Sleep data were acquired for 2 weeks using a Fitbit. Participants’ sleep records were collected daily through chatbot conversations while wearing the Fitbit, and the two sets of data were statistically compared. Results In total, 736 people aged 30-59 years were recruited for this study, and the sleep data of 543 people who wore a Fitbit and responded to the chatbot for more than 7 days on the same day were analyzed. Research participants tended to respond to sleep-related times on the hour or in 30-minute increments, and each participant responded within the range of 60-90 minutes from the value measured by the Fitbit. On average for all participants, the chat responses and the Fitbit data were similar within a difference of approximately 15 minutes. Regarding sleep onset, the participant response was 8 minutes and 39 seconds (SD 58 minutes) later than that of the Fitbit data, whereas with respect to sleep offset, the response was 5 minutes and 38 seconds (SD 57 minutes) earlier. The participants’ actual sleep time (AST) indicated in the chat was similar to that obtained by subtracting the WASO from the TST measured by the Fitbit. The AST was 13 minutes and 39 seconds (SD 87 minutes) longer than the time WASO was subtracted from the Fitbit TST. On days when the participants reported good sleep, they responded 19 (SD 90) minutes longer on the AST than the Fitbit data. However, for each sleep event, the probability that the participant’s AST was within ±30 and ±60 minutes of the Fitbit TST-WASO was 50.7% and 74.3%, respectively. Conclusions The chatbot sleep response and Fitbit measured time were similar on average and the study participants had a slight tendency to perceive a relatively long sleep time if the quality of sleep was self-reported as good. However, on a participant-by-participant basis, it was difficult to predict participants’ sleep duration responses with Fitbit data. Individual variations in sleep time perception significantly affect patient responses related to sleep, revealing the limitations of objective measures obtained through wearable devices.

show abstract

State of the science and recommendations for using wearable technology in sleep and circadian research

de Zambotti,

Goldstein,

Cook

et al. 2023

Sleep

View full text Add to dashboard Cite

Wearable sleep-tracking technology is of growing use in the sleep and circadian fields, including for applications across other disciplines, inclusive of a variety of disease states. Patients increasingly present sleep data derived from their wearable devices to their providers and the ever-increasing availability of commercial devices and new-generation research/clinical tools has led to the wide adoption of wearables in research, which has become even more relevant given the discontinuation of the Philips Respironics Actiwatch. Standards for evaluating the performance of wearable sleep-tracking devices have been introduced and the available evidence suggests that consumer-grade devices exceed the performance of traditional actigraphy in assessing sleep as defined by polysomnogram. However, clear limitations exist, for example, the misclassification of wakefulness during the sleep period, problems with sleep tracking outside of the main sleep bout or nighttime period, artifacts, and unclear translation of performance to individuals with certain characteristics or comorbidities. This is of particular relevance when person-specific factors (like skin color or obesity) negatively impact sensor performance with the potential downstream impact of augmenting already existing healthcare disparities. However, wearable sleep-tracking technology holds great promise for our field, given features distinct from traditional actigraphy such as measurement of autonomic parameters, estimation of circadian features, and the potential to integrate other self-reported, objective and passively recorded health indicators. Scientists face numerous decision points and barriers when incorporating traditional actigraphy, consumer-grade multi-sensor devices, or contemporary research/clinical-grade sleep trackers into their research. Considerations include wearable device capabilities and performance, target population and goals of the study, wearable device outputs and availability of raw and aggregate data, and data extraction, processing, and analysis. Given the difficulties in the implementation and utilization of wearable sleep-tracking technology in real-world research and clinical settings, the following State of the Science review requested by the Sleep Research Society aims to address the following questions. What data can wearable sleep-tracking devices provide? How accurate are these data? What should be taken into account when incorporating wearable sleep-tracking devices into research? These outstanding questions and surrounding considerations motivated this work, outlining practical recommendations for using wearable technology in sleep and circadian research.

show abstract

Call to action: an open-source pipeline for standardized performance evaluation of sleep-tracking technology

Cited by 5 publications

References 11 publications

Is it Time to Include Wearable Sleep Trackers in the Applied Psychologists’ Toolbox?

Is it Time to Include Wearable Sleep Trackers in the Applied Psychologists’ Toolbox?

Exploring Variations in Sleep Perception: Comparative Study of Chatbot Sleep Logs and Fitbit Sleep Data

State of the science and recommendations for using wearable technology in sleep and circadian research

Contact Info

Product

Resources

About