About the Accuracy and Problems of Consumer Devices in the Assessment of Sleep

Abstract

Commercial sleep devices and mobile-phone applications for scoring sleep are gaining ground. In order to provide reliable information about the quantity and/or quality of sleep, their performance needs to be assessed against the current gold-standard, i.e. polysomnography (PSG; measuring brain, eye and muscle activity). We here assessed some commercially available sleep trackers, namely; a commercial activity tracker: Mi band (Xiaomi, BJ, CHN), a scientific actigraph: Motionwatch 8 (CamNTech, CB, UK), and a much used sleep application: Sleep Cycle (Northcube, GOT, SE). We recorded 27 nights in healthy sleepers using PSG and these devices. Surprisingly, all devices had very poor agreement with the gold standard. Sleep parameter comparisons revealed that specifically the Mi band and the sleep cycle application had difficulties in detecting wake periods which negatively affected the total sleep time and sleep efficiency estimations. However, all 3 devices were good in detecting the most basic parameter, the actual time in bed. In summary, our results suggest that, to-date; available sleep trackers do not provide meaningful sleep analysis but may be interesting for simply tracking times in bed. A much closer interaction with the scientific field seems necessary if reliable information shall be derived from such devices in the future.

Evaluation of a Low-Cost Commercial Actigraph and Its Potential Use in Detecting Cultural Variations in Physical Activity and Sleep

“…In Study 1, we evaluated the performance of MB to objectively measure physical activity and sleep measures. The low-cost of MB in combination with its promising performance in extracting physical activity and sleep-related measures [ 23 , 24 , 29 , 30 ] endorsed the utilization of this device for a cross-cultural actigraphy study. We revealed that step counts were not influenced by the exact actigraph wrist position, neither for MB, nor for the GT3X.…”

“…Furthermore, we considered the use of subjective sleep reports for evaluating the performance of MB in extracting sleep-related variables as a limitation of the current study, but this was an ad hoc study during the COVID-19 pandemic that made the use of PSG impossible. Future studies with a focus on assessing the performance of low-cost actigraphs in extracting sleep parameters should include PSG as the “gold-standard”, as has been previously done [ 30 , 54 ]. In the current study, we used the Xiaomi Mi Band Model 3, which to our knowledge has no major hardware/sensor changes compared to the Xiaomi Band Model 2, so discrepancies between the studies are unlikely to be explained by differences in the hardware of the Xiaomi models [ 30 ].…”

“…Future studies with a focus on assessing the performance of low-cost actigraphs in extracting sleep parameters should include PSG as the “gold-standard”, as has been previously done [ 30 , 54 ]. In the current study, we used the Xiaomi Mi Band Model 3, which to our knowledge has no major hardware/sensor changes compared to the Xiaomi Band Model 2, so discrepancies between the studies are unlikely to be explained by differences in the hardware of the Xiaomi models [ 30 ]. However, since the MB software updates as it advances to the next model, it is possible that some of the above-mentioned discrepancies between the current and previous studies were due to software upgrades.…”

“…In both conditions, the MB performed better than all high-cost smartwatches, but worse than the GT3X. The performance of MB to track sleep has been evaluated against PSG in an epoch-by-epoch analysis only in one study so far [ 30 ]. Specifically, Ameen et al [ 30 ] found a mean overestimation of the MB device for a total-sleep-time of 69 min.…”

“…The performance of MB to track sleep has been evaluated against PSG in an epoch-by-epoch analysis only in one study so far [ 30 ]. Specifically, Ameen et al [ 30 ] found a mean overestimation of the MB device for a total-sleep-time of 69 min. In conclusion, the MB appears to be a promising low-cost smartwatch for measuring physical activity with unclear or yet ambiguous results for sleep-related measures.…”