A comprehensive guideline for Bland–Altman and intra class correlation calculations to properly compare two methods of measurement and interpret findings

Haghayegh, Shahab; Kang, Hyeon‐Ah; Khoshnevis, Sepideh; Smolensky, Michael H.; Diller, Kenneth R.

doi:10.1088/1361-6579/ab86d6

Cited by 73 publications

(77 citation statements)

References 28 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In other words, a difference of 30 minutes or more between the two methods in the estimation of sleep parameters was considered of clinical significance. 19 , 20 In our study, the differences between the two devices fell within 30-minutes for all the parameters examined, except for WASO. Despite our findings, whether a 30-minute cutoff is acceptable for the interchangeable use of different sleep assessment modalities remains a question that needs to be answered from the clinical perspective.…”

Section: Discussionsupporting

confidence: 52%

“…Still, the deviations were not substantial (Q-Q plots can be found in Supplementary Figure 1 ). When analyzing Bland–Altman plots, we report the 95% limits of agreement, the minimal detectable change, and the minimal clinically important change as recommended by Haghayegh et al 19 The 95% limits of agreement between the two devices are defined as the mean difference ± 1.96 SD of the differences. The minimal detectable change is defined as the smallest amount of change in the score detected by a method, independent of measurement error.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Emfit Bed Sensor Activity Shows Strong Agreement with Wrist Actigraphy for the Assessment of Sleep in the Home Setting

Piantino¹,

Luther²,

Reynolds³

et al. 2021

NSS

View full text Add to dashboard Cite

Wrist-worn actigraphy via research-grade devices, a well-established approach to the assessment of rest-activity, is limited by poor compliance, battery life, and lack of direct evidence for time spent physically in the bed. A non-invasive bed sensor (Emfit) may provide advantages over actigraphy for long-term sleep assessment in the home. This study compared sleep-wake measurements between this sensor and a validated actigraph. Patients and Methods: Thirty healthy subjects (6 to 54 years) underwent simultaneous monitoring with both devices for 14 days and filled out a daily sleep diary. Parameters included bed entry time, sleep start, sleep end, bed exit time, rest interval duration, and wake after sleep onset (WASO). The agreement between the two devices was measured using Bland-Altman plots and inter-class correlation coefficients (ICC). In addition, sensitivity, specificity, and accuracy were obtained from epoch-by-epoch comparisons of Emfit and actigraphy. Results: Fifteen percent of the subjects reported that wearing the actigraph was a burden. None reported that using the bed sensor was a burden. The minimal detectable change between Emfit and actigraphy was 11 minutes for bed entry time, 14 minutes for sleep start, 14 minutes for sleep end, 10 minutes for bed exit time, 20 minutes for rest interval duration, and 110 minutes for WASO. Inter-class correlation coefficients revealed an excellent agreement for all sleep parameters (ICC=0.99, 95% CI 98-99) except for WASO (ICC=0.46, 95% CI 0.33-0.56). Sensitivity, specificity, and accuracy were 0.62, 0.93, and 0.88, respectively. Kappa correlation analysis revealed a moderate correlation between the two devices (κ=0.55, p<0.0001). Conclusion:Emfit is an acceptable alternative to actigraphy for the estimation of bed entry time, sleep start, sleep end, bed exit time, and rest interval duration. However, WASO estimates are poorly correlated between the two devices. Emfit may offer methodological advantages in situations where actigraphy is challenging to implement.

show abstract

Section: Discussionsupporting

confidence: 52%

Section: Methodsmentioning

confidence: 99%

Emfit Bed Sensor Activity Shows Strong Agreement with Wrist Actigraphy for the Assessment of Sleep in the Home Setting

Piantino¹,

Luther²,

Reynolds³

et al. 2021

NSS

View full text Add to dashboard Cite

show abstract

“…With our study design, the Bland-Altman plot, as a measure of agreement, was appropriate to perform within-subject comparisons for the two methods. 43 The Kappa agreement between OSASS and technologist 1 and technologist 2 was 0.81 and 0.84, respectively, which suggests that OSASS is capable of reasonably estimating OSA severity. These results show that OSASS generalizes reasonably well.…”

Section: Dovepressmentioning

confidence: 87%

Deep Learning for Diagnosis and Classification of Obstructive Sleep Apnea: A Nasal Airflow-Based Multi-Resolution Residual Network

Yue¹,

Yang²,

Wu³

et al. 2021

NSS

View full text Add to dashboard Cite

This study evaluated a novel approach for diagnosis and classification of obstructive sleep apnea (OSA), called Obstructive Sleep Apnea Smart System (OSASS), using residual networks and single-channel nasal pressure airflow signals. Methods: Data were collected from the sleep center of the First Affiliated Hospital, Sun Yat-sen University, and the Integrative Department of Guangdong Province Traditional Chinese Medical Hospital. We developed a new model called the multi-resolution residual network (Mr-ResNet) based on a residual network to detect nasal pressure airflow signals recorded by polysomnography (PSG) automatically. The performance of the model was assessed by its sensitivity, specificity, accuracy, and F1-score. We built OSASS based on Mr-ResNet to estimate the apnea-hypopnea index (AHI) and to classify the severity of OSA, and compared the agreement between OSASS output and the registered polysomnographic technologist (RPSGT) score, assessed by two technologists. Results: In the primary test set, the sensitivity, specificity, accuracy, and F1-score of Mr-ResNet were 90.8%, 90.5%, 91.2%, and 90.5%, respectively. In the independent test set, the Spearman correlation for AHI between OSASS and the RPSGT score determined by two technologists was 0.94 (p < 0.001) and 0.96 (p < 0.001), respectively. Cohen's Kappa scores for classification between OSASS and the two technologists' scores were 0.81 and 0.84, respectively. Conclusion: Our results indicated that OSASS can automatically diagnose and classify OSA using signals from a single-channel nasal pressure airflow, which is consistent with polysomnographic technologists' findings. Thus, OSASS holds promise for clinical application.

show abstract

“…the absence of measurement error) [15]. Thus, in the present article "performance" refers to the qualities of a sleep tracker describing its measurement error (reliability and accuracy), as quantified by the agreement with a reference method [16,17].…”

Section: Step-by-step Guidelines For Testing the Performance Of Sleep-trackersmentioning

confidence: 99%

A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code

Menghini

Cellini

Goldstone

et al. 2020

Sleep

101

View full text Add to dashboard Cite

Sleep-tracking devices, particularly within the consumer sleep technology (CST) space, are increasingly used in both research and clinical settings, providing new opportunities for large-scale data collection in highly ecological conditions. Due to the fast pace of the CST industry combined with the lack of a standardized framework to evaluate the performance of sleep trackers, their accuracy and reliability in measuring sleep remains largely unknown. Here, we provide a step-by-step analytical framework for evaluating the performance of sleep trackers (including standard actigraphy), as compared to gold-standard polysomnography (PSG) or other reference methods. The analytical guidelines are based on recent recommendations for evaluating and using CST from our group and others (de Zambotti, Cellini, Goldstone, Colrain & Baker, 2019; Depner et al., 2019), and include raw data organization as well as critical analytical procedures, including discrepancy analysis, Bland-Altman plots, and epoch-by-epoch analysis. Analytical steps are accompanied by open-source R functions (depicted at https://sri-human-sleep.github.io/sleep-trackers-performance/AnalyticalPipeline_v1.0.0.html). In addition, an empirical sample dataset is used to describe and discuss the main outcomes of the proposed pipeline. The guidelines and the accompanying functions are aimed at standardizing the testing of CSTs performance, to not only increase the replicability of validation studies, but also to provide ready-to-use tools to researchers and clinicians. All in all, this work can help to increase the efficiency, interpretation, and quality of validation studies, and to improve the informed adoption of CST in research and clinical settings.

show abstract

A comprehensive guideline for Bland–Altman and intra class correlation calculations to properly compare two methods of measurement and interpret findings

Cited by 73 publications

References 28 publications

Emfit Bed Sensor Activity Shows Strong Agreement with Wrist Actigraphy for the Assessment of Sleep in the Home Setting

Emfit Bed Sensor Activity Shows Strong Agreement with Wrist Actigraphy for the Assessment of Sleep in the Home Setting

Deep Learning for Diagnosis and Classification of Obstructive Sleep Apnea: A Nasal Airflow-Based Multi-Resolution Residual Network

A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code

Contact Info

Product

Resources

About