Interrater reliability for sleep scoring according to the Rechtschaffen & Kales and the new AASM standard

Abstract

SUMMARY Interrater variability of sleep stage scorings has an essential impact not only on the reading of polysomnographic sleep studies (PSGs) for clinical trials but also on the evaluation of patientsÕ sleep. With the introduction of a new standard for sleep stage scorings (AASM standard) there is a need for studies on interrater reliability (IRR). The SIESTA database resulting from an EU-funded project provides a large number of studies (n = 72; 56 healthy controls and 16 subjects with different sleep disorders, mean age ± SD: 57.7 ± 18.7, 34 females) for which scorings according to both standards (AASM and R&K) were done. Differences in IRR were analysed at two levels: (1) based on quantitative sleep parameter by means of intraclass correlations; and (2) based on an epoch-by-epoch comparison by means of CohenÕs kappa and FleissÕ kappa. The overall agreement was for the AASM standard 82.0% (CohenÕs kappa = 0.76) and for the R&K standard 80.6% (CohenÕs kappa = 0.68). Agreements increased from R&K to AASM for all sleep stages, except N2. The results of this study underline that the modification of the scoring rules improve IRR as a result of the integration of occipital, central and frontal leads on the one hand, but decline IRR on the other hand specifically for N2, due to the new rule that cortical arousals with or without concurrent increase in submental electromyogram are critical events for the end of N2.k e y w o r d s AASM scoring standard, interrater reliability, Rechtschaffen and Kales, SIESTA project, sleep stage scoring

The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring

“…This is remarkably similar to the 83% agreement reported by Ruehland and colleagues 6 using the AASM Manual scoring methodology, a study that compared 2 experienced scorers and 1 scorer with a single year of experience. Danker-Hopfe and colleagues 5 reported 82% agreement using a pool of 7 experienced scorers scoring 72 records. This provides encouraging news in that agreement among a large group of scorers is similar to agreement between small samples of experienced scorers.…”

“…Danker-Hopfe and colleagues found that use of the AASM Manual resulted in slightly improved inter-rater reliability to 82.0% compared to 80.6% using R & K rules. 5 Her study was based on 72 recordings that were scored with both methods by a pool of 7 experienced scorers.The AASM inter-scorer reliability (ISR) program was developed to aid sleep centers in fulfi lling accreditation standards. The standards require that a sample of randomly chosen records be scored by the center director and each of the technologists involved in record scoring.…”

“…Danker-Hopfe and colleagues found that use of the AASM Manual resulted in slightly improved inter-rater reliability to 82.0% compared to 80.6% using R & K rules. 5 Her study was based on 72 recordings that were scored with both methods by a pool of 7 experienced scorers.…”

The AASM Recommended and Acceptable EEG Montages are Comparable for the Staging of Sleep and Scoring of EEG Arousals

“…The absolute agreement and hierarchy of staging agreement associated with each stage was consistent with the literature. 20,26 Agreement was lowest in staging N1 sleep, better with stages N2 and N3, and highest with staging wakefulness and REM sleep.The finding that the scoring of EEG arousals was similar for both EEG montages was not really surprising. Since the scoring of EEG arousals has quite poor agreement, it was unlikely that the number of PSGs used in this study would have enough power to find differences between the two EEG montages.…”

Automatic sleep staging using state machine-controlled decision trees

“…This would not only save time and costs associated with sleep testing but also make the tests more accessible to a larger population. Further, this also helps to reduce interrater disagreement [3] as well as subjective error in human scoring and improve consistency between different tests. These motivations lead to the research and development of algorithms for automatic scoring of sleep stages.…”