Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

Abstract

Study Objectives: To determine the reasons for inter-scorer variability in sleep staging of polysomnograms (PSGs). Methods: Fifty-six PSGs were scored (5-stage sleep scoring) by 2 experienced technologists, (first manual, M1). Months later, the technologists edited their own scoring (second manual, M2) based upon feedback from the investigators that highlighted differences between their scoring. The PSGs were then scored with an automatic system (Auto) and the technologists edited them, epoch-by-epoch (Edited-Auto). This resulted in 6 different manual scores for each PSG. Epochs were classified as scorer errors (one M1 score differed from the other 5 scores), scorer bias (all 3 scores of each technologist were similar, but differed from the other technologist) and equivocal (sleep scoring was inconsistent within and between technologists). Results: Percent agreement after M1 was 78.9% ± 9.0% and was unchanged after M2 (78.1% ± 9.7%) despite numerous edits (≈40/PSG) by the scorers. Agreement in Edited-Auto was higher (86.5% ± 6.4%, p < 1E−9). Scorer errors (< 2% of epochs) and scorer bias (3.5% ± 2.3% of epochs) together accounted for < 20% of M1 disagreements. A large number of epochs (92 ± 44/PSG) with scoring agreement in M1 were subsequently changed in M2 and/or EditedAuto. Equivocal epochs, which showed scoring inconsistency, accounted for 28% ± 12% of all epochs, and up to 76% of all epochs in individual patients. Disagreements were largely between awake/NREM, N1/N2, and N2/N3 sleep. Conclusion: Inter-scorer variability is largely due to epochs that are difficult to classify. Availability of digitally identified events (e.g., spindles) or calculated variables (e.g., depth of sleep, delta wave duration) during scoring may greatly reduce scoring variability.

The Accuracy, Night-to-Night Variability, and Stability of Frontopolar Sleep Electroencephalography Biomarkers

“…Our study supports previously published studies reporting a fairly broad range of interscorer variability. 24,25 Thus, our findings also underscore the difficulty in validating the accuracy of autostaging software against the current gold standard practice of technician-based visual scoring 24,[26][27][28] or by RPSGT certification, [29][30][31] even when the same equipment and software is used.…”

Performance of an Ambulatory DRY-EEG Device for Auditory Closed-Loop Stimulations in the Home Environment

“…The detection of N3 sleep, as compared to a PSG gold-standard, led to good results with a specificity of 0.90 and a sensitivity of 0.70 (Table I), which has to be put in perspective with the fact that the inter-scorer variability for sleep stage classification along the AASM rules is about 82% [33]. These results were obtained with dry frontal electrodes referred to mastoids whereas the PSG encompassed EEG, EOG and EMG.…”

Noncontact identification of sleep-disturbed breathing from smartphone-recorded sounds validated by polysomnography

“…While screening questionnaires may help identify at-risk individuals, e.g., the Berlin or STOP-BANG scores, such tools are not tailored to individual physiology and may have limited accuracy [17]. The PSG, while the gold standard for diagnosis, introduces limitations for screening [18, 19] including its in-laboratory setting, need for multiple physical connections, expense, inter-test variability including first-night effect, and variability in interpreting test results due to competing scoring criteria [20]. Improved, cost-efficient screening for SDB may enable rapid triage of high-risk individuals who are currently unscreened for gold standard PSG followed by prescription and titration of therapy as needed.…”

“…In this way, acoustic analysis also has the potential to augment in-laboratory or home sleep testing. Some inter-observer variability of sleep studies reflect “equivocal epochs” which account for > 25% of sleep, particularly in awake/NREM, N1/N2, and N2/N3 sleep [20]. Analysis of breaths could help to understand such epochs, and provide additional information to help in coding hypopnea [21] or arousals [12].…”