Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms

Abstract

The validity of using electroencephalograms (EEGs) to diagnose epilepsy requires reliable detection of interictal epileptiform discharges (IEDs). Prior interrater reliability (IRR) studies are limited by small samples and selection bias.OBJECTIVE To assess the reliability of experts in detecting IEDs in routine EEGs. DESIGN, SETTING, AND PARTICIPANTSThis prospective analysis conducted in 2 phases included as participants physicians with at least 1 year of subspecialty training in clinical neurophysiology. In phase 1, 9 experts independently identified candidate IEDs in 991 EEGs (1 expert per EEG) reported in the medical record to contain at least 1 IED, yielding 87 636 candidate IEDs. In phase 2, the candidate IEDs were clustered into groups with distinct morphological features, yielding 12 602 clusters, and a representative candidate IED was selected from each cluster. We added 660 waveforms (11 random samples each from 60 randomly selected EEGs reported as being free of IEDs) as negative controls. Eight experts independently scored all 13 262 candidates as IEDs or non-IEDs. The 1051 EEGs in the study were recorded at the Massachusetts General Hospital between 2012 and 2016.MAIN OUTCOMES AND MEASURES Primary outcome measures were percentage of agreement (PA) and beyond-chance agreement (Gwet κ) for individual IEDs (IED-wise IRR) and for whether an EEG contained any IEDs (EEG-wise IRR). Secondary outcomes were the correlations between numbers of IEDs marked by experts across cases, calibration of expert scoring to group consensus, and receiver operating characteristic analysis of how well multivariate logistic regression models may account for differences in the IED scoring behavior between experts. RESULTS Among the 1051 EEGs assessed in the study, 540 (51.4%) were those of females and 511 (48.6%) were those of males. In phase 1, 9 experts each marked potential IEDs in a median of 65 (interquartile range [IQR], 28-332) EEGs. The total number of IED candidates marked was 87 636. Expert IRR for the 13 262 individually annotated IED candidates was fair, with the mean PA being 72.4% (95% CI, 67.0%-77.8%) and mean κ being 48.7% (95% CI, 37.3%-60.1%). The EEG-wise IRR was substantial, with the mean PA being 80.9% (95% CI, 76.2%-85.7%) and mean κ being 69.4% (95% CI, 60.3%-78.5%). A statistical model based on waveform morphological features, when provided with individualized thresholds, explained the median binary scores of all experts with a high degree of accuracy of 80% (range, 73%-88%).CONCLUSIONS AND RELEVANCE This study's findings suggest that experts can identify whether EEGs contain IEDs with substantial reliability. Lower reliability regarding individual IEDs may be largely explained by various experts applying different thresholds to a common underlying statistical model.

Persistent abnormalities in Rolandic thalamocortical white matter circuits in childhood epilepsy with centrotemporal spikes

“…following standard criteria 30 ; (2) using a validated commercial automated wavelet-based detector, Persyst 31,32 ; and (3) using SpikeNet, a validated automated spike detector developed at our institution. 33,34 For each method, all available non-REM sleep epochs were included (mean = 15.2 minutes, range = 4-48 minutes), and a spike rate for each hemisphere was calculated. To minimize artifactual detections, only spikes detected as maximal in the central and temporal channels (consistent with classic CECTS neurophysiology) were included.…”

“…Manual detection of epileptiform activity on EEG demonstrates inconsistent interrater agreement; however, review by a neurophysiology-trained specialist, as in this study, may increase reliability. 33,41,42 The Persyst algorithm, although validated, has been previously associated with a high rate of false positives. 43 In its validation study, the SpikeNet algorithm was directly compared to neurophysiology-trained experts and Persyst, exceeding both in performance.…”

Monitoring the Burden of Seizures and Highly Epileptiform Patterns in Critical Care with a Novel Machine Learning Method

“…Given the wealth of evidence [33][34][35][36] describing the variability between expert EEG reviewers, whether due to human error or to differences in interpretation, we sought to contextualize the observed diagnostic accuracy of the seizure detection algorithm by quantifying the inter-rater variability. We calculated the sensitivity and specificity of identifying EEGs with status epilepticus for each of the experts that reviewed at least 250 h of EEG.…”

Automatische Erkennung von epilepsietypischen Potenzialen und Anfällen im EEG

“…In einer rezenten Studie [4] zeigte sich zwar eine substanzielle Übereinstimmung von 9 erfahrenen EEG-Befundern hinsichtlich des Vorhandenseins von ETPs in einem gegebenen EEG (mittlere prozentuale Übereinstimmung 80,9 %, mittleres Kappa 69,4 %), die Übereinstimmung bei der Annotation einzelner ETPs war jedoch lediglich moderat (mittlere prozentuale Übereinstimmung 72,4 %, mittleres Kappa 48,7 %). Die Autoren erklären die Unterschiede bei der Annotation einzelner ETPs durch unterschiedliche subjek tive individuelle Schwellenwer te der verschiedenen EEG-Befunder für die binäre Entscheidung ETP vs. kein ETP, die sich durch wenige morphologische Eigenschaften erklären ließen [4]. Alternativ könnte man das Spike-Detektionsprogramm auf einzelne EEG-Befunder individuell trainieren und zuschneiden.…”