Characteristics of EEG Interpreters Associated With Higher Interrater Agreement

Abstract

Purpose The goal of the project is to determine characteristics of academic neurophysiologist EEG interpreters (EEGers), which predict good interrater agreement (IRA) and to determine the number of EEGers needed to develop an ideal standardized testing and training data set for epileptiform transient (ET) detection algorithms. Methods A three-phase scoring method was used. In phase 1, 19 EEGers marked the location of ETs in two hundred 30-second segments of EEG from 200 different patients. In phase 2, EEG events marked by at least 2 EEGers were annotated by 18 EEGers on a 5-point scale to indicate whether they were ETs. In phase 3, a third opinion was obtained from EEGers on any inconsistencies between phase 1 and phase 2 scoring. Results The IRA for the 18 EEGers was only fair. A select group of the EEGers had good IRA and the other EEGers had low IRA. Board certification by the American Board of Clinical Neurophysiology was associated with better IRA performance but other board certifications, years of fellowship training, and years of practice were not. As the number of EEGers used for scoring is increased, the amount of change in the consensus opinion decreases steadily and is quite low as the group size approaches 10. Conclusions The IRA among EEGers varies considerably. The EEGers must be tested before use as scorers for ET annotation research projects. The American Board of Clinical Neurophysiology certification is associated with improved performance. The optimal size for a group of experts scoring ETs in EEG is probably in the 6 to 10 range.

Interictal epileptiform discharge characteristics underlying expert interrater agreement

“…Nevertheless, the performance of our models levels off at a top scorer group size of around 12. This is consistent with a detailed analysis of our expert opinion published elsewhere which showed that the average opinion of our experts ceases to change significantly when the top scorer group gets to the size of 10–12 (Halford et al, 2017). …”

“…The performance appears to level off when the top scorer group size increases to 12. We would not expect the SVR performance to improve much after 12 top scorers have been added, because the mean expert opinion in these large top scorer groups above size 12 does not change much as more EEGers are added (Halford et al, 2017). Nevertheless, because the group of all scorers produced the best overall performance, we choose “all experts” to be the optimum set of experts to be used in the further analysis presented below.…”

“…This provided a third “tie breaker” opinion on these events. Further details about this method are published elsewhere (Halford et al, 2017). …”

“…In addition, there is a substantial discrepancy in interpretations between board-certified academic clinical neurophysiologists (Halford et al, 2013, 2017). …”

The effect of reducing EEG electrode number on the visual interpretation of the human expert for neonatal seizure detection

“…It is now well established that the inter-expert agreement is not perfect for the detection of a range of EEG patterns (Scher et al, 1994;Abend et al, 2011a,b;Stevenson et al, 2015;Wusthoff et al, 2017). In an era of precision medicine, the important clinical question is how tarnished can the supposed gold standard become and still be useful for neuro-critical care, or be used as an outcome measure in clinical trials (Halford et al, 2016). We show that reducing the number of electrodes in an EEG montage from 19 to 8 or 4 significantly changes the annotation of seizures beyond inter-expert differences, although the relationship between seizure detection and electrode number is nonlinear.…”