Interictal epileptiform discharge characteristics underlying expert interrater agreement

Abstract

Objective The presence of interictal epileptiform discharges (IED) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. However, inter-rater agreement (IRA) regarding the presence of IED is imperfect, leading to incorrect and delayed diagnoses. An improved understanding of which IED attributes mediate expert IRA might help in developing automatic methods for IED detection able to emulate the abilities of experts. Therefore, using a set of IED scored by a large number of experts, we set out to determine which attributes of IED predict expert agreement regarding the presence of IED. Methods IED were annotated on a 5-point scale by 18 clinical neurophysiologists within 200 30-s EEG segments from recordings of 200 patients. 5538 signal analysis features were extracted from the waveforms, including wavelet coefficients, morphological features, signal energy, nonlinear energy operator response, electrode location, and spectrogram features. Feature selection was performed by applying elastic net regression and support vector regression (SVR) was applied to predict expert opinion, with and without the feature selection procedure and with and without several types of signal normalization. Results Multiple types of features were useful for predicting expert annotations, but particular types of wavelet features performed best. Local EEG normalization also enhanced best model performance. As the size of the group of EEGers used to train the models was increased, the performance of the models leveled off at a group size of around 11. Conclusions The features that best predict inter-rater agreement among experts regarding the presence of IED are wavelet features, using locally standardized EEG. Our models for predicting expert opinion based on EEGer’s scores perform best with a large group of EEGers (more than 10). Significance By examining a large group of EEG signal analysis features we found that wavelet features with certain wavelet basis functions performed best to identify IEDs. Local normalization also improves predictability, suggesting the importance of IED morphology over amplitude-based features. Although most IED detection studies in the past have used opinion from three or fewer experts, our study suggests a “wisdom of the crowd” effect, such that pooling over a larger number of expert opinions produces a better correlation between expert opinion and objectively quantifiable features of the EEG.

Consensus in Traditional Chinese Medical Diagnosis in Open Populations

“…A limitation in our study was the low numbers of subjects and practitioners utilized, and further studies should be carried out to confirm our findings. As mentioned in the introduction of this paper, larger numbers of practitioners have been found to be associated with higher levels of consensus in another modality 21 and this should be investigated.…”

“…Investigations into diagnostic reliability in other areas have also reported mixed results ranging from poor 17,18 to good 19 accord, with training often shown to improve agreement, 20 and the inclusion of more raters was found to improve accord. 21 The inappropriate use of kappa statistics is not limited to CM or TCM, with recent examples occurring in other modalities. 18,[22][23][24][25][26] Furthermore, the use of kappa statistics may have contributed to the surprising result of training not improving diagnostic reliability that was reported in one paper.…”

Automatische Erkennung von epilepsietypischen Potenzialen und Anfällen im EEG

“…In einer Studie, in der 18 klinische Neurophysiologen Spike-Annotationen in 200 EEG-Segmenten von 30 s Dauer bei 200 Patienten durchführten, konnten bestimmte Wavelet-Merkmale die Übereinstimmung der EEG-Experten am besten vorhersagen. Diese Eigenschaften können wiederum als Basis von Machine-Learning-Algorithmen genutzt werden [3]. Dabei konnte eine Sensitivität von 47,4 % und eine Spezifität von 98,0 % (0,6 falsch-positive Detektionen pro Minute) erreicht werden, bei normalen EEGs betrug die Häufigkeit von falsch-positiven Detektionen 0,3 pro Minute [18].…”

Labeler-hot Detection of EEG Epileptic Transients

“…Misinterpretation of scalp electroencephalogram (sEEG) is not uncommon in clinical practice [1], [2]. At the same time, it can have severe negative consequences on health and well-being of patients undergoing epileptic diagnosis [3].…”