Performance assessment for EEG-based neonatal seizure detectors

Abstract

ObjectiveThis study discusses an appropriate framework to measure system performance for the task of neonatal seizure detection using EEG. The framework is used to present an extended overview of a multi-channel patient-independent neonatal seizure detection system based on the Support Vector Machine (SVM) classifier.MethodsThe appropriate framework for performance assessment of neonatal seizure detectors is discussed in terms of metrics, experimental setups, and testing protocols. The neonatal seizure detection system is evaluated in this framework. Several epoch-based and event-based metrics are calculated and curves of performance are reported. A new metric to measure the average duration of a false detection is proposed to accompany the event-based metrics. A machine learning algorithm (SVM) is used as a classifier to discriminate between seizure and non-seizure EEG epochs. Two post-processing steps proposed to increase temporal precision and robustness of the system are investigated and their influence on various metrics is shown. The resulting system is validated on a large clinical dataset of 267 h.ResultsIn this paper, it is shown how a complete set of metrics and a specific testing protocol are necessary to extensively describe neonatal seizure detection systems, objectively assess their performance and enable comparison with existing alternatives. The developed system currently represents the best published performance to date with an ROC area of 96.3%. The sensitivity and specificity were ∼90% at the equal error rate point. The system was able to achieve an average good detection rate of ∼89% at a cost of 1 false detection per hour with an average false detection duration of 2.7 min.ConclusionsIt is shown that to accurately assess the performance of EEG-based neonatal seizure detectors and to facilitate comparison with existing alternatives, several metrics should be reported and a specific testing protocol should be followed. It is also shown that reporting only event-based metrics can be misleading as they do not always reflect the true performance of the system.SignificanceThis is the first study to present a thorough method for performance assessment of EEG-based seizure detection systems. The evaluated SVM-based seizure detection system can greatly assist clinical staff, in a neonatal intensive care unit, to interpret the EEG.

Automatic detection of fast oscillations (40–200Hz) in scalp EEG recordings

“…The sensitivity as a function of the positive predicted value is sometimes called precision-recall (PR) curve (Temko et al, 2011). The sensitivity and positive predicted values are computed as (1) (2) where N expert is the total number of expert's marks (detected plus undetected by the automatic detector), N dem is the number of detected expert's marks, N auto is the total number of automatic detections (true plus false positives), and N tp is the number of true positive automatic detections.…”

A nonparametric feature for neonatal EEG seizure detection based on a representation of pseudo-periodicity

“…Finally, the NFM based estimate of η was combined with aEEG amplitude using a trained LDC, with pre-processing and post-processing stages, to form a simple SDA. This SDA was tested on the full database of neonatal EEG and assessed with the AUC, seizure detection rate and false alarms per hour metrics [31]. Two clinically relevant measures of a SDA are the ability of the SDA to discriminate seizure from nonseizure neonates when applied to long duration EEG recordings and the ability of the SDA to accurately estimate the seizure burden of the neonate.…”

“…There are many issues with comparing the performance of SDAs [31]. Differences in performance metrics and neonatal EEG databases make any comparisons difficult.…”

Automatic seizure detection based on the combination of newborn multi-channel EEG and HRV information

“…Although there is no standard for reporting detection performance, two complementary procedures are gaining wide acceptance. These methods are the event-based and the time-based detection methods [19,56]. Temko et al [56] correctly concluded that the best way to report the performance of a seizure detector is to include a number of different, but complementary, measures.…”