Epilepsyecosystem.org: crowd-sourcing reproducible seizure prediction with long-term human intracranial EEG

Abstract

Accurate seizure prediction will transform epilepsy management by offering warnings to patients or triggering interventions. However, state-of-the-art algorithm design relies on accessing adequate long-term data. Crowd-sourcing ecosystems leverage quality data to enable cost-effective, rapid development of predictive algorithms. A crowd-sourcing ecosystem for seizure prediction is presented involving an international competition, a follow-up held-out data evaluation, and an online platform, Epilepsyecosystem.org, for yielding further improvements in prediction performance. Crowd-sourced algorithms were obtained via the 'Melbourne-University AES-MathWorks-NIH Seizure Prediction Challenge' conducted at kaggle.com. Long-term continuous intracranial electroencephalography (iEEG) data (442 days of recordings and 211 lead seizures per patient) from prediction-resistant patients who had the lowest seizure prediction performances from the NeuroVista Seizure Advisory System clinical trial were analysed. Contestants (646 individuals in 478 teams) from around the world developed algorithms to distinguish between 10-min inter-seizure versus pre-seizure data clips. Over 10 000 algorithms were submitted. The top algorithms as determined by using the contest data were evaluated on a much larger held-out dataset. The data and top algorithms are available online for further investigation and development. The top performing contest entry scored 0.81 area under the classification curve. The performance reduced by only 6.7% on held-out data. Many other teams also showed high prediction reproducibility. Pseudo-prospective evaluation demonstrated that many algorithms, when used alone or weighted by circadian information, performed better than the benchmarks, including an average increase in sensitivity of 1.9 times the original clinical trial sensitivity for matched time in warning. These results indicate that clinically-relevant seizure prediction is possible in a wider range of patients than previously thought possible. Moreover, different algorithms performed best for different patients, supporting the use of patient-specific algorithms and long-term monitoring. The crowd-sourcing ecosystem for seizure prediction will enable further worldwide community study of the data to yield greater improvements in prediction performance by way of competition, collaboration and synergism.10.1093/brain/awy210_video1awy210media15817489051001.

Ensembling crowdsourced seizure prediction algorithms using long‐term human intracranial EEG

“…After the competition, eight of the top‐performing teams participated in an analysis of a much larger, held‐out dataset from the same patients. These teams used a variety of machine learning algorithms as described previously . In addition, circadian weighting was incorporated into a sub‐analysis of results in which the original preictal probability predictions from the machine learning algorithms were multiplied by the probability of a seizure occurring at a given time of day …”

“…Success was defined by an algorithm having higher AUC scores than the original contest algorithms, or higher sensitivities for matching proportion of time in warning. Statistical tests for performance comparisons are reported with a significance level of 0.05 with subsequent Bonferroni correction . Pseudo‐prospective seizure prediction performance was compared against the performance of random analytic Poisson prediction .…”

“…Prospective seizure prediction has been shown to be feasible using long‐term intracranial electroencephalography (iEEG) recordings obtained during the first‐in‐human trial of a seizure prediction device; however, prediction accuracy should be increased for a broader range of patients in order to demonstrate clinical utility . Several studies have looked at achieving retrospective improvements with the same long‐term data . The recent Melbourne‐University AES‐MathWorks‐NIH Seizure Prediction Challenge yielded improvements for the three patients from the NeuroVista trial whose seizures were the hardest to predict.…”

“…Over 10 000 algorithms were submitted in the aforementioned contest, and eight of the top algorithms were evaluated on held‐out data from the same patients . After such extensive development, an important question to answer is whether the performance level obtained from this competition was the upper limit of what is possible for certain patients?…”

“…4 Several studies have looked at achieving retrospective improvements with the same long-term data. [5][6][7] The recent Melbourne-University AES-MathWorks-NIH Seizure Prediction Challenge yielded improvements for the three patients from the NeuroVista trial whose seizures were the hardest to predict. The top internationally crowdsourced algorithms increased seizure prediction sensitivity by 90% relative to the original trial, for matched time in warning.…”