Development and validation of machine learning models for prediction of seizure outcome after pediatric epilepsy surgery

Abstract

Objective: There is substantial variability in reported seizure outcome following pediatric epilepsy surgery, and lack of individualized predictive tools that could evaluate the probability of seizure freedom postsurgery. The aim of this study was to develop and validate a supervised machine learning (ML) model for predicting seizure freedom after pediatric epilepsy surgery.Methods: This is a multicenter retrospective study of children who underwent epilepsy surgery at five pediatric epilepsy centers in North America. Clinical information, diagnostic investigations, and surgical characteristics were collected, and used as features to predict seizure-free outcome 1 year after surgery. The dataset was split randomly into 80% training and 20% testing data. Thirty-five combinations of five feature sets with seven ML classifiers were assessed on the training cohort using 10-fold cross-validation for model development. The performance of the optimal combination of ML classifier and feature set was evaluated in the testing cohort, and compared with logistic regression, a classical statistical approach.Results: Of the 801 patients included, 61.3% were seizure-free 1 year postsurgery.During model development, the best combination was XGBoost ML algorithm with five features from the univariate feature set, including number of antiseizure medications, magnetic resonance imaging lesion, age at seizure onset, videoelectroencephalography concordance, and surgery type, with a mean area under the curve (AUC) of .73 (95% confidence interval [CI] = .69-.77). The combination of XGBoost and univariate feature set was then evaluated on the testing cohort

Predicting seizure outcome after epilepsy surgery: do we need more complex models, larger samples, or better data?

“…Machine learning in clinical research is placing an increasing emphasis on model generalizability, where the highest level of evidence is achieved from applying models externally -to new centers. When we tested the model by Yossofzai et al 23 on our data, we found that it did not generalize well. This may at first glance seem surprising, as there is a striking similarity between our cohort and the cohort of Yossofzai et al 23 -not only in terms of sample size, but also in terms of patient characteristics and variables found to be predictive of outcome.…”

“…When we applied the XGBoost model developed by Yossofzai et al 23 to our data, it achieved an accuracy of 63% (95% CI=59-67%) and an AUC of 0.62.…”

“…20 Previous studies that have used both machine learning techniques and traditional statistical modelling approaches to predict post-operative seizure outcome have found that logistic regression models perform as well as, or even better than, machine learning ones. 11,15,22 To our knowledge, only one study by Yossofzai et al 23 has found that a machine learning model outperforms a logistic regression; however, this was a 0.1-0.2 difference in AUC (0.72 versus 0.73 in the train dataset; 0.72 versus 0.74 in the test dataset). This small improvement is unlikely to deliver an advantage in clinical practice.…”

“…We trained the MLP with two hidden layers and ten hidden neurons, respectively, balancing the need for sufficient complexity to learn feature interactions across multiple features, while limiting the capacity of the network to overfit to the training data. We chose an XGBoost model to ensure that we could compare the performance of this to the performance of the XGBoost model published by Yossofzai et al 23 After training our own three models, we applied the XGBoost model by Yossofzai et al 23 to the same patient cohort. We evaluated the performance of all models using stratified 10-fold cross-validation.…”

“…Past studies have reported models capable of achieving accuracies as high as 90-100% using features extracted from MRI and EEG (Table 1); however, none of these findings can be reproduced, nor can any of these models be adopted by other centers, as there is insufficient information about how they were generated. Yossofzai et al 23 are to be commended for sharing their model in a way that allowed for it to be externally tested by ourselves and others.…”