Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection

Abstract

In recent years, machine learning techniques have been frequently applied to uncovering neuropsychiatric biomarkers with the aim of accurately diagnosing neuropsychiatric diseases and predicting treatment prognosis. However, many studies did not perform cross validation (CV) when using machine learning techniques, or others performed CV in an incorrect manner, leading to significantly biased results due to overfitting problem. The aim of this study is to investigate the impact of CV on the prediction performance of neuropsychiatric biomarkers, in particular, for feature selection performed with high-dimensional features. To this end, we evaluated prediction performances using both simulation data and actual electroencephalography (EEG) data. The overall prediction accuracies of the feature selection method performed outside of CV were considerably higher than those of the feature selection method performed within CV for both the simulation and actual EEG data. The differences between the prediction accuracies of the two feature selection approaches can be thought of as the amount of overfitting due to selection bias. Our results indicate the importance of correctly using CV to avoid biased results of prediction performance of neuropsychiatric biomarkers.

Alteration of Neural Network Activity With Aging Focusing on Temporal Complexity and Functional Connectivity Within Electroencephalography

“…We also used 5-fold cross-validation to prevent overfitting; AUC= 1.0 corresponds to perfect discrimination, and AUC= 0.5 corresponds to random discrimination. Here, the principal component analysis was conducted within cross validation (Shim et al, 2021) to avoid the inaccurate estimation of performance of discrimination. AUC values were averaged among 20 trials to choose tested and evaluated data set in 5-fold cross-validation and their standard deviations (SD) were also derived.…”

Accelerated functional brain aging in major depressive disorder: evidence from a large scale fMRI analysis of Chinese participants

“…We compare the brain-PAD scores of the controls in this hold-out validation set to the MDD patients in the test set. As the validation set is not involved in the development of the brain age prediction model, the risk of overfitting is effectively prevented [64]. The application of four different machine learning algorithms allows us to further validate the consistency of the patterns observed.…”

Machine Learning-Based 30-Day Hospital Readmission Predictions for COPD Patients Using Physical Activity Data of Daily Living with Accelerometer-Based Device

“…The organized total data comprised 1695 datasets which were used in pipeline for the 10-fold cross-validation of the four ML models considered in this study, i.e., the logistic regression model, support vector machine, random forest model, and multilayer perceptron, to generate 30-day hospital readmission predictions by identifying the nonlinear classifying characteristic relationships among different activity-based PA parameters ( Table 1 ) and actual hospital readmissions. We adopted 10-fold cross-validation (sometimes as blocked cross-validation for timeseries splits) to prevent any data leakage [ 32 , 33 , 34 ]. The ML models were trained by a combination of supervised and reinforced learning methods, and their performance was 10-fold cross-validated using the total dataset to acquire a final trained ML model with an averaged training score.…”