Nested cross-validation when selecting classifiers is overzealous for most practical applications

Wainer, Jacques; Cawley, Gavin C.

doi:10.1016/j.eswa.2021.115222

Cited by 134 publications

(83 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Training the model involved identifying the optimal model parameters that define a projection from the multivariate feature space into the probability of each sample belonging to one of the two classes (PRE, POST). We used K-fold nested cross-validation (where K = 7 subjects) to fit the model parameters, optimize the hyperparameters [13], and evaluate the out-of-sample performance of the classification model. The average area under the receiver operating characteristic curve (AUC) across outer-loops was used to measure the separability between baseline and post-stimulation neural states.…”

Section: Logistic Regression With Elastic-net Regularization To Classify Lfp Datamentioning

confidence: 99%

“…A machine learning classification method was used to discriminate between intracranial local field potentials (LFPs) recorded at baseline (stimulation-naïve) and after the first exposure to SCC DBS during surgical procedures. Spectral inputs (theta,(4)(5)(6)(7)(8) alpha,(9)(10)(11)(12) beta,(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30) to the model were then evaluated for importance to classifier success and tested as predictors of the antidepressant response. A decline in depression scores by 45.6% was observed after 1 week and this early antidepressant response correlated with a decrease in SCC LFP beta power, which most contributed to classifier success.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Intraoperative neural signals predict rapid antidepressant effects of deep brain stimulation

et al. 2021

View full text Add to dashboard Cite

Deep brain stimulation (DBS) of the subcallosal cingulate (SCC) is a promising intervention for treatment-resistant depression (TRD). Despite the failure of a clinical trial, multiple case series have described encouraging results, especially with the introduction of improved surgical protocols. Recent evidence further suggests that tractography targeting and intraoperative exposure to stimulation enhances early antidepressant effects that further evolve with ongoing chronic DBS. Accelerating treatment gains is critical to the care of this at-risk population, and identification of intraoperative electrophysiological biomarkers of early antidepressant effects will help guide future treatment protocols. Eight patients underwent intraoperative electrophysiological recording when bilateral DBS leads were implanted in the SCC using a connectomic approach at the site previously shown to optimize 6-month treatment outcomes. A machine learning classification method was used to discriminate between intracranial local field potentials (LFPs) recorded at baseline (stimulation-naïve) and after the first exposure to SCC DBS during surgical procedures. Spectral inputs (theta, 4–8 Hz; alpha, 9–12 Hz; beta, 13–30 Hz) to the model were then evaluated for importance to classifier success and tested as predictors of the antidepressant response. A decline in depression scores by 45.6% was observed after 1 week and this early antidepressant response correlated with a decrease in SCC LFP beta power, which most contributed to classifier success. Intraoperative exposure to therapeutic stimulation may result in an acute decrease in symptoms of depression following SCC DBS surgery. The correlation of symptom improvement with an intraoperative reduction in SCC beta power suggests this electrophysiological finding as a biomarker for treatment optimization.

show abstract

Section: Logistic Regression With Elastic-net Regularization To Classify Lfp Datamentioning

confidence: 99%

mentioning

confidence: 99%

Intraoperative neural signals predict rapid antidepressant effects of deep brain stimulation

et al. 2021

View full text Add to dashboard Cite

show abstract

“…However, how to define the proportions in which the database will be segmented is a subject under development. Therefore, cross-validation strategies such as leave-one-out cross-validation (LOOCV) or k-fold cross-validation have been used more frequently than techniques such as hold-out validation because they obtain better Ac, Se and Sp in laboratory tests[ 4 - 6 ]; moreover, they consider a larger population in the training process compared to hold-out.…”

Section: To the Editormentioning

confidence: 99%

Artificial intelligence model validation before its application in clinical diagnosis assistance

Vázquez-Zapién¹,

Mata-Miranda²,

Garibay-Gonzalez³

et al. 2022

WJG

View full text Add to dashboard Cite

show abstract

“…Therefore, overfitting was prevented since the parameter exploration was limited to a specific subset of the data. The disadvantage was that the computational cost increased highly [76]. Grid search parameter tuning and random search parameter tuning are popular ways of experimenting on the values [52,57,58].…”

Section: Hyperparameter Tuningmentioning

confidence: 99%

A Machine Learning Framework for Olive Farms Profit Prediction

Christias

Mocanu

2021

Water

View full text Add to dashboard Cite

Agricultural systems are constantly stressed due to higher demands for products. Consequently, water resources consumed on irrigation are increased. In combination with the climatic change, those are major obstacles to maintaining sustainable development, especially in a semi-arid land. This paper presents an end-to-end Machine Learning framework for predicting the potential profit from olive farms. The objective is to estimate the optimal economic gain while preserving water resources on irrigation by considering various related factors such as climatic conditions, crop management practices, soil characteristics, and crop yield. The case study focuses on olive tree farms located on the Hellenic Island of Crete. Real data from the farms and the weather in the area will be used. The target is to build a framework that will preprocess input data, compare the results among a group of Machine Learning algorithms and propose the best-predicted value of economic profit. Various aspects during this process will be thoroughly examined such as the bias-variance tradeoff and the problem of overfitting, data transforms, feature engineering and selection, ensemble methods as well as pursuing optimal resampling towards better model accuracy. Results indicated that through data preprocessing and resampling, Machine Learning algorithms performance is enhanced. Ultimately, prediction accuracy and reliability are greatly improved compared to algorithms’ performances without the framework’s operation.

show abstract

Nested cross-validation when selecting classifiers is overzealous for most practical applications

Cited by 134 publications

References 25 publications

Intraoperative neural signals predict rapid antidepressant effects of deep brain stimulation

Intraoperative neural signals predict rapid antidepressant effects of deep brain stimulation

Artificial intelligence model validation before its application in clinical diagnosis assistance

A Machine Learning Framework for Olive Farms Profit Prediction

Contact Info

Product

Resources

About