Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines

Abstract

Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within- and across-subject predictions, on multiple datasets -anatomical and functional MRI and MEG- and simulations. Theory and experiments outline that the popular "leave-one-out" strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non-sparse decoders.

Predicting personality from network-based resting-state functional connectivity

“…In accordance with recent recommendations, the current study used 10-fold cross-validation, which has been showed to be less susceptible to overly optimistic estimates as compared with a leave-one-out approach (LOO-CV) (Varoquaux et al 2016). Moreover, we repeated the cross-validation procedure 250 times, averaging the prediction performance over all replications to obtain robust and generalizable estimates of the capability of different brain networks to predict personality scores in new individuals.…”

Joint prediction of multiple scores captures better individual traits from brain images

“…This experiment is carried on COBRE, to predict schizophrenia, and on ADNI, to predict AD profiles. Models are evaluated with a cross-validation procedure as recom-290 mended in Varoquaux et al (2016): The data are split into stratified train/test sets at the subject level, to avoid fitting and testing on data from the same subject. Splits are randomized over 100 runs.…”

Interpretability of Multivariate Brain Maps in Linear Brain Decoding: Definition, and Heuristic Quantification in Multivariate Analysis of MEG Time-Locked Effects

“…One possible approach is to exclude the entire test set from the model selection procedure using a nested nested cross-validation strategy. An alternative approach is employing model averaging techniques to neatly get advantage of the whole dataset (Varoquaux et al, 2017). Since our focus is on the model selection, in the remaining text, we implicitly assume the test data is excluded from the experiments, thus, all the experimental results are reported on the training and validation sets.…”

“…The most common model selection criterion is based on an estimator of generalization performance, i.e., the predictive power. In the context of brain decoding, especially when the interpretability of brain maps matters, employing the predictive power as the only decisive criterion in model selection is problematic in terms of interpretability of MBMs (Gramfort et al, 2012; Rasmussen et al, 2012; Conroy et al, 2013; Varoquaux et al, 2017). Valverde-Albacete and Peláez-Moreno (2014) experimentally showed that in a classification task optimizing only classification error rate is insufficient to capture the transfer of crucial information from the input to the output of a classifier.…”