Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics

Abstract

In this critical review, we examine the application of predictive models, e.g. classifiers, trained using Machine Learning (ML) to assist in interpretation of functional neuroimaging data. Our primary goal is to summarize how ML is being applied and critically assess common practices. Our review covers 250 studies published using ML and resting-state functional MRI (fMRI) to infer various dimensions of the human functional connectome. Results for hold-out (“lockbox”) performance was, on average, ~13% less accurate than performance measured through cross-validation alone, highlighting the importance of lockbox data which was included in only 16% of the studies. There was also a concerning lack of transparency across the key steps in training and evaluating predictive models. The summary of this literature underscores the importance of the use of a lockbox and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that, ideally, studies are motivated both by the reproducibility and generalizability of findings as well as the potential clinical significance of the insights. We offer recommendations for principled integration of machine learning into the clinical neurosciences with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks, and parsing heterogeneous patient outcomes.

Multimodal Brain Signal Complexity Predicts Human Intelligence

“…Importantly, according to the criteria of Cohen (1988) all observed effect sizes ( r ~ .2) can be considered as small and only few out of all calculated measures reached statistical significance, when correcting for multiple comparisons. These results contribute to the current debate about the effect size to be expected in investigations on brain-behavior relations (Marek et al, 2022; Rosenberg and Finn, 2022) in demonstrating that the combination of cross-validation (Sui et al, 2020; Cwiek et al, 2022) and multimodal analyses approaches can identify robust brain-behavior relations despite sample sizes that lie clearly below thousand. We compared three forms of analyses (explanation of intelligence scores, internal cross-validation, and out-of-sample prediction: prediction of intelligence scores in a replication sample with the model constructed in the main sample) to show, in line with Cwiek et al, (2022), that cross-validation reduces the overall effect size markedly as compared to explanation ( r = .31 to r = .22 and r = .23).…”

“…These results contribute to the current debate about the effect size to be expected in investigations on brain-behavior relations (Marek et al, 2022; Rosenberg and Finn, 2022) in demonstrating that the combination of cross-validation (Sui et al, 2020; Cwiek et al, 2022) and multimodal analyses approaches can identify robust brain-behavior relations despite sample sizes that lie clearly below thousand. We compared three forms of analyses (explanation of intelligence scores, internal cross-validation, and out-of-sample prediction: prediction of intelligence scores in a replication sample with the model constructed in the main sample) to show, in line with Cwiek et al, (2022), that cross-validation reduces the overall effect size markedly as compared to explanation ( r = .31 to r = .22 and r = .23). That internally cross-validated effect size reflects a more realistic estimate of the ‘true’ effect size (Yarkoni and Westfall, 2017) is supported by our out-of-sample-prediction in the independent sample ( r = .23).…”

How failure to falsify in high-volume science contributes to the replication crisis

“…Adding flexibility, predictive algorithms built on top of these large datasets typically involve a great number of investigator decisions -the combined effects of which undermine reliability of findings [for an example in connectivity modeling see Hallquist and Hillary, 2018]. Results of machine learning models, for example, are sensitive to model specification and parameter tuning [Pineau et al, 2021Bouthillier et al, 2019Cwiek et al, 2021]. Computational approaches permit systematically combing through a great number of potential variables of interest and their statistical relationships (specifically, at scales which would be manually infeasible).…”

How failure to falsify in high-volume science contributes to the replication crisis

“…The strength of a hypothesis refers to how specific and how refutable it is ( Popper, 1963 ; see Table 1 for examples). We also argue for greater emphasis on testing and refuting strong hypotheses through a “team science” framework that allows us to address the heterogeneity in samples and/or methods that makes so many published findings tentative ( Cwiek et al, 2021 ; Bryan et al, 2021 ).…”

“…Adding flexibility, predictive algorithms built on top of these large datasets typically involve a great number of investigator decisions – the combined effects of which undermine reliability of findings [for an example in connectivity modeling see Hallquist and Hillary, 2019 ]. Results of machine learning models, for example, are sensitive to model specification and parameter tuning ( Pineau, 2021 ; Bouthillier et al, 2019 ; Cwiek et al, 2021 ). Computational approaches permit systematically combing through a great number of potential variables of interest and their statistical relationships (specifically, at scales which would be manually infeasible).…”