Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within‐site and between‐site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting‐state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within‐site generalizability of the classification framework in the main data set using cross‐validation. Then, we trained a model in the main data set and investigated between‐site generalization in the validated data set using external validation. Finally, recognizing the poor between‐site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between‐site classification performance. Cross‐validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within‐site cross‐validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.

Background Schizotypy refers to schizophrenia-like traits below the clinical threshold in the general population. The pathological development of schizophrenia has been postulated to evolve from the initial coexistence of ‘brain disconnection’ and ‘brain connectivity compensation’ to ‘brain connectivity decompensation’. Methods In this study, we examined the brain connectivity changes associated with schizotypy by combining brain white matter structural connectivity, static and dynamic functional connectivity analysis of diffusion tensor imaging data and resting-state functional magnetic resonance imaging data. A total of 87 participants with a high level of schizotypal traits and 122 control participants completed the experiment. Group differences in whole-brain white matter structural connectivity probability, static mean functional connectivity strength, dynamic functional connectivity variability and stability among 264 brain sub-regions of interests were investigated. Results We found that individuals with high schizotypy exhibited increased structural connectivity probability within the task control network and within the default mode network; increased variability and decreased stability of functional connectivity within the default mode network and between the auditory network and the subcortical network; and decreased static mean functional connectivity strength mainly associated with the sensorimotor network, the default mode network and the task control network. Conclusions These findings highlight the specific changes in brain connectivity associated with schizotypy and indicate that both decompensatory and compensatory changes in structural connectivity within the default mode network and the task control network in the context of whole-brain functional disconnection may be an important neurobiological correlate in individuals with high schizotypy.