Multimodal Integration of Brain Images for MRI-Based Diagnosis in Schizophrenia

Abstract

Magnetic resonance imaging (MRI) has been proposed as a source of information for automatic prediction of individual diagnosis in schizophrenia. Optimal integration of data from different MRI modalities is an active area of research aimed at increasing diagnostic accuracy. Based on a sample of 96 patients with schizophrenia and a matched sample of 115 healthy controls that had undergone a single multimodal MRI session, we generated individual brain maps of gray matter vbm, 1back, and 2back levels of activation (nback fMRI), maps of amplitude of low-frequency fluctuations (resting-state fMRI), and maps of weighted global brain connectivity (resting-state fMRI). Four unimodal classifiers (Ridge, Lasso, Random Forests, and Gradient boosting) were applied to these maps to evaluate their classification accuracies. Based on the assignments made by the algorithms on test individuals, we quantified the amount of predictive information shared between maps (what we call redundancy analysis). Finally, we explored the added accuracy provided by a set of multimodal strategies that included post-classification integration based on probabilities, two-step sequential integration, and voxel-level multimodal integration through one-dimensional-convolutional neural networks (1D-CNNs). All four unimodal classifiers showed the highest test accuracies with the 2back maps (80% on average) achieving a maximum of 84% with the Lasso. Redundancy levels between brain maps were generally low (overall mean redundancy score of 0.14 in a 0–1 range), indicating that each brain map contained differential predictive information. The highest multimodal accuracy was delivered by the two-step Ridge classifier (87%) followed by the Ridge maximum and mean probability classifiers (both with 85% accuracy) and by the 1D-CNN, which achieved the same accuracy as the best unimodal classifier (84%). From these results, we conclude that from all MRI modalities evaluated task-based fMRI may be the best unimodal diagnostic option in schizophrenia. Low redundancy values point to ample potential for accuracy improvements through multimodal integration, with the two-step Ridge emerging as a suitable strategy.

Machine Learning of Schizophrenia Detection with Structural and Functional Neuroimaging

“…The machine learning (ML) technique is a new approach that can extract relevant information from images and construct models to determine the probability of disease onset, and it can make a higher accurate prediction compared with conventional methods [ 5 , 6 , 13 , 17 , 18 ]. Salvador et al [ 15 ] achieved 75.76% accuracy in schizophrenia diagnosis, and de Filippis et al [ 5 ] reported that support vector machine associated with other ML techniques could achieve accuracy close to 100%.…”

“…Functional MRI (fMRI) and structural MRI (sMRI) are gaining importance and becoming more widely acceptable techniques with the potential to help diagnose neurological illnesses [8][9][10][11], including schizophrenia [5,12,13]. An increasing number of studies have reported that multimodal brain data can improve diagnostic accuracy by combining the information obtained from different MRI imaging modalities [8,[14][15][16]. The machine learning (ML) technique is a new approach that can extract relevant information from images and construct models to determine the probability of disease onset, and it can make a higher accurate prediction compared with conventional methods [5,6,13,17,18].…”

A deep learning fusion model for brain disorder classification

“…Indeed, there has been some work which incorporates multimodal measures to differentiate mental disorders via deep learning. Salvador et al [20] used gray matter volumes, functional activation † Yuhui Du, corresponding author Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.…”

Statistical analysis and machine learning in multimodal brain imaging of neuropsychiatric disorders

“…Multimodal features provide information from different perspectives allowing models to understand the neural substrates associated with schizophrenia with complementary information from various modalities. Previous studies illustrated the benefits of combining sMRI and dMRI as well as fMRI for the classification of schizophrenia using handcrafted featurebased machine learning (Isobe et al, 2016;Lei et al, 2020;Saarinen et al, 2020;Salvador et al, 2019), suggesting possible future research with incorporation of additional neuroimaging modalities.…”

“…Studies using transfer learning approaches that utilize powerful pre-trained 2D CNN networks to extract features from deep layers are also lacking. Of note, multimodal feature extraction from different neuroimaging modalities is critical for understanding the neural substrates underlying schizophrenia from complementary perspectives, potentially leading to higher classification performance (Lei et al, 2020;Lerman-Sinkoff et al, 2019;Salvador et al, 2019). The automatic feature learning capability of CNN enables more prominent integration of multimodal inputs, yet, multi-channel 2D and 3D CNN have not been employed and evaluated for schizophrenia discrimination.…”