Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

Hu, Wei; Meng, Xiang-He; Bai, Yuntong; Zhang, Aiying; Qu, Gang; Cai, Biao; Zhang, Gemeng; Wilson, Tony W.; Stephen, Julia M.; Calhoun, Vince D.; Wang, Yuping

doi:10.1109/tmi.2021.3057635

Cited by 38 publications

(12 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the attention mechanism is introduced to distribute weight, formula ( 1 ) can be used to distribute weight [ 18 ]:

where L represents the number of keywords and Similarity ( ) means similarity calculation function, which usually includes the following three functions:

where W represents the learnable parameter and d represents the dimension of keyword and weight value.…”

Section: Basic Methodsmentioning

confidence: 99%

Using Convolutional Neural Networks for the Assessment Research of Mental Health

Liu

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Existing mental health assessment methods mainly rely on experts’ experience, which has subjective bias, so convolutional neural networks are applied to mental health assessment to achieve the fusion of face, voice, and gait. Among them, the OpenPose algorithm is used to extract facial and posture features; openSMILE is used to extract voice features; and attention mechanism is introduced to reasonably allocate the weight values of different modal features. As can be seen, the effective identification and evaluation of 10 indicators such as mental health somatization, depression, and anxiety are realized. Simulation results show that the proposed method can accurately assess mental health. Here, the overall recognition accuracy can reach 77.20%, and the F1 value can reach 0.77. Compared with the recognition methods based on face single-mode fusion, face + voice dual-mode fusion, and face + voice + gait multimodal fusion, the recognition accuracy and F1 value of proposed method are improved to varying degrees, and the recognition effect is better, which has certain practical application value.

show abstract

“…When the attention mechanism is introduced to distribute weight, formula ( 1 ) can be used to distribute weight [ 18 ]:

where L represents the number of keywords and Similarity ( ) means similarity calculation function, which usually includes the following three functions:

where W represents the learnable parameter and d represents the dimension of keyword and weight value.…”

Section: Basic Methodsmentioning

confidence: 99%

Using Convolutional Neural Networks for the Assessment Research of Mental Health

Liu

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Various studies have confirmed that a combination of multi-modal data can improve the performance of predictive models [41,42,31,43,27,44]. Most of these studies concatenated the feature vectors into a single vector for prediction or classification.…”

Section: Multi-modal Data Fusionmentioning

confidence: 99%

An Integrated Multi-modal Learning Method for Early-stage Knee Osteoarthritis Disease Classification

Liu

Jing

2022

Preprint

View full text Add to dashboard Cite

Background: Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. Methods: In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. Results: MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Based on MMLM, the accuracy of SVM and DT are 83.45% and 81.27%, respectively, which are much higher than other feature fusion method. In addition, the visual analysis of the important features in the multi-modal data verified the relations among the different modalities. Conclusion: MMLM uses the internal correlations among different modalities to enhance the efficiency of feature extraction and fusion, which improves grade classification of early-stage knee OA.

show abstract

“…Neuroimaging studies [35], [36] have found that visual pathways across in several brain regions are responsible for visual information processing. Besides, paper [37] has indicated that the complicated relationships among different EEG electrodes are significant for recognition tasks. Therefore, the characteristics of spatial connections in the brain needs to be explored when designing networks for EEG classification.…”

Section: Introductionmentioning

confidence: 99%

Multiscale Temporal Self-Attention and Dynamical Graph Convolution Hybrid Network for EEG-Based Stereogram Recognition

Song

Sun

et al. 2022

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Stereopsis is the ability of human beings to get the 3D perception on real scenarios. The conventional stereopsis measurement is based on subjective judgment for stereograms, leading to be easily affected by personal consciousness. To alleviate the issue, in this paper, the EEG signals evoked by dynamic random dot stereograms (DRDS) are collected for stereogram recognition, which can help the ophthalmologists diagnose strabismus patients even without real-time communication. To classify the collected EEG signals, a novel multi-scale temporal self-attention and dynamical graph convolution hybrid network (MTS-DGCHN) is proposed, including multi-scale temporal self-attention module, dynamical graph convolution module and classification module. Firstly, the multi-scale temporal self-attention module is employed to learn time continuity information, where the temporal self-attention block is designed to highlight the global importance of each time segments in one EEG trial, and the multi-scale convolution block is developed to further extract advanced temporal features in multiple receptive fields. Meanwhile, the dynamical graph convolution module is utilized to capture spatial functional relationships between different EEG electrodes, in which the adjacency matrix of each GCN layer is adaptively tuned to explore the optimal intrinsic relationship. Finally, the temporal and spatial features are fed into the classification module to obtain prediction results. Extensive experiments are conducted on collected datasets i.e., SRDA and SRDB, and the results demonstrate the proposed MTS-DGCHN achieves outstanding classification performance compared with the other methods. The datasets are available at https://github.com/YANGeeg/ TJU-SRD-datasets and the code is at https://github. com/YANGeeg/MTS-DGCHN.

show abstract

Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

Cited by 38 publications

References 32 publications

Using Convolutional Neural Networks for the Assessment Research of Mental Health

Using Convolutional Neural Networks for the Assessment Research of Mental Health

An Integrated Multi-modal Learning Method for Early-stage Knee Osteoarthritis Disease Classification

Multiscale Temporal Self-Attention and Dynamical Graph Convolution Hybrid Network for EEG-Based Stereogram Recognition

Contact Info

Product

Resources

About