Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification

Han, Zongbo; Yang, Fan; Huang, Junzhou; Zhang, Changqing; Yao, Jianhua

doi:10.1109/cvpr52688.2022.02005

Cited by 41 publications

(27 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A total of fourteen computational methods are compared, encompassing six single-omics classifiers that employ early integration (K-nearest neighbors (KNN) 37 , support vector machine (SVM) 38 , Lasso 39 , random forest (RF) 40 , eXtreme Gradient Boosting (XGboost) 41 , and fully connected neural networks (NN) 42 ), and seven multi-omics classifiers including group-regularized ridge regression 43 , BPLSDA that projects data to latent structures with discriminant analysis 44 , block PLSDA with additional sparse constraints (BSPLSDA) 44 , concatenation of final representations (CF) that concatenates late-stage multi-omics representations 45, 46 , gated multimodal fusion (GMU) that implements fusion with intermediate representation 47 , and two state-of-the-art methods (Mogonet 22 and Dynamic 30 ).…”

Section: Resultsmentioning

confidence: 99%

“…In practice, the decision-making process for each omics data should be transparent and provide feedback for its own predictions. Consequently, we introduce a confidence-adaptive technique capable of delivering more accurate predictions 30, 34 , addressing the aforementioned challenges and facilitating a more effective integration of multi-omics data.…”

Section: Methodsmentioning

confidence: 99%

“…They then leverage a view correlation discovery network to integrate these initial classifications and produce a final prediction. Han et al 30 employs encoder networks to extract feature-level omics-specific informativeness and adopts true-class probability to evaluate the classification confidence of different omics. They show that confidence-based integration has the potential to improve classification performance.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

GREMI: an Explainable Multi-omics Integration Framework for Enhanced Disease Prediction and Module Identification

Luo

Liang

Jia

et al. 2023

Preprint

View full text Add to dashboard Cite

By capturing complementary information from multiple omics data, multi-omics integration has demonstrated promising performance in disease prediction. However, with the increasing number of omics data, accurately quantifying the characterization ability of each omics and avoiding mutual interference becomes challenging due to the intricate relationships among omics data. Here, we propose a novel multi-omics integration framework that improves diagnostic prediction. Specifically, our framework involves constructing co-expression and co-methylation networks for each subject, followed by applying multi-level graph attention to incorporate biomolecule interaction information. The true-class-probability strategy is employed to evaluate omics-level confidence for classification, and the loss is designed using an adaptive mechanism to leverage both within- and across-omics information. Extensive experiments demonstrate that the proposed framework outperforms state-of-the-art methods on classification tasks, and indicate that the integration of three omics yields superior performance compared to employing only one or two data types. Code and data are available at https://github.com/Yaolab-fantastic/GRAMI-NET.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GREMI: an Explainable Multi-omics Integration Framework for Enhanced Disease Prediction and Module Identification

Luo

Liang

Jia

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The predicted location p r is an 8-dimensional probability vector, with each element representing whether the corresponding patch is masked. Here, we can employ the 0 loss function [26] for location prediction task, which counts nonzero elements in p r , defined as:…”

Section: Patch-masking Perceptionmentioning

confidence: 99%

HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation

Xing¹,

Li²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Masked image modeling (MIM) with transformer backbones has recently been exploited as a powerful self-supervised pre-training technique. The existing MIM methods adopt the strategy to mask random patches of the image and reconstruct the missing pixels, which only considers semantic information at a lower level, and causes a long pre-training time. This paper presents Hy-bridMIM, a novel hybrid self-supervised learning method based on masked image modeling for 3D medical image segmentation. Specifically, we design a two-level masking hierarchy to specify which and how patches in sub-volumes are masked, effectively providing the constraints of higher level semantic information. Then we learn the semantic information of medical images at three levels, including: 1) partial region prediction to reconstruct key contents of the 3D image, which largely reduces the pre-training time burden (pixel-level); 2) patch-masking perception to learn the spatial relationship between the patches in each subvolume (region-level). and 3) drop-out-based contrastive learning between samples within a mini-batch, which further improves the generalization ability of the framework (sample-level). The proposed framework is versatile to support both CNN and transformer as encoder backbones, and also enables to pre-train decoders for image segmentation. We conduct comprehensive experiments on four widelyused public medical image segmentation datasets, including BraTS2020, BTCV, MSD Liver, and MSD Spleen. The experimental results show the clear superiority of Hybrid-MIM against competing supervised methods, masked pretraining approaches, and other self-supervised methods, in terms of quantitative metrics, timing performance and qualitative observations. The codes of HybridMIM are available at https://github.com/ge-xing/HybridMIM.

show abstract

“…Feature-level self-attention. Inspired by Han et al 9 , we employ the self-attention component for featurelevel feature selection and omics-level feature selection. Though sparsity of the high-dimensional data enables a high-level feature embedding, the informativeness of different features among different subjects are more important for multi-omics integration 30 .…”

Section: Self-attention Based Dynamical Integrationmentioning

confidence: 99%

CLCLSA: Cross-omics Linked embedding with Contrastive Learning and Self Attention for multi-omics integration with incomplete multi-omics data

Zhao

Liu

Zhang

et al. 2023

Preprint

View full text Add to dashboard Cite

Integration of heterogeneous and high-dimensional multi-omics data is becoming increasingly important in understanding genetic data. Each omics technique only provides a limited view of the underlying biological process and integrating heterogeneous omics layers simultaneously would lead to a more comprehensive and detailed understanding of diseases and phenotypes. However, one obstacle faced when performing multi-omics data integration is the existence of unpaired multi-omics data due to instrument sensitivity and cost. Studies may fail if certain aspects of the subjects are missing or incomplete. In this paper, we propose a deep learning method for multi-omics integration with incomplete data by Cross-omics Linked unified embedding with Contrastive Learning and Self Attention (CLCLSA). Utilizing complete multi-omics data as supervision, the model employs cross-omics autoencoders to learn the feature representation across different types of biological data. The multi-omics contrastive learning, which is used to maximize the mutual information between different types of omics, is employed before latent feature concatenation. In addition, the feature-level self-attention and omics-level self-attention are employed to dynamically identify the most informative features for multi-omics data integration. Extensive experiments were conducted on four public multi-omics datasets. The experimental results indicated that the proposed CLCLSA outperformed the state-of-the-art approaches for multi-omics data classification using incomplete multiomics data.

show abstract

Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification

Cited by 41 publications

References 40 publications

GREMI: an Explainable Multi-omics Integration Framework for Enhanced Disease Prediction and Module Identification

GREMI: an Explainable Multi-omics Integration Framework for Enhanced Disease Prediction and Module Identification

HybridMIM: A Hybrid Masked Image Modeling Framework for 3D Medical Image Segmentation

CLCLSA: Cross-omics Linked embedding with Contrastive Learning and Self Attention for multi-omics integration with incomplete multi-omics data

Contact Info

Product

Resources

About