Adaptive Fusion Techniques for Multimodal Data

Sahu, Gaurav; Vechtomova, Olga

doi:10.48550/arxiv.1911.03821

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Auto-fusion: In this work [28], an adaptive fusion is proposed that allows for effective multimodal fusion. Instead of using static techniques such as concatenation, it lets the network decide how to combine a given set of multimodal features more effectively.…”

Section: Adaptive Fusionmentioning

confidence: 99%

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

Amel,

Siebert,

Mahmoudi

2024

Electronics

View full text Add to dashboard Cite

The growing demand for advanced tools to ensure safety in railway construction projects highlights the need for systems that can smoothly integrate and analyze multiple data modalities, such as multimodal learning algorithms. The latter, inspired by the human brain’s ability to integrate many sensory inputs, has emerged as a promising field in artificial intelligence. In light of this, there has been a rise in research on multimodal fusion approaches, which have the potential to outperform standard unimodal solutions. However, the integration of multiple data sources presents significant challenges to be addressed. This work attempts to apply multimodal learning to detect dangerous actions using RGB-D inputs. The key contributions include the evaluation of various fusion strategies and modality encoders, as well as identifying the most effective methods for capturing complex cross-modal interactions. The superior performance of the MultConcat multimodal fusion method was demonstrated, achieving an accuracy of 89.3%. Results also underscore the critical need for robust modality encoders and advanced fusion techniques to outperform unimodal solutions.

show abstract

Section: Adaptive Fusionmentioning

confidence: 99%

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

Amel,

Siebert,

Mahmoudi

2024

Electronics

View full text Add to dashboard Cite

show abstract

“…Abavisani et al [30] provided an autoencoder-based approach for multimodal clustering to project data into the latent space representation. The automatic fusion method [31] alleviates the static nature of existing fusion methods, effectively combining multimodal inputs through autoencoders. Autoencoder-based fusion method is irregular and successfully escapes the limitations of linear projection, thus enabling a more accurate fit to the common feature space.…”

Section: B Autoencoder-based Fusionmentioning

confidence: 99%

Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity

Huang

Wang

2023

IEEE Access

View full text Add to dashboard Cite

Multimodal biometric system has received increasing interest as it offers a more secure and accurate authentication solution than unimodal systems. However, existing biometric fusion methods are still inadequate in dealing with correlations and redundancy of multimodal features simultaneously, causing bottlenecks in performance improvement. To overcome the above problem, this paper proposes an end-toend multimodal finger recognition model that incorporates attention mechanisms into a similarity-aware encoder to achieve accurate recognition results for accurate identification results. Firstly, due to the different distribution of fingerprint and finger vein images, we propose a finger asymmetric backbone network (FAB-Net) for extracting discriminative intra-modal features, which reduces the network width by efficient utilization of feature maps. Then, a novel attention-based encoder fusion network (AEF-Net) with fused similarity performs dimensionality reduction-based fusion on multimodal multilevel features to alleviate performance degradation due to information redundancy. We also introduce channel attention in AEF-Net, which differs from the traditional attention mechanism by considering interdependencies between modalities to further improve performance. Extensive recognition experiments are conducted on three multimodal finger databases to verify the effectiveness of our method compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrated that encoder-based reconstruction of redundant information can improve recognition performance.INDEX TERMS Multimodal biometric recognition, feature fusion, autoencoder, deep learning.

show abstract

“…In the joint embedding model, they minimize the distance of the outputs of the deep video model and compositional language model in the joint space and update these two models jointly. Sahu et al [35] first encode all modalities, then use decoding to restore features, and finally calculate the loss between features.…”

Section: Modality Aggregationmentioning

confidence: 99%

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Chai

Wang

2022

Applied Sciences

View full text Add to dashboard Cite

Deep vision multimodal learning aims at combining deep visual representation learning with other modalities, such as text, sound, and data collected from other sensors. With the fast development of deep learning, vision multimodal learning has gained much interest from the community. This paper reviews the types of architectures used in multimodal learning, including feature extraction, modality aggregation, and multimodal loss functions. Then, we discuss several learning paradigms such as supervised, semi-supervised, self-supervised, and transfer learning. We also introduce several practical challenges such as missing modalities and noisy modalities. Several applications and benchmarks on vision tasks are listed to help researchers gain a deeper understanding of progress in the field. Finally, we indicate that pretraining paradigm, unified multitask framework, missing and noisy modality, and multimodal task diversity could be the future trends and challenges in the deep vision multimodal learning field. Compared with existing surveys, this paper focuses on the most recent works and provides a thorough discussion of methodology, benchmarks, and future trends.

show abstract

Adaptive Fusion Techniques for Multimodal Data

Cited by 5 publications

References 0 publications

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

Multimodal Finger Recognition Based on Asymmetric Networks With Fused Similarity

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Contact Info

Product

Resources

About