Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition

Boulahia, Said Yacine; Amamra, Abdenour; Madi, Mohamed Ridha; Daikh, Said

doi:10.1007/s00138-021-01249-8

Cited by 72 publications

(47 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our multimodal strategy is decoupled from any particular artificial neural network architecture, and thus, it can be easily integrated into an existing multimodal pipeline. Practically multimodal learning is done utilizing different data fusion strategies, i.e., early fusion, middle fusion, and late fusion [47][48][49].…”

Section: Deeptlf and Multimodal Datamentioning

confidence: 99%

DeepTLF: robust deep neural networks for heterogeneous tabular data

Borisov

Broelemann²,

Kasneci

et al. 2022

Int J Data Sci Anal

View full text Add to dashboard Cite

Although deep neural networks (DNNs) constitute the state of the art in many tasks based on visual, audio, or text data, their performance on heterogeneous, tabular data is typically inferior to that of decision tree ensembles. To bridge the gap between the difficulty of DNNs to handle tabular data and leverage the flexibility of deep learning under input heterogeneity, we propose DeepTLF, a framework for deep tabular learning. The core idea of our method is to transform the heterogeneous input data into homogeneous data to boost the performance of DNNs considerably. For the transformation step, we develop a novel knowledge distillations approach, TreeDrivenEncoder, which exploits the structure of decision trees trained on the available heterogeneous data to map the original input vectors onto homogeneous vectors that a DNN can use to improve the predictive performance. Within the proposed framework, we also address the issue of the multimodal learning, since it is challenging to apply decision tree ensemble methods when other data modalities are present. Through extensive and challenging experiments on various real-world datasets, we demonstrate that the DeepTLF pipeline leads to higher predictive performance. On average, our framework shows 19.6% performance improvement in comparison to DNNs. The DeepTLF code is publicly available.

show abstract

Section: Deeptlf and Multimodal Datamentioning

confidence: 99%

DeepTLF: robust deep neural networks for heterogeneous tabular data

Borisov

Broelemann²,

Kasneci

et al. 2022

Int J Data Sci Anal

View full text Add to dashboard Cite

show abstract

“…After extracting multimodal features, it is important to aggregate them together. The more typical fusion methods [25,26] are divided into early fusion and late fusion. Early fusion refers to directly splicing the multimodal input signals and then sending them into a unified network structure for training.…”

Section: Modality Aggregationmentioning

confidence: 99%

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Chai

Wang

2022

Applied Sciences

View full text Add to dashboard Cite

Deep vision multimodal learning aims at combining deep visual representation learning with other modalities, such as text, sound, and data collected from other sensors. With the fast development of deep learning, vision multimodal learning has gained much interest from the community. This paper reviews the types of architectures used in multimodal learning, including feature extraction, modality aggregation, and multimodal loss functions. Then, we discuss several learning paradigms such as supervised, semi-supervised, self-supervised, and transfer learning. We also introduce several practical challenges such as missing modalities and noisy modalities. Several applications and benchmarks on vision tasks are listed to help researchers gain a deeper understanding of progress in the field. Finally, we indicate that pretraining paradigm, unified multitask framework, missing and noisy modality, and multimodal task diversity could be the future trends and challenges in the deep vision multimodal learning field. Compared with existing surveys, this paper focuses on the most recent works and provides a thorough discussion of methodology, benchmarks, and future trends.

show abstract

“…Different sensors can provide different perception information on the same context. Studies [31,32] have confirmed that the feature fusion strategy of different convolutional layer feature maps can effectively solve these problems. Multimodal fusion uses different models to extract different perception features and combine them to improve the performances obtained by using only one modality.…”

Section: Multi-feature Fusion Cnnsmentioning

confidence: 99%

“…There are three types of multimodal fusion architectures: early fusion, late fusion, and intermediate fusion [31,32]. An early fusion architecture [33,34] integrates multiple data before conducting the analysis.…”

Section: Multi-feature Fusion Cnnsmentioning

confidence: 99%

Multi-Scale Features Fusion Convolutional Neural Networks for Rice Leaf Disease Identification

Wang¹,

Li²,

Chan³

et al. 2022

jist

View full text Add to dashboard Cite

It is time-consuming and labor-intensive to detect rice diseases manually. The purpose of this research is to develop a convolutional neural networks (CNNs)-based system to automatically detect the diseased rice leaf infected with rice leaf blast, helminthosporium leaf blight, and bacterial leaf blight. The sizes of rice leaf spots vary with the severity of disease infection. A single model based CNN cannot effectively classify images, especially for images with objects of small size as well as multiple object scales, and complicated image background. In this research, a multiscale serial convolution neural network (MSSCNN) and a multiscale parallel convolution neural network (MSPCNN) are proposed to identify diseased rice leaf spots based on multi-modal fusion to extract different perception features and combine them to improve the performances obtained by using only one modality. Experimental results delineate that MSSCNN and MSPCNN can get better performance in identifying the diseased rice leaves. In MSPCNN, the features of tiny spots on diseased rice leaves can be completely preserved. MSPCNN can hence offer better performances than MSSCNN. Additionally, MSPCNN architecture is suitable for parallel computing environment.

show abstract

Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition

Cited by 72 publications

References 40 publications

DeepTLF: robust deep neural networks for heterogeneous tabular data

DeepTLF: robust deep neural networks for heterogeneous tabular data

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Multi-Scale Features Fusion Convolutional Neural Networks for Rice Leaf Disease Identification

Contact Info

Product

Resources

About