Deep Multimodal Neural Architecture Search

Yu, Zhou; Cui, Yuhao; Yu, Jun; Wang, Meng; Tao, Dacheng

doi:10.1145/3394171.3413977

Cited by 63 publications

(24 citation statements)

References 43 publications

(72 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(1) The framework does not have special requirements on the input modes, which means we can smoothly choose the inputs from commonly-used modalities such as image, spectrogram, skeleton, etc., according to our needs. (2) Compared with previous works (Pérez-Rúa et al 2019;Yu et al 2020), which aim to search for the entire network architectures, our fusion structures are relatively independent modules. Thus, our SepFusion also possesses good compatibility and can be easily plugged into existing pipelines.…”

Section: Skeleton-guided Sound Separation Stereo Generationmentioning

confidence: 99%

“…Both the appearance and motion modalities belong to the vision domain, while basically larger gaps may appear between cross-censor modalities. Some works try to solve the cross-sensor modality fusion problem by finding ideal feature connections in VQA (Gao et al 2019;Yu et al 2020) and audio-visual classification tasks (Pérez-Rúa et al 2019). These previous works mainly focus on the high-level scenes, but our work can deal with the finegrained dense prediction tasks on the pixel level.…”

Section: Related Workmentioning

confidence: 99%

“…These previous works primarily target homogeneous features of similar semantics, but our work can handle more diverse features even of various sensor domains. Some works (Pérez-Rúa et al 2019;Yu et al 2020) also explore the application of NAS algorithms to find structures for features from different modalities. The searching objective of these works is the entire processing system, while our SepFusion focuses on an independent fusion module, which is more compatible with existing frameworks.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation

Zhou

et al. 2022

AAAI

View full text Add to dashboard Cite

Multiple modalities can provide rich semantic information; and exploiting such information will normally lead to better performance compared with the single-modality counterpart. However, it is not easy to devise an effective cross-modal fusion structure due to the variations of feature dimensions and semantics, especially when the inputs even come from different sensors, as in the field of audio-visual learning. In this work, we propose SepFusion, a novel framework that can smoothly produce optimal fusion structures for visual-sound separation. The framework is composed of two components, namely the model generator and the evaluator. To construct the generator, we devise a lightweight architecture space that can adapt to different input modalities. In this way, we can easily obtain audio-visual fusion structures according to our demands. For the evaluator, we adopt the idea of neural architecture search to select superior networks effectively. This automatic process can significantly save human efforts while achieving competitive performances. Moreover, since our SepFusion provides a series of strong models, we can utilize the model family for broader applications, such as further promoting performance via model assembly, or providing suitable architectures for the separation of certain instrument classes. These potential applications further enhance the competitiveness of our approach.

show abstract

Section: Skeleton-guided Sound Separation Stereo Generationmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation

Zhou

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…However, in RGB-D salient object detection, the multimodal feature fusion architectures are still designed by hand. Although there are several NAS works [46,57] for multi-modal fusion, their design purpose is especially for the visual question answering task [57] or image-audio fusion task [46]. As far as we know, our work is the first attempt to utilize the NAS algorithms to tackle the multimodal multi-scale feature fusion problem for RGB-D SOD.…”

Section: Neural Architecture Searchmentioning

confidence: 99%

Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion

Sun¹,

Zhang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

RGB-D salient object detection (SOD) is usually formulated as a problem of classification or regression over two modalities, i.e., RGB and depth. Hence, effective RGB-D feature modeling and multi-modal feature fusion both play a vital role in RGB-D SOD. In this paper, we propose a depth-sensitive RGB feature modeling scheme using the depth-wise geometric prior of salient objects. In principle, the feature modeling scheme is carried out in a depth-sensitive attention module, which leads to the RGB feature enhancement as well as the background distraction reduction by capturing the depth geometry prior. Moreover, to perform effective multi-modal feature fusion, we further present an automatic architecture search approach for RGB-D SOD, which does well in finding out a feasible architecture from our specially designed multi-modal multi-scale search space. Extensive experiments on seven standard benchmarks demonstrate the effectiveness of the proposed approach against the state-of-the-art.

show abstract

“…Compared to earlier methods that are only adapted to one V+L task [46,47,49], VLP models is generalizable across multiple tasks and also achieves significantly better performance on respective tasks. Learning fine-grained semantic alignments between image regions and text words plays a key role in V+L tasks.…”

Section: Introductionmentioning

confidence: 99%

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Cui

Wang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cROSs-and InTrA-modal knowledge in a unified scene graph to enhance the semantic alignments. Specifically, we introduce a novel structural knowledge masking (SKM) strategy to use the scene graph structure as a priori to perform masked language (region) modeling, which enhances the semantic alignments by eliminating the interference information within and across modalities. Extensive ablation studies and comprehensive analysis verifies the effectiveness of ROSITA in semantic alignments. Pretrained with both in-domain and out-ofdomain datasets, ROSITA significantly outperforms existing stateof-the-art VLP methods on three typical vision-and-language tasks over six benchmark datasets. CCS CONCEPTS• Computing methodologies → Multi-task learning.

show abstract

Deep Multimodal Neural Architecture Search

Cited by 63 publications

References 43 publications

SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation

SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation

Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Contact Info

Product

Resources

About