Balanced Multimodal Learning via On-the-fly Gradient Modulation

Peng, Xiaokang; Wei, Yake; Deng, Andong; Wang, Dong; Hu, Di

doi:10.48550/arxiv.2203.15332

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is also a class of cases where the problem of unbalanced weights between modalities occurs. Peng et al [234] point out that during multimodal training, one dominant modality can inhibit the training of another modality, resulting in the failure to realize the full potential of multimodality, which is also a matter of concern.…”

Section: Missing and Noisy Modalitymentioning

confidence: 99%

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Chai

Wang

2022

Applied Sciences

View full text Add to dashboard Cite

Deep vision multimodal learning aims at combining deep visual representation learning with other modalities, such as text, sound, and data collected from other sensors. With the fast development of deep learning, vision multimodal learning has gained much interest from the community. This paper reviews the types of architectures used in multimodal learning, including feature extraction, modality aggregation, and multimodal loss functions. Then, we discuss several learning paradigms such as supervised, semi-supervised, self-supervised, and transfer learning. We also introduce several practical challenges such as missing modalities and noisy modalities. Several applications and benchmarks on vision tasks are listed to help researchers gain a deeper understanding of progress in the field. Finally, we indicate that pretraining paradigm, unified multitask framework, missing and noisy modality, and multimodal task diversity could be the future trends and challenges in the deep vision multimodal learning field. Compared with existing surveys, this paper focuses on the most recent works and provides a thorough discussion of methodology, benchmarks, and future trends.

show abstract

Section: Missing and Noisy Modalitymentioning

confidence: 99%