Generalized Zero-Shot Learning Using Multimodal Variational Auto-Encoder With Semantic Concepts

Abstract

With the ever-increasing amount of data, the central challenge in multimodal learning involves limitations of labelled samples For the task of classification, techniques such as meta-learning, zero-shot learning, and few-shot learning showcase the ability to learn information about novel classes based on prior knowledge . Recent techniques try to learn a cross-modal mapping between the semantic space and the image space. However, they tend to ignore the local and global semantic knowledge. To overcome this problem, we propose a Multimodal Variational Auto-Encoder (M-VAE) which can learn the shared latent space of image features and the semantic space. In our approach we concatenate multimodal data to a single embedding before passing it to the VAE for learning the latent space. We propose the use of a multi-modal loss during the reconstruction of the feature embedding through the decoder. Our approach is capable to correlating modalities and exploit the local and global semantic knowledge for novel sample predictions. Our experimental results using a MLP classifier on four benchmark datasets show that our proposed model outperforms the current state-of-the-art approaches for generalized zero-shot learning.

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…Each of the three articles in the biology domain pertained to recognition [397], prediction [398] and identification [399] accordingly. In total, 4 bird species domain relevant papers were identified, where 3 were in classification [45], [400], [401] and 1 was in integration [402]. There were 2 articles each in the signal processing and gender domains.…”

“…Of eight articles in parallel, seven pertained to co-training [66], [192], [209], [210], [329], [376], [379] and one related to transfer learning [201]. On the other side, from 25 non-parallel articles, 11 pertained to transfer learning [83], [107], [108], [146], [208], [215], [223], [246], [285], [286], [333], four related to concept grounding [82], [219], [332], [335] and 10 related to zeroshot learning [84], [85], [175], [191], [224], [247], [287], [377], [400], [401]. In hybrid co-learning, two articles were related to bridging [195], [196].…”

“…Conceptual grounding shares the semantic concept with modalities where most of them are related to linguistics, such as [219], [332], [335]. Zero-shot learning classifies data without having any labels, and approaches such as cross-models [191], [224] and auto-encoders [400], [401] are used for the solution. Bridging is hybrid co-learning where two non-parallel modalities share information.…”

A review on multimodal zero‐shot learning

“…A novel multimodal variational auto‐encoder (M‐VAE) algorithm for GZSL was proposed in Bendre et al (2021), which uses two modalities, images, and semantic concepts, as the input of the encoder. The encoder maps the input of these two modalities to a unified latent space, and the decoder reconstructs the latent embedding vector into the corresponding visual features.…”

A Unified Software/Hardware Scalable Architecture for Brain-Inspired Computing Based on Self-Organizing Neural Models

“…They achieve state-of-the-art results for solving particular machine learning problems. It is practically impossible to analyze all of them, but a significant number of them, at one step or another, use the classical concatenation of multimodal vectors (Chen et al, 2020 ; Xie et al, 2020 ; Bendre et al, 2021 ), without a deep examination of unique dependencies between them. Nevertheless, there are other models proposing smarter modality aggregation, such as the Contrastive Multimodal Fusion method (Liu et al, 2021 ), showing there is growing interest in the ML community for nontrivial multimodal fusion.…”