Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning

Abstract

Multimodal few-shot learning is challenging due to the large domain gap between vision and language modalities. Existing methods are trying to communicate visual concepts as prompts to frozen language models, but rely on hand-engineered task induction to reduce the hypothesis space. To make the whole process learnable, we introduce a multimodal meta-learning approach. Specifically, our approach decomposes the training of the model into a set of related multimodal few-shot tasks. We define a meta-mapper network, acting as a meta-learner, to efficiently bridge frozen large-scale vision and language models and leverage their already learned capacity. By updating the learnable parameters only of the meta-mapper, it learns to accrue shared meta-knowledge among these tasks. Thus, it can rapidly adapt to newly presented samples with only a few gradient updates. Importantly, it induces the task in a completely data-driven manner, with no need for a hand-engineered task induction. We evaluate our approach on recently proposed multimodal few-shot benchmarks, measuring how rapidly the model can bind novel visual concepts to words and answer visual questions by observing only a limited set of labeled examples. The experimental results show that our meta-learning approach outperforms the baseline across multiple datasets and various training settings while being computationally more efficient.

VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning

“…Frozen [56] freezes the language decoder and trains the vision encoder, which is also the baseline of this problem. MML [54] is a method that uses a mapper to convert the vision feature into language feature space, which trains by using the meta learning to construct a new training dataset. Our method VL-Few proposes to use the learnable vision prompt and vision-language prompt to carry out the vision information to make the model learn images, which could help the model understand the images that are the different modal data.…”

“…However, this method requires a large amount of data to train the visual encoder, thus making it difficult to apply to small sample scenarios. MML [54] proposes the meta mapper to map the image features to language feature space. Specifically, MML extracts image features through a visual encoder and inputs them together with a visual prefix into the meta mapper.…”

“…There are some differences between other research studies and ours. For model architecture, other methods require retraining of the vision model (Frozen) or only consider mapping vision image features and language question features into the same (CLIPCap [53]) or fewer (MML [54]) tokens through mappers to help the model understand vision image features. However, our method realizes that the prominent problem with understanding multimodal data is that the features are not in the same feature space, and the failure to align multimodal features effectively results in the model being unable to understand other modal data.…”

VL-Meta: Vision-Language Models for Multimodal Meta-Learning

“…It also constructs a meta-learning dataset to make the model learn how to learn and perform a task that has not been seen before. MML [40] assumes that Frozen [39] needs to re-train the vision model, which requires much computing power and time. So, it designs a light module that can obtain the vision features by using four learnable prefix tokens and then extracts these four learnable prefix tokens as the image embedding and concatenates them with the text embedding.…”

“…But it needs much data and computing power to train the vision encoder, which is not friendly for low-source learners. MML [40] proposes a mapper module to convert the vision feature to language feature space with limited data. It can reduce the training computation, but the vision feature cannot learn the language information, which is important for multimodal tasks.…”