Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning

Abstract

Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained state-of-the-art results in image-to-text generation. However, these models store all the knowledge within their parameters, thus often requiring enormous model parameters to model the abundant visual concepts and very rich textual descriptions. Additionally, they are inefficient in incorporating new data, requiring a computationalexpensive fine-tuning process. In this work, we introduce a Retrieval-augmented Visual Language Model, Re-ViLM, built upon the Flamingo, that supports retrieving the relevant knowledge from the external database for zero and in-context fewshot image-to-text generations. By storing certain knowledge explicitly in the external database, our approach reduces the number of model parameters and can easily accommodate new data during evaluation by simply updating the database. We also construct an interleaved image and text data that facilitates in-context few-shot learning capabilities. We demonstrate that Re-ViLM significantly boosts performance for image-to-text generation tasks, especially for zero-shot and fewshot generation in out-of-domain settings with 4× less parameters compared with baseline methods.* Equal contribution . ‡ Work done during an internship at NVIDIA. † Equal advising 1 NVIDIA 2 UIUC. 3 UT Austin 4 ASU 5 Caltech.

An Outbreak of Fungal Endophthalmitis After Cataract Surgery in South Korea

“…In addition, they introduced contrastive learning [17], similarity-bucket, retrieval-augmented [37] Captions in NICE dataset often contain new concepts like camera angle descriptions and proper nouns like place names, which are difficult to predict under zero-shot settings. Motivated from retrieval-augmented models [5,20], the RETRIEVER framework aims to complement such lim-ited data condition by efficiently utilizing external knowledge in model training and inference.…”

Application of 3D Printing and WebGL-Based 3D Visualisation Technology in Imaging Teaching of Ankle Joints

“…Recently retrieval-augmented generation (RAG) attracted increasing attention of both the natural language processing [1,9,10,2] and vision-and-language [3,4,18]. REALM [1] uses the query to retrieve the top 𝑘 most relevant article snippets, and uses large language models (LLMs) to generate 𝑘 responses, which are then combined to obtain a final output for question answering.…”

“…The memory, encoder, retriever, and generator are pre-trained in an end-to-end manner. Re-ViLM [4] augments Flamingo [5], by retrieving relevant image-text pairs from the external image-text datasets [6,7,8] for zero and in-context few-shot image-to-text generations. RA-CM3 [40] performs retrieval from an external memory for generating images and text.…”