Learning Customized Visual Models with Retrieval-Augmented Knowledge

Abstract

Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-scale data collection process to ensure broad concept coverage, followed by expensive pre-training to feed all the knowledge into model weights. Alternatively, we propose REACT, REtrieval-Augmented CusTomization, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs (∼3% of CLIP pre-training data) from the web-scale database as external knowledge, and propose to customize the model by only training new modualized blocks while freezing all the original weights. The effectiveness of REACT is demonstrated via extensive experiments on classification, retrieval, detection and segmentation tasks, including zero, few, and full-shot settings. Particularly, on the zero-shot classification task, compared with CLIP, it achieves up to 5.4% improvement on ImageNet and 3.7% on the ELEVATER benchmark (20 datasets).

Learning with Neighbor Consistency for Noisy Labels

“…Several works have focused on ways of improving upon different aspects of the contrastive vision-text models, such as their training objectives [12,15,66] or through scaling [9,47]. Yet, only little exploration has been done on their combination with memory or knowledge-based techniques [2,14,37,54]. REACT [37] retrieves image-text pairs from an external memory in order to build a training dataset specialized for a specific downstream task.…”

“…Yet, only little exploration has been done on their combination with memory or knowledge-based techniques [2,14,37,54]. REACT [37] retrieves image-text pairs from an external memory in order to build a training dataset specialized for a specific downstream task. Unlike REACT [37], our work does not require any pre-knowledge about the nature of the downstream task, and is hence applicable in a full zero-shot transfer.…”

“…REACT [37] retrieves image-text pairs from an external memory in order to build a training dataset specialized for a specific downstream task. Unlike REACT [37], our work does not require any pre-knowledge about the nature of the downstream task, and is hence applicable in a full zero-shot transfer. Another key difference is that our model can leverage items from the memory at inference time, while REACT uses retrieved items to automatically generate a training set to finetune their model.…”

“…Retrieval-based methods have shown their promise in various NLP tasks [4,21,30,34,60,62]. More recently, there is an increasing interest in the computer vision for retrieval-based methods as well [19,22,24,26,37,38,54]. Chen et al [7] and Blattmann et al [3] retrieve from an external memory for generative vision modeling.…”

“…However, one can wonder if this is a viable direction. An orthogonal effort has focused instead on memory or knowledge-based approaches [19,22,24,26,37,38,54]. These methods, instead of statically ingesting and memorizing all the world knowledge into model parameters, propose to rely on the access to an external source of knowledge.…”