RegionCLIP: Region-based Language-Image Pretraining

Abstract

Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification in both zero-shot and transfer learning settings. However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans. To mitigate this issue, we propose a new method called RegionCLIP that significantly extends CLIP to learn region-level visual representations, thus enabling fine-grained alignment between image regions and textual concepts. Our method leverages a CLIP model to match image regions with template captions, and then pretrains our model to align these region-text pairs in the feature space. When transferring our pretrained model to the open-vocabulary object detection task, our method outperforms the state of the art by 3.8 AP50 and 2.2 AP for novel categories on COCO and LVIS datasets, respectively. Further, the learned region representations support zero-shot inference for object detection, showing promising results on both COCO and LVIS datasets. Our code is available at https://github.com/microsoft/RegionCLIP.

Localized Vision-Language Matching for Open-vocabulary Object Detection

“…To learn the semantics of novel classes, recent methods [3,13,16,41,44] have simplified the problem by providing image-caption pairs as a weak supervision signal. Such pairs are cheap to acquire and make the problem tractable.…”

“…Most of these methods require big dataset with millions of image-caption pairs to train such a model. They either use this model to align image-regions with captions and generate object-box pseudo labels [16,44] or as region-image feature extractor to classify the regions [13]. Many weakly-supervised [1,3,7,34,43] approaches have been proposed to perform such object grounding.…”

“…We compare our method with recent state-of-the-art models on Open-Vocabulary. RegionClip [44] uses the CLIP [30] pre-trained model to produce region-image pseudo labels and train an object detector. CLIP (cropped reg) [13] uses the CLIP pre-trained model on 400M image-caption pairs on object proposals obtained by an object detector trained on known classes.…”

“…The closed-world setting restricts the object detector to only discover known annotated objects and annotating all possible objects in the world is infeasible due to high labeling costs. Therefore, research of open-world detectors, which can also discover unmarked objects, has recently come into focus [13,41,44].…”

Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization

“…Recently, open-vocabulary object detection (OVD) [49] has attracted increasing attention due to its ability to expand the detection vocabulary with the help of pre-trained vision-language models (VLMs) [32]. Typical OVD methods [14,51,53] first learn an unbounded vocabulary of concepts from image-caption pairs, and then transfer the general vision-language knowledge to facilitate OVD with detection annotations of base categories alone.…”