TransVG: End-to-End Visual Grounding with Transformers

Abstract

In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.

FindIt: Generalized Localization with Natural Language Queries

“…To make the image size uniform across tasks (see Table 1), we adopt the LOC task's image size of 640 [21] as a middle ground. This is larger but comparable to the image size of REC task [89,88,13]. It is smaller than the size of DET task's images [23] which might limit performance on smaller objects.…”

“…Concretely, we utilize the more expressive cross-attention fusion on lower resolution features, and the more efficient product fusion on higher resolution features to combine the best of both worlds. Last but not least, we discover that a standard object detector and detection losses [69] are sufficient and surprisingly effective for REC, LOC, and DET tasks without a need for task-specific design and losses [13,21,51,55,88,89,91]. In short, FindIt is a simple, efficient, and end-to-end trainable model for unified visual grounding and object detection.…”

“…Yu et al and Mao et al [92,56] expand the COCO benchmark with referring expression annotations, while the Referit game [72] crowd-sources such labels through game-play. One-stage [7,44,89,13,55] and two-stage [94,91,27,83,87,50] methods have been popular for these tasks.…”

“…To localize objects at various scales, existing REC works have used multi-level fusion by applying activation and product fusion [55,88] or concatenation and convolution fusion [89]. Inspired by recent works [13,8,53,37] on single-scale cross-attention, we propose multi-scale fusion to satisfy the disparate requirements of REC and detection tasks, where REC requires complex reasoning while detection requires accurate localization and recognition. The fusion module enables us to unify these tasks in a single model and surpass the state of the art on REC, LOC and maintains competitive DET.…”

“…All losses have equal weights across tasks. We note that it is unclear how to use existing grounding models out-of-the-box for task unification due to the task-specific architectures, losses, and training strategies [13,21,51,55,88,89,91].…”