A Fast and Accurate One-Stage Approach to Visual Grounding

Abstract

We propose a simple, fast, and accurate one-stage approach to visual grounding, inspired by the following insight. The performances of existing propose-and-rank twostage methods are capped by the quality of the region candidates they propose in the first stage -if none of the candidates could cover the ground truth region, there is no hope in the second stage to rank the right region to the top. To avoid this caveat, we propose a one-stage model that enables end-to-end joint optimization. The main idea is as straightforward as fusing a text query's embedding into the YOLOv3 object detector, augmented by spatial features so as to account for spatial mentions in the query. Despite being simple, this one-stage approach shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension, according to our experiments. Given these results along with careful investigations into some popular region proposals, we advocate for visual grounding a paradigm shift from the conventional two-stage methods to the one-stage framework.

Language-Aware Fine-Grained Object Representation for Referring Expression Comprehension

“…The task of referring expression comprehension has attracted increasing attention in recent years, which expects to locate corresponding objects within an image based on input expressions. Previous referring expression comprehension methods [2, 6, 11, 16, 22, 25, 27, 28, 35, 41-43, 45, 47-51] can be mainly divided into two types, including proposal-region-based [2,6,11,22,25,27,28,[41][42][43][47][48][49][50][51] and grid-region-based methods [16,35,45] .…”

“…Grid-region-based methods [16,35,45] usually fuse the language features with grid region features, and then leverage one-stage object detectors (e.g. YOLOv3 [33]) to directly localize the object corresponding to the input expression.…”

“…To model the relationships between language and vision, existing methods [2, 16, 22, 25, 27, 28, 35, 41-43, 45, 47-50] usually combine the language features and the regular image region features, such as object proposal regions [2,22,25,27,28,[41][42][43][47][48][49][50] and grid regions [16,35,45], as shown in Figure 1 (a) and (b), respectively. However, these methods ignore some fine-grained object information related to the natural language, such as object shapes and poses, which are often described in language expressions and important in referring expression comprehension to localize and distinguish the target objects.…”

Cross-Modal Omni Interaction Modeling for Phrase Grounding

“…Bajaj et al [3] achieved significant improvement by using Gated Graph Neural Networks to formulate the dependency among phrases and image regions. Yang et al [51] proposed a one-stage approach, fusing text query embeddings into the YOLOv3 object detector while augmenting by using spatial features. Lai et al [29] proposed to use transformers to capture contextual representations for text tokens and image regions.…”

Activity-driven Weakly-Supervised Spatio-Temporal Grounding from Untrimmed Videos

“…description sentence, for example "break the eggs", visual grounding aims at localizing the query objects described in the sentence on the given image or video. Recently, great progress has been made on image grounding [4,15,30,31]. On the basis of this, researchers started to explore grounding in the video domain [5,7,14,25,35].…”