Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding

Abstract

In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21%), RefCOCO+ (39.18%) and RefCOCOg (43.24%) datasets, that is 4.17%, 4.08% and 7.8% higher than the previous one, respectively. The code is available at https://github.com/insomnia94/DTWREG.

Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding

“…We report comparison results with existing unsupervised [54,62,63] and weakly-supervised [38,49,55] methods. Note that the weakly-supervised methods are trained with expensive annotated queries.…”

“…Visual grounding is a crucial component in vision and language, and it serves as the fundamental of other tasks, such as VQA. Recent visual grounding methods can be summarized into three categories: fully-supervised [8,13,22,23,33,35], weakly-supervised [6,10,19,36,38,49,55,58], and unsupervised [54,63]. Fully-supervised methods rely heavily on the manual labeled patch-query pairs.…”

“…Previous visual grounding methods rely on expensive human annotations, i.e., image region-query pairs for fullysupervised approaches [13,22,35] or image-query pairs for weakly-supervised approaches [36,37,49]. We firstly propose a pseudo language query based method without using any task-related annotations at training.…”

“…Most existing visual grounding methods can be categorized into two types: fully-supervised [8,13,22,23,33,35] and weakly-supervised [6,10,19,36,38,49,55,58]. Although these two lines of works have made remarkable successes, they rely heavily on manually annotated datasets.…”

Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning

“…To extract the linguistic feature f l q , q is first parsed into multiple discriminative triads {t k } M k=1 [44]. Each triad represents a piece of discriminative information to distinguish the target from the distracting or reference objects.…”