Dynamic Graph Attention for Referring Expression Comprehension

Abstract

Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.

Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge

“…Liu et al [15] develop a neural module tree network to regularize the visual grounding along the dependency parsing tree of the sentence. The works in [3,36,42] argue to learn the representations from expression and image regions in a stepwise manner, and perform multi-step reasoning for better matching performance. Wang et al [30] propose a graph-based language-guided attention network to highlight the inter-object and intra-object relationships that are closely relevant to the expression for better performance.…”

Language-Aware Fine-Grained Object Representation for Referring Expression Comprehension

“…The task of referring expression comprehension has attracted increasing attention in recent years, which expects to locate corresponding objects within an image based on input expressions. Previous referring expression comprehension methods [2, 6, 11, 16, 22, 25, 27, 28, 35, 41-43, 45, 47-51] can be mainly divided into two types, including proposal-region-based [2,6,11,22,25,27,28,[41][42][43][47][48][49][50][51] and grid-region-based methods [16,35,45] .…”

“…Most of proposal-region-based methods [2,22,[41][42][43]47, 50] are based on the "listener" strategy, which first combines language features with visual features of proposal regions, and then select the target region that best matches the input expression from these proposals. The proposal regions are typically extracted by a pretrained object detector (e.g., Faster R-CNN [34], Mask R-CNN [9] and others [4,17,[30][31][32]).…”

“…To align visual regions with the expression more powerful, methods in [22,47] proposed to decompose the expression into three components (subject, localization and relationship), and leveraged cross-modality attentions to focus on relevant regions. To adaptively grounding complex referring expressions, a dynamic graph attention network in [43] performed multi-step visual reasoning based on the multi-modal context relationship graph to identify the compound objects step by step. In several works [25,27,28,48], a "speaker" strategy first predicted expressions from the visual features of every object and then matched the predicted expressions with the input expression to find out the desired object.…”

“…To model the relationships between language and vision, existing methods [2, 16, 22, 25, 27, 28, 35, 41-43, 45, 47-50] usually combine the language features and the regular image region features, such as object proposal regions [2,22,25,27,28,[41][42][43][47][48][49][50] and grid regions [16,35,45], as shown in Figure 1 (a) and (b), respectively. However, these methods ignore some fine-grained object information related to the natural language, such as object shapes and poses, which are often described in language expressions and important in referring expression comprehension to localize and distinguish the target objects.…”