Modeling Context in Referring Expressions

Yu, Licheng; Poirson, Patrick; Yang, Shan; Berg, Alexander C.; Berg, Tamara L.

doi:10.1007/978-3-319-46475-6_5

Cited by 596 publications

(774 citation statements)

References 34 publications

Supporting

Mentioning

772

Contrasting

Order By: Relevance

“…To learn the common feature space, they propose different matching loss functions to optimize, e.g., softmax loss [16,21] and triplet loss [25]. Another work [18,30,19] learns to maximize the likelihood of the expression given the referent and the image, and the work inputs the fusion of visual object feature, visual context feature (e.g., entire image CNN feature [18], the visual difference between the objects belonging to the same category in the image [30] and context region CNN features [19]), object location feature and the word embedding to an LSTM to parameterize the distribution. Different from the previous work, recent work [32,4] adopts co-attention mechanisms to build up the interactions between the expression and the objects in the image.…”

Section: Referring Expression Comprehensionmentioning

confidence: 99%

“…Those approaches ignore the relationships among objects in the image and the linguistic structure in the expression, which is the key to referring expression comprehension. For an image, they represent the image as a set of independent visual objects [16,21,25,13,18] or compound objects only including direct relationship [19,30]. For an expression, they encode the expression sequentially and ignore the dependencies in the expression.…”

Section: Referring Expression Comprehensionmentioning

confidence: 99%

“…Therefore, we resort to a differential analyzer module to predict constituent expressions of the input expression step by step to capture the linguistic structure, and the input expression is represented as a sequence of constituent expressions. Second, it is necessary to take into consideration the relationships among the objects in the image because unambiguous referring expressions normally not only describe the attributes of the referent itself, but also its relationships to other objects in the image [30,7,27]. Therefore, the proposed DGA constructs a directed graph over the objects in the image.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamic Graph Attention for Referring Expression Comprehension

Yang

Li²,

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

169

View full text Add to dashboard Cite

Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.

show abstract

Section: Referring Expression Comprehensionmentioning

confidence: 99%

Section: Referring Expression Comprehensionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Graph Attention for Referring Expression Comprehension

Yang

Li²,

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

169

View full text Add to dashboard Cite

show abstract

“…A common method is to first extract regions of interest from the image, using a region proposal network (RPN). Yu et al (2016); Mao et al (2016) decode these proposals as a caption using a recurrent neural network (RNN). The predicted region corresponds to the caption that is ranked most similar to the referring expression.…”

Section: Related Workmentioning

confidence: 99%

Talk2Car: Taking Control of Your Self-Driving Car

Deruyttere¹,

Vandenhende²,

Grujicic³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

A long-term goal of artificial intelligence is to have an agent execute commands communicated through natural language. In many cases the commands are grounded in a visual environment shared by the human who gives the command and the agent. Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving setting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands written in natural language for self-driving cars. We provide a detailed comparison with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-ofthe-art models. The results show that the proposed object referral task is a challenging one for which the models show promising results but still require additional research in natural language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http:// macchina-ai.eu/

show abstract

“…For testing, we use the training section of REFCOCO corpus collected by (Yu et al, 2016), which is based on the MSCOCO collection (Lin et al, 2014) containing over 300k images with object segmentations. This gives us a large enough test set to make stable predictions about the quality of individual word predictors, which often only have a few positive instances in the test set of the REFERIT corpus.…”

Section: Experimental Set-upmentioning

confidence: 99%

Is this a Child, a Girl or a Car? Exploring the Contribution of Distributional Similarity to Learning Referential Word Meanings

Zarrieß

Schlangen

2017

Proceedings of the 15th Conference of the European Chapter of The Association for Computational Linguistics: Volume 2

View full text Add to dashboard Cite

There has recently been a lot of work trying to use images of referents of words for improving vector space meaning representations derived from text. We investigate the opposite direction, as it were, trying to improve visual word predictors that identify objects in images, by exploiting distributional similarity information during training. We show that for certain words (such as entry-level nouns or hypernyms), we can indeed learn better referential word meanings by taking into account their semantic similarity to other words. For other words, there is no or even a detrimental effect, compared to a learning setup that presents even semantically related objects as negative instances.

show abstract

Modeling Context in Referring Expressions

Cited by 596 publications

References 34 publications

Dynamic Graph Attention for Referring Expression Comprehension

Dynamic Graph Attention for Referring Expression Comprehension

Talk2Car: Taking Control of Your Self-Driving Car

Is this a Child, a Girl or a Car? Exploring the Contribution of Distributional Similarity to Learning Referential Word Meanings

Contact Info

Product

Resources

About