Leveraging Past References for Robust Language Grounding

Abstract

Grounding referring expressions to objects in an environment has traditionally been considered a one-off, ahistorical task. However, in realistic applications of grounding, multiple users will repeatedly refer to the same set of objects. As a result, past referring expressions for objects can provide strong signals for grounding subsequent referring expressions. We therefore reframe the grounding problem from the perspective of coreference detection and propose a neural network that detects when two expressions are referring to the same object. The network combines information from vision and past referring expressions to resolve which object is being referred to. Our experiments show that detecting referring expression coreference is an effective way to ground objects described by subtle visual properties, which standard visual grounding models have difficulty capturing. We also show the ability to detect object coreference allows the grounding model to perform well even when it encounters object categories not seen in the training data. * Equal contribution. Utterances from a User: (1) The open text book. (2) Page with a dark border and images of two people at the bottom. (3) The open book with typed pages. (4) Pages of handwritten notes under the open book.

Continual Adaptation for Efficient Machine Communication

“…Incorporating discourse history. Previous work has incorporated discourse history in reference games using explicit co-reference detection (Roy et al, 2019) or contribution tracking (DeVault and Stone, 2009) techniques. An alternative approach is to include embeddings of the history as conditional input to the model at test time (Haber et al, 2019).…”

Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contexts

“…In contrast, we generate the surface realisation of first and subsequent referring utterances end-to-end, grounding them in continuous visual features of real images. Our work is related to a recent line of research on reference resolution in visually-grounded dialogue, where previous mentions have been shown to be useful (Shore and Skantze, 2018;Haber et al, 2019;Roy et al, 2019). Here we focus on generation.…”

Grounding ‘Grounding’ in NLP

“…• Non-textual modality: Fusion is applied with images for tasks like referring expressions (Roy et al, 2019), SRL etc., For videos, some tasks are grounding action descriptions (Regneri et al, 2013), spatio-temporal QA (Lei et al, 2020), concept similarity , mapping events (Fleischman and Roy, 2008) etc.,…”

“…• Non-textual Modality: Multitasking with images is used to perform spoken image captioning (Chrupala, 2019) and grammar induction (Zhao and Titov, 2020). Joint modeling was used in multiresolution language grounding Koncel-Kedziorski et al (2014), identifying referring expressions Roy et al (2019), multimodal MT (Zhou et al, 2018c), video parsing Ross et al (2018), learning latent semantic annotations (Qin et al, 2018) etc.,…”