A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

Abstract

Recent models achieve promising results in visually grounded dialogues. However, existing datasets often contain undesirable biases and lack sophisticated linguistic analyses, which make it difficult to understand how well current models recognize their precise linguistic structures. To address this problem, we make two design choices: first, we focus on OneCommon Corpus (Udagawa and Aizawa, 2019, 2020), a simple yet challenging common grounding dataset which contains minimal bias by design. Second, we analyze their linguistic structures based on spatial expressions and provide comprehensive and reliable annotation for 600 dialogues. We show that our annotation captures important linguistic structures including predicate-argument structure, modification and ellipsis. In our experiments, we assess the model's understanding of these structures through reference resolution. We demonstrate that our annotation can reveal both the strengths and weaknesses of baseline models in essential levels of detail. Overall, we propose a novel framework and resource for investigating fine-grained language understanding in visually grounded dialogues.

SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning

“…Spatial reasoning is a cognitive process based on the construction of mental representations for spatial objects, relations, and transformations (Clements and Battista, 1992), which is necessary for many natural language understanding (NLU) tasks such as natural language navigation Roman Roman et al, 2020;Kim et al, 2020), human-machine interaction (Landsiedel et al, 2017;Roman Roman et al, 2020), dialogue systems (Udagawa et al, 2020), and clinical analysis (Datta and Roberts, 2020).…”

Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches

“…Referring expressions (e.g., the red one) often can only be resolved in a visual context, and deictic expressions, like English here, there, this and that, are frequently used in language to individuate referents in their immediate context, relying on mutual knowledge of what the speaker and listener can see (Clark and Marshall, 1981). Reference intepretation can also be affected by the location of the speaker and hearer in the world (Birner, 2012), and can involve physical analogues of implicature (e.g., the black one might be a good description for a dark grey object if all other visible objects are lighter) (Golland et al, 2010;Udagawa et al, 2020).…”

A Meta-framework for Spatiotemporal Quantity Extraction from Text

“…Existing works on spatial semantics have focused on natural language navigation (Chen et al, 2019;Kim et al, 2020), human-machine interaction (Landsiedel et al, 2017;Roman Roman et al, 2020), dialogue systems (Udagawa et al, 2020), and clinical analysis (Kordjamshidi et al, 2015;Datta and Roberts, 2020). Works on geocoding (Gritta et al, 2018;Kulkarni et al, 2020) map spatial mentions to coordinates, which can be applied to our work for finer geolocation mapping.…”