Visually Guided Spatial Relation Extraction from Text

Abstract

Extraction of spatial relations from sentences with complex/nesting relationships is very challenging as often needs resolving inherent semantic ambiguities. We seek help from visual modality to fill the information gap in the text modality and resolve spatial semantic ambiguities. We use various recent vision and language datasets and techniques to train inter-modality alignment models, visual relationship classifiers and propose a novel global inference model to integrate these components into our structured output prediction model for spatial role and relation extraction. Our global inference model enables us to utilize the visual and geometric relationships between objects and improves the state-of-art results of spatial information extraction from text.

Anaphora Resolution for Improving Spatial Relation Extraction from Text

“…The code is publicly available here 2 . We compare our approach with the state-of-theart (Rahgooy et al, 2018). However, in the mentioned paper, the authors use visual data from the accompanying images to improve the models.…”

Representation, Learning and Reasoning on Spatial Language for Downstream NLP Tasks

“…Spatial semantics is very closely connected and relevant to visualization of natural language and grounding language into perception, central to dealing with configurations in the physical world and motivating a combination of vision and language for richer spatial understanding. The related tasks include: text-to-scene conversion; image captioning; spatial and visual question answering; and spatial understanding in multimodal settings (Rahgooy et al, 2018) for robotics and navigation tasks and language grounding (Thomason et al, 2018).…”

Proceedings of the First International Workshop on Spatial Language Understanding

“…This function is a linear discriminant function defined over combined feature representation of inputs and outputs denoted by f (x, y). However, in this work, independent classifiers are trained per role and relations and only the predication is performed based on the global inference as in (Kordjamshidi et al, 2017a;Rahgooy et al, 2018) .…”

“…The global constraints used in our proposed model is combination of previously proposed constraints (1-7) (Rahgooy et al, 2018) and new one (constraint 8) described in Table 3.3. In fact, the global inference is performed using integer linear programming techniques subject to these constraints.…”

“…This is a learning only model where each classifier makes independent predictions. This model doesn't use any constraints, and is compared with similar (Rahgooy et al, 2018) baseline model in section 4.…”