What a neural language model tells us about spatial relations

Abstract

Understanding and generating spatial descriptions requires knowledge about what objects are related, their functional interactions, and where the objects are geometrically located. Different spatial relations have different functional and geometric bias. The wide usage of neural language models in different areas including generation of image description motivates the study of what kind of knowledge is encoded in neural language models about individual spatial relations. With the premise that the functional bias of relations is expressed in their word distributions, we construct multi-word distributional vector representations and show that these representations perform well on intrinsic semantic reasoning tasks, thus confirming our premise. A comparison of our vector representations to human semantic judgments indicates that different bias (functional or geometric) is captured in different data collection tasks which suggests that the contribution of the two meaning modalities is dynamic, related to the context of the task.

From “Before” to “After”: Generating Natural Language Instructions from Image Pairs in a Simple Visual Domain

“…Generation of spatial expressions has seen some attention in recent years, e.g. by Ghanimifard and Dobnik (2019a), who investigate the spatial language that neural language models can learn and express.…”