A Framework for Distributional Formal Semantics

Abstract

Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. More specifically, proposals have been put forward to augment formal semantic machinery with distributional meaning representations, thereby introducing the notion of semantic similarity into formal semantics, or to define distributional systems that aim to incorporate formal notions such as entailment and compositionality. However, given the fundamentally different 'representational currency' underlying formal and distributional approaches-models of the world versus linguistic co-occurrence-their unification has proven extremely difficult.Here, we define a Distributional Formal Semantics that integrates distributionality into a formal semantic system on the level of formal models. This approach offers probabilistic, distributed meaning representations that are also inherently compositional, and that naturally capture fundamental semantic notions such as quantification and entailment. Furthermore, we show how the probabilistic nature of these representations allows for probabilistic inference, and how the information-theoretic notion of "information" (measured in terms of Entropy and Surprisal) naturally follows from it. Finally, we illustrate how meaning representations can be derived incrementally from linguistic input using a recurrent neural network model, and how the resultant incremental semantic construction procedure intuitively captures key semantic phenomena, including negation, presupposition, and anaphoricity.

Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model

“…The utterance meaning representations that the model produces—at its integration_output layer—are rich “situation model”-like meaning representations that encode meaning as points in a Distributed Situation-state Space (DSS; Frank et al, 2003 , 2009 ; for a recent reconceptualization of these representations grounded in formal semantics, see Venhuizen et al, 2019c ). DSS offers distributed representations that allow for encoding world knowledge, and that are both compositional and probabilistic (see section 2.2.3 below for more detail).…”

“…Following Venhuizen et al ( 2019a ), the semantics associated with the training sentences presented to the model are derived from the Distributed Situation-state Space model (DSS, Frank et al, 2003 , 2009 ; see also the formalization in terms of Distributional Formal Semantics described in Venhuizen et al, 2019c ). In DSS, utterance meaning vectors are derived from a meaning space that defines co-occurrences of individual propositional meanings across a set of observations (formalized as formal semantic models in Venhuizen et al, 2019c ). For the current meaning space, a set of propositions was generated using the predicates enter(p,l), leave(p,l) , and go_to(p,g) , in combination with arguments that identify a person ( p ), location ( l ), and goal ( g ) (see Table 1 ).…”

Semantic Entropy in Language Comprehension

“…These representations, which are based on the Distributed Situation-state Space framework [ 36 , 37 ], are argued to instantiate situation models that allow for world knowledge-driven inference. Following [ 35 ], we here reconceptualize this approach in terms of model-theoretic semantics, thereby emphasizing the generalizability of the framework.…”

“…However, in order to empirically evaluate how the information-theoretic notion of entropy (reduction) is affected by the structure of the world, the co-occurrence between propositions needs to be defined in a controlled manner. Therefore, the meaning representations used here (following VCB [ 33 ]) are induced from a high-level description of the structure of the world, using an incremental, inference-driven construction procedure [ 35 ].…”

“…In what follows, we first introduce the probabilistic, distributed meaning representations used by VCB [ 33 ], from a novel, formal semantic perspective (cf. [ 35 ]) ( Section 2.1 ). Next, we describe the comprehension model ( Section 2.2.1 ) as well as how processing in this model gives rise to a comprehension-centric notion of surprisal ( Section 2.2.2 ).…”