Representing Syntax and Composition with Geometric Transformations

Abstract

The exploitation of syntactic graphs (SyGs) as a word's context has been shown to be beneficial for distributional semantic models (DSMs), both at the level of individual word representations and in deriving phrasal representations via composition. However, notwithstanding the potential performance benefit, the syntactically-aware DSMs proposed to date have huge numbers of parameters (compared to conventional DSMs) and suffer from data sparsity. Furthermore, the encoding of the SyG links (i.e., the syntactic relations) has been largely limited to linear maps. The knowledge graphs' literature, on the other hand, has proposed light-weight models employing different geometric transformations (GTs) to encode edges in a knowledge graph (KG). Our work explores the possibility of adopting this family of models to encode SyGs. Furthermore, we investigate which GT better encodes syntactic relations, so that these representations can be used to enhance phrase-level composition via syntactic contextualisation.

Using dependency parsing for few-shot learning in distributional semantics

“…To allow Skip-Gram to use only one vector per word,Zobnin and Elistratova (2019) propose using an indefinite inner product, which corresponds to T in (9) being a diagonal matrix of 1s and −1s. In a similar vein,Bertolini et al (2021) propose a more radical simplification of the Dependency Matrix model, which uses matrices that are non-zero only on the diagonal and off-diagonal.…”