Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

Emerson, Guy

doi:10.18653/v1/2020.acl-main.367

“…My own recent work in this direction has been to develop the Pixie Autoencoder (Emerson, 2020a), and I look forward to seeing alternative approaches from other authors, as the field of distributional semantics continues to grow. I hope that this survey paper will help other researchers to develop the field in a way that keeps long-term goals in mind.…”

Section: Discussionmentioning

confidence: 99%

“…In my own work, I have learnt classifiers Copestake, 2016, 2017a,b), but with a computationally expensive model that is difficult to train. The computational challenge is partially resolved in my most recent work (Emerson, 2020a), but there is still work to be done in scaling up the model to make full use of the corpus data. The best way to design such a model, so that it can both make full use of the data and can be trained efficiently, is an open question.…”

Section: Concepts and Referentsmentioning

confidence: 99%

“…For example, in my own work I have represented words as truth-conditional functions that are compatible with first-order logic (Emerson and Copestake, 2017b;Emerson, 2020b). Since referents are not observed in distributional semantics, this introduces latent variables that make the model computationally expensive, although there are ways to mitigate this (Emerson, 2020a).…”

Section: Logicmentioning

confidence: 99%

See 1 more Smart Citation

What are the Goals of Distributional Semantics?

Emerson

¹

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Distributional semantic models have become a mainstay in NLP, providing useful features for downstream tasks. However, assessing long-term progress requires explicit long-term goals. In this paper, I take a broad linguistic perspective, looking at how well current models can deal with various semantic challenges. Given stark differences between models proposed in different subfields, a broad perspective is needed to see how we could integrate them. I conclude that, while linguistic insights can guide the design of model architectures, future progress will require balancing the often conflicting demands of linguistic expressiveness and computational tractability.

show abstract

“…Alongside count-based models, a variety of neural ones have been proposed to encode syntactic structure, focusing on different depths of the graph (Levy and Goldberg, 2014;Komninos and Manandhar, 2016;Marcheggiani and Titov, 2017;Vashishth et al, 2019;Emerson, 2020)). Of particular note here, Levy and Goldberg (2014) and Komninos and Manandhar (2016) each proposed models (DEP and EXT, respectively) which learn from local dependency relations, by extending the Skip-Gram with Negative sampling (SGNS) architecture from word2vec (Mikolov et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

Representing Syntax and Composition with Geometric Transformations

Bertolini

¹

,

Weeds

²

,

Weir

³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

The exploitation of syntactic graphs (SyGs) as a word's context has been shown to be beneficial for distributional semantic models (DSMs), both at the level of individual word representations and in deriving phrasal representations via composition. However, notwithstanding the potential performance benefit, the syntactically-aware DSMs proposed to date have huge numbers of parameters (compared to conventional DSMs) and suffer from data sparsity. Furthermore, the encoding of the SyG links (i.e., the syntactic relations) has been largely limited to linear maps. The knowledge graphs' literature, on the other hand, has proposed light-weight models employing different geometric transformations (GTs) to encode edges in a knowledge graph (KG). Our work explores the possibility of adopting this family of models to encode SyGs. Furthermore, we investigate which GT better encodes syntactic relations, so that these representations can be used to enhance phrase-level composition via syntactic contextualisation.

show abstract

“…Alongside count-based models, a variety of neural ones have been proposed to encode syntactic structure, focusing on different depths of the graph (Levy and Goldberg, 2014;Komninos and Manandhar, 2016;Marcheggiani and Titov, 2017;Vashishth et al, 2019;Emerson, 2020)). Of particular note here, Levy and Goldberg (2014) and Komninos and Manandhar (2016) each proposed models (DEP and EXT, respectively) which learn from local dependency relations, by extending the Skip-Gram with Negative sampling (SGNS) architecture from word2vec (Mikolov et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

Representing Syntax and Composition with Geometric Transformations

Bertolini

¹

,

Weeds

²

,

Weir

³

et al. 2021

Preprint

0

View full text Add to dashboard Cite

The exploitation of syntactic graphs (SyGs) as a word's context has been shown to be beneficial for distributional semantic models (DSMs), both at the level of individual word representations and in deriving phrasal representations via composition. However, notwithstanding the potential performance benefit, the syntactically-aware DSMs proposed to date have huge numbers of parameters (compared to conventional DSMs) and suffer from data sparsity. Furthermore, the encoding of the SyG links (i.e., the syntactic relations) has been largely limited to linear maps. The knowledge graphs' literature, on the other hand, has proposed light-weight models employing different geometric transformations (GTs) to encode edges in a knowledge graph (KG). Our work explores the possibility of adopting this family of models to encode SyGs. Furthermore, we investigate which GT better encodes syntactic relations, so that these representations can be used to enhance phrase-level composition via syntactic contextualisation.1 dobj indicating the inverse of the dobj relation

show abstract

Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

Cited by 6 publications

References 59 publications

What are the Goals of Distributional Semantics?

What are the Goals of Distributional Semantics?

Representing Syntax and Composition with Geometric Transformations

Representing Syntax and Composition with Geometric Transformations

Contact Info

Product

Resources

About