Random Positive-Only Projections: PPMI-Enabled Incremental Semantic Space Construction

QasemiZadeh, Behrang; Kallmeyer, Laura

doi:10.18653/v1/s16-2024

Cited by 10 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For runs involving T RI, we experimented with a varying vector size from 200 to 1, 000. Moreover, we investigated (1) the initialization of the count matrix at time j with the matrix at time j − 1, (2) the contribution of positive-only projections, and (3) the application of PPMI weights, as explained in QasemiZadeh and Kallmeyer (2016). For DW 2V , we use the parameter setting proposed in Yao et al (2018).…”

Section: Methodsmentioning

confidence: 99%

GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering

Cassotti¹,

Caputo²,

Polignano³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper describes the system proposed for the SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. We focused our approach on the detection problem. Given the semantics of words captured by temporal word embeddings in different time periods, we investigate the use of unsupervised methods to detect when the target word has gained or loosed senses. To this end, we defined a new algorithm based on Gaussian Mixture Models to cluster the target similarities computed over the two periods. We compared the proposed approach with a number of similarity-based thresholds. We found that, although the performance of the detection methods varies across the word embedding algorithms, the combination of Gaussian Mixture with Temporal Referencing resulted in our best system.

show abstract

Section: Methodsmentioning

confidence: 99%

GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering

Cassotti¹,

Caputo²,

Polignano³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…I implemented the model using DyNet (Neubig et al, 2017) and Pydmrs . 6 I initialised the generative model following Emerson and Copestake (2017b) using sparse PPMI vectors (QasemiZadeh and Kallmeyer, 2016). I first trained the encoder on the initial generative model, then trained both together.…”

Section: Training Detailsmentioning

confidence: 99%

Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

Emerson¹

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Functional Distributional Semantics provides a linguistically interpretable framework for distributional semantics, by representing the meaning of a word as a function (a binary classifier), instead of a vector. However, the large number of latent variables means that inference is computationally expensive, and training a model is therefore slow to converge. In this paper, I introduce the Pixie Autoencoder, which augments the generative model of Functional Distributional Semantics with a graphconvolutional neural network to perform amortised variational inference. This allows the model to be trained more effectively, achieving better results on two tasks (semantic similarity in context and semantic composition), and outperforming BERT, a large pre-trained language model.

show abstract

“…Random Indexing (RI) is a simple and efficient method for dimensionality reduction (Sahlgren 2005), originally used to solve clustering problems (Kaski 1998). It is also a less-travelled technique in distributional semantics (Kanerva, Kristoferson, and Holst 2000;Qasemizadeh, Kallmeyer, and Herbelot 2017;QasemiZadeh and Kallmeyer 2016). Its advocates argue that it fulfils a number of requirements of an ideal vector space construction method, in particular incrementality.…”

Section: Lexical Acquisition and The Fruit Fly Algorithmmentioning

confidence: 99%

Biodiversity in NLP: modelling lexical meaning with the Fruit Fly Algorithm

Preissner¹,

Herbelot²

2020

ijcol

View full text Add to dashboard Cite

The natural world is very diverse in terms of biological organisation, and solves problems in a wide variety of efficient and creative manners. This biodiversity is in stark contrast with the landscape of artificial models in the field of Natural Language Processing (NLP). In the last years, NLP algorithms have clustered around a few very expensive architectures, the cost of which has many facets, including training times, storage, replicability, interpretability, equality of access to experimental paradigms, and even environmental impact. Inspired by the biodiversity of the real world, we argue for a methodology which promotes 'artificial diversity', and we further propose that cognitively-inspired algorithms are a good starting point to explore new architectures. As a case study, we investigate the fruit fly's olfactory system as a distributional semantics model. We show that, even in its rawest form, it provides many of the features that we might require from a good model of meaning acquisition, and that the original architecture can serve as a basis for cognitively-inspired extensions. We focus on one such extension by implementing a mechanism of neural adaptation.

show abstract

Random Positive-Only Projections: PPMI-Enabled Incremental Semantic Space Construction

Cited by 10 publications

References 28 publications

GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering

GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering

Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics

Biodiversity in NLP: modelling lexical meaning with the Fruit Fly Algorithm

Contact Info

Product

Resources

About