Learning Geometric Word Meta-Embeddings

Jawanpuria, Pratik; Dev, N. T. V. Satya; Kunchukuttan, Anoop; Mishra, Bamdev

doi:10.18653/v1/2020.repl4nlp-1.6

Cited by 9 publications

(7 citation statements)

References 13 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With these alignment transformations on monolingual space, one can obtain a better cross-lingual integration of the vector spaces. Recent work has also found that applying orthogonal transformations to source embeddings facilitates averaging (García et al, 2020;Jawanpuria et al, 2020). We expand on this work by incorporating orthogonal transformations in word prisms, which learn word meta-embeddings for specific downstream tasks.…”

Section: Word Meta-embeddingsmentioning

confidence: 95%

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Tsiolis

Kenyon-Dean

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Word embeddings are trained to predict word cooccurrence statistics, which leads them to possess different lexical properties (syntactic, semantic, etc.) depending on the notion of context defined at training time. These properties manifest when querying the embedding space for the most similar vectors, and when used at the input layer of deep neural networks trained to solve downstream NLP problems. Meta-embeddings combine multiple sets of differently trained word embeddings, and have been shown to successfully improve intrinsic and extrinsic performance over equivalent models which use just one set of source embeddings. We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings according to the task at hand. Word prisms learn orthogonal transformations to linearly combine the input source embeddings, which allows them to be very efficient at inference time. We evaluate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements in performance on all tasks. 1 * Equal contribution. † This work was pursued prior to Kian's employment at BMO.

show abstract

Section: Word Meta-embeddingsmentioning

confidence: 95%

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Tsiolis

Kenyon-Dean

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Although averaging does not increase the dimensionality of the ME space as with concatenation, it does not consistently outperform concatenation, especially when the orthogonality condition does not hold. To overcome this problem, Jawanpuria et al [2020] proposed to first learn orthogonal projection matrices for each source embedding space. They measure the Mahalanobis metric between the projected source embeddings, which is a generalisation of the innerproduct that does not assume the dimensions in the vector space to be uncorrelated.…”

Section: Averagingmentioning

confidence: 99%

“…Given independently trained multiple word representations (aka embeddings) learnt using diverse algorithms and lexical resources, word meta-embedding (ME) learning methods [Yin and Schütze, 2016;Bao and Bollegala, 2018;Bollegala et al, 2018a;Wu et al, 2020;He et al, 2020;Jawanpuria et al, 2020;Coates and Bollegala, 2018] attempt to learn more accurate and wide-coverage word embeddings. The input and output word embeddings to the ME algorithm are referred respectively as the source and meta-embeddings.…”

Section: Introductionmentioning

confidence: 99%

“…Prior work [Yin and Shen, 2018;Levy et al, 2015] studying word embeddings have shown that the performance of a static word embedding is directly influenced by its dimensionality. ME learning methods use different techniques such as concatenation [Yin and Schütze, 2016], orthogonal projections [Jawanpuria et al, 2020;He et al, 2020] and averaging [Coates and Bollegala, 2018] after applying zero-padding to the sources with smaller dimensionalities as necessary to handle source embeddings with different dimensionalities.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Survey on Word Meta-Embedding Learning

Bollegala¹,

O’Neill²

2022

Preprint

View full text Add to dashboard Cite

Meta-embedding (ME) learning is an emerging approach that attempts to learn more accurate word embeddings given existing (source) word embeddings as the sole input. Due to their ability to incorporate semantics from multiple source embeddings in a compact manner with superior performance, ME learning has gained popularity among practitioners in NLP. To the best of our knowledge, there exist no prior systematic survey on ME learning and this paper attempts to fill this need. We classify ME learning methods according to multiple factors such as whether they (a) operate on static or contextualised embeddings, (b) trained in an unsupervised manner or (c) fine-tuned for a particular task/domain. Moreover, we discuss the limitations of existing ME learning methods and highlight potential future research directions.

show abstract

“…This requires minimal bilingual supervision while still leveraging large amounts of monolingual corpora with very competitive results (Artetxe et al, 2016(Artetxe et al, , 2018. These techniques are used by Doval et al (2018); García-Ferrero et al (2020); Jawanpuria et al (2020); He et al (2020) to generate meta-embeddings. This usually involves mapping all the source embeddings to a common vector space followed by averaging.…”

Section: Papermentioning

confidence: 99%

Benchmarking Meta-embeddings: What Works and What Does Not

García-Ferrero¹,

Agerri²,

Rigau³

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

In the last few years, several methods have been proposed to build meta-embeddings. The general aim was to obtain new representations integrating complementary knowledge from different source pre-trained embeddings thereby improving their overall quality. However, previous meta-embeddings have been evaluated using a variety of methods and datasets, which makes it difficult to draw meaningful conclusions regarding the merits of each approach. In this paper we propose a unified common framework, including both intrinsic and extrinsic tasks, for a fair and objective meta-embeddings evaluation. Furthermore, we present a new method to generate meta-embeddings, outperforming previous work on a large number of intrinsic evaluation benchmarks. Our evaluation framework also allows us to conclude that previous extrinsic evaluations of meta-embeddings have been overestimated.

show abstract

Learning Geometric Word Meta-Embeddings

Cited by 9 publications

References 13 publications

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

A Survey on Word Meta-Embedding Learning

Benchmarking Meta-embeddings: What Works and What Does Not

Contact Info

Product

Resources

About