Learning Efficient Task-Specific Meta-Embeddings with Word Prisms

Abstract

Word embeddings are trained to predict word cooccurrence statistics, which leads them to possess different lexical properties (syntactic, semantic, etc.) depending on the notion of context defined at training time. These properties manifest when querying the embedding space for the most similar vectors, and when used at the input layer of deep neural networks trained to solve downstream NLP problems. Meta-embeddings combine multiple sets of differently trained word embeddings, and have been shown to successfully improve intrinsic and extrinsic performance over equivalent models which use just one set of source embeddings. We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings according to the task at hand. Word prisms learn orthogonal transformations to linearly combine the input source embeddings, which allows them to be very efficient at inference time. We evaluate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements in performance on all tasks. 1 * Equal contribution. † This work was pursued prior to Kian's employment at BMO.

Benchmarking Meta-embeddings: What Works and What Does Not

“…Intrinsic Tasks Extrinsic Tasks (Kiela et al, 2018) SST, SNLI, Image Caption (1) (He et al, 2020) SST2, SNLI, NER (1), POS(1), Semcor (Bollegala and Bao, 2018) Sim. (4), An.…”

“…This requires minimal bilingual supervision while still leveraging large amounts of monolingual corpora with very competitive results (Artetxe et al, 2016(Artetxe et al, , 2018. These techniques are used by Doval et al (2018); García-Ferrero et al (2020); Jawanpuria et al (2020); He et al (2020) to generate meta-embeddings. This usually involves mapping all the source embeddings to a common vector space followed by averaging.…”

“…Another recent research line tries to dynamically generate meta-embeddings for specific tasks (He et al, 2020;Kiela et al, 2018;O'Neill and Bollegala, 2020). These methods extend already existing algorithms to generate meta-embeddings by learning task specific weights.…”

A Survey on Word Meta-Embedding Learning

“…Given independently trained multiple word representations (aka embeddings) learnt using diverse algorithms and lexical resources, word meta-embedding (ME) learning methods [Yin and Schütze, 2016;Bao and Bollegala, 2018;Bollegala et al, 2018a;Wu et al, 2020;He et al, 2020;Jawanpuria et al, 2020;Coates and Bollegala, 2018] attempt to learn more accurate and wide-coverage word embeddings. The input and output word embeddings to the ME algorithm are referred respectively as the source and meta-embeddings.…”

“…Prior work [Yin and Shen, 2018;Levy et al, 2015] studying word embeddings have shown that the performance of a static word embedding is directly influenced by its dimensionality. ME learning methods use different techniques such as concatenation [Yin and Schütze, 2016], orthogonal projections [Jawanpuria et al, 2020;He et al, 2020] and averaging [Coates and Bollegala, 2018] after applying zero-padding to the sources with smaller dimensionalities as necessary to handle source embeddings with different dimensionalities.…”