On the Downstream Performance of Compressed Word Embeddings

Abstract

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging-existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression.We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to 2× lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.

word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement

“…Dictionary learning [20] and word embedding clustering [21] approaches have been proposed. An optimized method for uniform quantization of floating point numbers in the embedding matrix has been proposed recently [22]. To compress a model for low-memory inference, [23] used pruning and quantization for lowering the number of parameters.…”

On Geodesic Distances and Contextual Embedding Compression for Text Classification

“…Although these models have been shown to achieve state-of-the-art performance on most NLP tasks, they are notably expensive to train. To help combat this, as mentioned by May et al (2019), model compression techniques like data quantization (Gong et al, 2014), model pruning (Han et al, 2016), and * Equal contribution knowledge distillation (Sanh et al, 2019, Hinton et al, 2015 have been developed. However, at 768 dimensions, the embeddings themselves can be prohibitively large for some tasks and settings.…”