Learning Compressed Sentence Representations for On-Device Text Processing

Abstract

Vector representations of sentences, trained on massive text corpora, are widely used as generic sentence embeddings across a variety of NLP problems. The learned representations are generally assumed to be continuous and real-valued, giving rise to a large memory footprint and slow retrieval speed, which hinders their applicability to low-resource (memory and computation) platforms, such as mobile devices. In this paper, we propose four different strategies to transform continuous and generic sentence embeddings into a binarized form, while preserving their rich semantic information. The introduced methods are evaluated across a wide range of downstream tasks, where the binarized sentence embeddings are demonstrated to degrade performance by only about 2% relative to their continuous counterparts, while reducing the storage requirement by over 98%. Moreover, with the learned binary representations, the semantic relatedness of two sentences can be evaluated by simply calculating their Hamming distance, which is more computational efficient compared with the inner product operation between continuous embeddings. Detailed analysis and case study further validate the effectiveness of proposed methods.

Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings

“…Simultaneously, we find that our static embeddings substantially outperform Word2Vec and GloVe and therefore suggests our method serves the dual purpose of being a lightweight mechanism for generating static embeddings that track with advances in contextualized representations. Since static embeddings have significant advantages with respect to speed, computational resources, and ease of use, these results have important implications for resource-constrained settings (Shen et al, 2019), environmental concerns (Strubell et al, 2019), and the broader accessibility of NLP technologies. 2 Alongside more developed methods for embedding analysis, the static embedding setting is also equipped with a richer body of work regarding social bias.…”

Learning distributed sentence vectors with bi-directional 3D convolutions

“…Research works have been carried out to model the order of words when learning the distributed sentence representation (Le and Mikolov, 2014;Kiros et al, 2015;Conneau et al, 2017;Pagliardini et al, 2018;Gupta et al, 2019;Shen et al, 2019). Le and Mikolov propose Doc2vec (Le and Mikolov, 2014) to add a paragraph vector to represent the missing information from the current context.…”

“…Gupta et al (2019) propose two modifications of Word2vec by considering higher-order word n-grams along with uni-gram during training. Shen et al (2019) use InferSent (Conneau et al, 2017) for sentence embeddings based on word vectors learned by Glove (Pennington et al, 2014) or FastText (Joulin et al, 2017). Gupta et al (2019) claim that training word embeddings along with higher n-gram embeddings helps in the removal of the contextual information from the uni-gram, resulting in better stand-alone word embeddings.…”

General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference

“…Quantization and other compression techniques have been explored for word embeddings (Shu and Nakayama, 2017;Tissier et al, 2019) and sentence embeddings (Shen et al, 2019 Fan et al, 2020). Quantization is complementary to the approaches we consider and is explored more in Section 5.…”