Embedding Compression with Isotropic Iterative Quantization

Liao, Siyu; Chen, Jie; Wang, Yanzhi; Qiu, Qinru; Yuan, Bo

doi:10.1609/aaai.v34i05.6350

Cited by 3 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One could examine the approach by Kim et al (2020), where they learn NLP task-specific features and compress embeddings accordingly. A different approach by Liao et al (2020) used quantization and dimensionality reduction to optimize the encoding of word embeddings, while preserving relevant information.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models

Jacobsen

Sørensen

Derczynski

2021

Preprint

View full text Add to dashboard Cite

Improvement in machine learning-based NLP performance are often presented with bigger models and more complex code. This presents a trade-off: better scores come at the cost of larger tools; bigger models tend to require more during training and inference time. We present multiple methods for measuring the size of a model, and for comparing this with the model's performance.In a case study over part-of-speech tagging, we then apply these techniques to taggers for eight languages and present a novel analysis identifying which taggers are size-performance optimal. Results indicate that some classical taggers place on the size-performance skyline across languages. Further, although the deep models have highest performance for multiple scores, it is often not the most complex of these that reach peak performance.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Word embeddings are an integral part in deep learning PoS taggers, but large. Several works have explored embedding compression (Chen et al, 2016;Tissier et al, 2019;Kim et al, 2020;Melas-Kyriazi et al, 2020;Liao et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models

Jacobsen

Sørensen

Derczynski

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…An isotropic iterative quantization (IIQ) method is suggested for compacting embedding feature vectors into binary ones to satisfy the required isotropic property of pointwise mutual information (PMI)-based approaches. This approach uses the iterative quantization technique, which is well-established for image retrieval (Liao et al 2020). A method for obtaining vector representations of noun phrases is suggested.…”

Section: Importance Of Word Embeddingmentioning

confidence: 99%

Impact of word embedding models on text analytics in deep learning environment: a review

2023

View full text Add to dashboard Cite

The selection of word embedding and deep learning models for better outcomes is vital. Word embeddings are an n-dimensional distributed representation of a text that attempts to capture the meanings of the words. Deep learning models utilize multiple computing layers to learn hierarchical representations of data. The word embedding technique represented by deep learning has received much attention. It is used in various natural language processing (NLP) applications, such as text classification, sentiment analysis, named entity recognition, topic modeling, etc. This paper reviews the representative methods of the most prominent word embedding and deep learning models. It presents an overview of recent research trends in NLP and a detailed understanding of how to use these models to achieve efficient results on text analytics tasks. The review summarizes, contrasts, and compares numerous word embedding and deep learning models and includes a list of prominent datasets, tools, APIs, and popular publications. A reference for selecting a suitable word embedding and deep learning approach is presented based on a comparative analysis of different techniques to perform text analytics tasks. This paper can serve as a quick reference for learning the basics, benefits, and challenges of various word representation approaches and deep learning models, with their application to text analytics and a future outlook on research. It can be concluded from the findings of this study that domain-specific word embedding and the long short term memory model can be employed to improve overall text analytics task performance.

show abstract