A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

Abstract

We propose a novel generative model to explore both local and global context for joint learning topics and topic-specific word embeddings. In particular, we assume that global latent topics are shared across documents, a word is generated by a hidden semantic vector encoding its contextual semantic meaning, and its context words are generated conditional on both the hidden semantic vector and global latent topics. Topics are trained jointly with the word embeddings. The trained model maps words to topic-dependent embeddings, which naturally addresses the issue of word polysemy. Experimental results show that the proposed model outperforms the word-level embedding methods in both word similarity evaluation and word sense disambiguation. Furthermore, the model also extracts more coherent topics compared with existing neural topic models or other models for joint learning of topics and word embeddings. Finally, the model can be easily integrated with existing deep contextualized word embedding learning methods to further improve the performance of downstream tasks such as sentiment classification.

Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

“…Thompson and Mimno (2020) showed that clustering the contextual representations of a given set of words can produce clusters of semantically related words, which were found to be similar in spirit to LDA topics. The idea of learning topic-specific representations of words has been extensively studied in the context of standard word embeddings (Liu et al, 2015;Li et al, 2016;Shi et al, 2017;Zhu et al, 2020). To the best of our knowledge, learning topic-specific word representations using CLMs has not yet been studied.…”

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

“…Some very recent works use pre-trained BERT (Devlin et al, 2019) either to leverage improved text representations (Bianchi et al, 2020;Sia et al, 2020) or to augment topic model through knowledge distillation (Hoyle et al, 2020a). Zhu et al (2020) and Dieng et al (2020) jointly train words and topics in a shared embedding space. However, we train topic-word distribution as part of our model, embed it using word embeddings being learned and use resultant topic embeddings to perform attention over sequentially processed tokens.…”