Topic Modeling with Wasserstein Autoencoders

Abstract

We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.

Nested variational autoencoder for topic modelling on microtexts with word vectors

“…NVI: the VAE‐based approach introduced in Miao et al (2016). Was‐A: The Wasserstein Autoencoder model presented in Nan et al (2019). AVITM: This technique is published in Srivastava and Sutton (2017), using VAE to approximate the LDA process. AATM:The model uses the attention‐based autoencoder technique for topic modelling in (Tian & Fang, 2019).…”

“…This method is based on probability and word distribution, making it difficult to process short text. A similar approach is presented in Nan, Ding, Nallapati, and Xiang (2019), in which the deep neural network is used to model the Dirichlet process in LDA. Work reported in Zhu, Feng, and Li (2018) introduces an approach that is similar to the clustering process, when an enhanced graph‐based VAE is proposed to capture varied learned words from large corpus and apply it to topic modelling problem.…”

Neural Topic Modeling by Incorporating Document Relationship Graph

“…We use the preprocessed 20Newsgroups of (Srivastava and Sutton, 2017), and preprocessed Grolier and NYTimes of (Wang et al, 2019a). We compare the performance of our model with LDA (Blei et al, 2003), NVDM (Miao et al, 2016), ProdLDA (Srivastava and Sutton, 2017), GraphBTM (Zhu et al, 2018), ATM (Wang et al, 2019a) and W-LDA (Nan et al, 2019) using topic coherence measures (Röder et al, 2015). To quantify the understandability of the extracted topics, a topic coherence measure aggregates the relatedness scores of the topic words (topweighted words) of each topic, where the word relatedness scores are estimated based on word co-occurrence statistics on a large external corpus.…”

“…We impose a Dirichlet prior, the conjugate prior of the multinomial distribution, to the latent topic distributions. Following W-LDA (Nan et al, 2019), we achieve this goal by minimizing the Maximum Mean Discrepancy (MMD) (Gretton et al, 2012) between the distribution QẐ of inferred topic distributionsẑ and the Dirichlet prior P Z from which we draw multinomial noises z:…”

“…There are also attempts that directly enforced a Dirichlet prior on the document topics. W-LDA (Nan et al, 2019) models topics in the Wasserstein autoencoders (Tolstikhin et al, 2017) framework and achieves distribution matching by minimizing their Maximum Mean Discrepancy (MMD) (Gretton et al, 2012), while adversarial topic model (Wang et al, 2019a(Wang et al, ,b, 2020 directly generates documents from the Dirichlet prior and such a process is adversarially trained with a discriminator under the framework of Generative Adversarial Network (GAN) (Goodfellow et al, 2014).…”