Coherence-Aware Neural Topic Modeling

Ding, Ran; Nallapati, Ramesh; Xiang, Bing

doi:10.18653/v1/d18-1096

Cited by 56 publications

(47 citation statements)

References 11 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments in Ding et al (2018) provide some insight into this behaviour. They find that when training neural topic models, model fit and NPMI initially tend to improve on each epoch.…”

Section: Npmi Versus Nll As Stopping Criteriamentioning

confidence: 95%

Variational Pretraining for Semi-supervised Text Classification

Gururangan¹,

Dang²,

Card³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

We introduce VAMPIRE, 1 a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to indomain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.

show abstract

Section: Npmi Versus Nll As Stopping Criteriamentioning

confidence: 95%

Variational Pretraining for Semi-supervised Text Classification

Gururangan¹,

Dang²,

Card³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Since W-LDA is not based on variational inference, we cannot compute the ELBO based perplexity as a performance metric as in (Miao et al, 2016;Srivastava and Sutton, 2017;Ding et al, 2018). To compare the predictive performance of the latent document-topic vectors across all models, we use document classification accuracy instead.…”

Section: Document Classificationmentioning

confidence: 99%

Topic Modeling with Wasserstein Autoencoders

Feng¹,

Ding²,

Nallapati³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.

show abstract

“…We select three experimental baseline models that represent diverse styles of neural topic modeling. 12 Each achieves the highest NPMI on the majority of its respective datasets, as well as a considerable improvement over previous neural and non-neural topic models (such as Srivastava and Sutton, 2017;Miao et al, 2016;Ding et al, 2018). All our baselines are roughly contemporaneous with one another, and had yet to be compared in a head-to-head fashion prior to our work.…”

Section: Experimental Baselinesmentioning

confidence: 83%

Improving Neural Topic Models using Knowledge Distillation

Alexander¹,

Goel²,

Resnik³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Topic models are often used to identify humaninterpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics. * Equal contribution. BoW BAT art chess gingerbread modernism painter picasso θ d • B Base neural topic model d Marcel Duchamp was a painter, sculptor, chess player, and writer whose work is associated with Cubism, Dada, and conceptual art. 7 qwone.com/˜jason/20Newsgroups 8 s3.amazonaws.com/research.metamind. io/wikitext/wikitext-103-v1.zip 9 ai.stanford.edu/˜amaas/data/sentiment

show abstract

Coherence-Aware Neural Topic Modeling

Cited by 56 publications

References 11 publications

Variational Pretraining for Semi-supervised Text Classification

Variational Pretraining for Semi-supervised Text Classification

Topic Modeling with Wasserstein Autoencoders

Improving Neural Topic Models using Knowledge Distillation

Contact Info

Product

Resources

About