Neural Topic Model via Optimal Transport

Zhao, He; Phung, Dinh; Huynh, Viet; Le, Trung; Buntine, Wray L.

doi:10.48550/arxiv.2008.13537

Cited by 3 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AG News has the 4 largest classes ("World", "Sports", "Business", "Sci/Tech") of AG's Corpus. The AG News contains We compare with the state-of-the-art NTMs, including: ProdLDA [7], Dirichlet VAE (DVAE) [27], Embedding Topic Model (ETM) [28], Wasserstein LDA (WLDA) [29] and NSTM [3].…”

Section: Datasets and Baseline Methodsmentioning

confidence: 99%

“…For NTM via OT (NSTM) [3], an encoder is leveraged to generate topic z from normalized word vector x by z = sof tmax(θ( x)). Since x and z are two distributions with different support for the same document, in order to learn the encoder, the OT distance is minimized to push z towards x, as min θ d M ( x, z).…”

Section: Neural Topic Models and Optimal Transportmentioning

confidence: 99%

“…The main goal of NTM is to achieve low reconstruction error, but it cares less about the interpretability of topic representations, specifically, whether the mapping between topics and text is clearly organized or the relationships that exist between texts are preserved in the subject space. Recently, an NTM based on the optimal transport (OT) framework proposed in [3] seeks to strike a better balance between obtaining good document representation and generating coherent/diverse topics. The model utilizes an encoder that outputs the topic distribution of the document by taking its word count vector as input like a standard NTM, but minimizes the OT distance between word count vector and topic distributions, which are two discrete distributions of the support for words and topics.…”

Section: Introductionmentioning

confidence: 99%

“…For topic models, such geometry is particularly important, i.e., whether the two documents that are similar on the word space are still similar in the topic space. In [3], even OT tries to find the shortest transfer path, adding OT to NTM can make it not transfer the associated elements too far, but it does not have an intuitive effect. The main idea of manifold learning [4], [5] is to map high-dimensional data to lowdimensional data, so that the low-dimensional data can reflect some essential structural features of the original high-dimensional data.…”

Section: Introductionmentioning

confidence: 99%

“…The topic distribution z is obtained from the encoder, and z can be defined as a low-dimensional vector. In NSTM [3], the learning process of the topic model is the process of the distribution of z approaching the distribution of x. In this paper, while retaining the document reconstruction ability of VAE and the distribution approximation ability of OT, we will emphasize the structure preservation ability of topic distribution to document collections.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Neural Topic Model with Distance Awareness

Chen

Zhao

Huynh

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural topic models (NTM) have shown their success in topic modelling with awide range of applications in text analysis. NTMs based on generative modelstend to have good refactoring capabilities in achieving good document represen-tations, rather than providing a topic representation that has an understandableassociation with documents and words. To bridge this gap, we have proposeda neural topic model that allows simultaneously learning topics from text dataas well as capturing the structural manifold in the original data space of doc-uments. Comprehensive experiments have shown that the proposed model canimprove learned topics in terms of intrinsic metrics (purity and coherence)and extrinsic downstream tasks (clustering, classification). Beyond that, themodel can provide interpretable low dimensional presentations of documents

show abstract

Section: Datasets and Baseline Methodsmentioning

confidence: 99%

Section: Neural Topic Models and Optimal Transportmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Neural Topic Model with Distance Awareness

Chen

Zhao

Huynh

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Panwar¹,

Shailabh²,

Aggarwal³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Topic models have been widely used to learn text representations and gain insight into document corpora. To perform topic discovery, most existing neural models either take document bag-of-words (BoW) or sequence of tokens as input followed by variational inference and BoW reconstruction to learn topic-word distribution. However, leveraging topic-word distribution for learning better features during document encoding has not been explored much. To this end, we develop a framework TAN-NTM, which processes document as a sequence of tokens through a LSTM whose contextual outputs are attended in a topic-aware manner. We propose a novel attention mechanism which factors in topic-word distribution to enable the model to attend on relevant words that convey topic related cues. The output of topic attention module is then used to carry out variational inference. We perform extensive ablations and experiments resulting in ∼ 9 -15 percentage improvement over score of existing SOTA topic models in NPMI coherence on several benchmark datasets -20Newsgroups, Yelp Review Polarity and AGNews. Further, we show that our method learns better latent document-topic features compared to existing topic models through improvement on two downstream tasks: document classification and topic guided keyphrase generation.

show abstract

Neural Attention-Aware Hierarchical Topic Model

Jin

Zhao

Liu

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural topic models (NTMs) apply deep neural networks to topic modelling. Despite their success, NTMs generally ignore two important aspects: (1) only document-level word count information is utilized for the training, while more fine-grained sentence-level information is ignored, and (2) external semantic knowledge regarding documents, sentences and words are not exploited for the training. To address these issues, we propose a variational autoencoder (VAE) NTM model that jointly reconstructs the sentence and document word counts using combinations of bag-of-words (BoW) topical embeddings and pre-trained semantic embeddings. The pre-trained embeddings are first transformed into a common latent topical space to align their semantics with the BoW embeddings. Our model also features hierarchical KL divergence to leverage embeddings of each document to regularize those of their sentences, thereby paying more attention to semantically relevant sentences. Both quantitative and qualitative experiments have shown the efficacy of our model in 1) lowering the reconstruction errors at both the sentence and document levels, and 2) discovering more coherent topics from real-world datasets.

show abstract

Neural Topic Model via Optimal Transport

Cited by 3 publications

References 18 publications

Neural Topic Model with Distance Awareness

Neural Topic Model with Distance Awareness

TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Neural Attention-Aware Hierarchical Topic Model

Contact Info

Product

Resources

About