Neural Topic Modeling with Deep Mutual Information Estimation

Abstract

The emerging neural topic models make topic modeling more easily adaptable and extendable in unsupervised text mining. However, the existing neural topic models is difficult to retain representative information of the documents within the learnt topic representation. In this paper, we propose a neural topic model which incorporates deep mutual information estimation, i.e., Neural Topic Modeling with Deep Mutual Information Estimation(NTM-DMIE). NTM-DMIE is a neural network method for topic learning which maximizes the mutual information between the input documents and their latent topic representation. To learn robust topic representation, we incorporate the discriminator to discriminate negative examples and positive examples via adversarial learning. Moreover, we use both global and local mutual information to preserve the rich information of the input documents in the topic representation. We evaluate NTM-DMIE on several metrics, including accuracy of text clustering, with topic representation, topic uniqueness and topic coherence. Compared to the existing methods, the experimental results show that NTM-DMIE can outperform in all the metrics on the four datasets.

InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling

“…These are also known as contrastive learning (Arora et al 2019;Wang and Isola 2020;Nguyen et al 2022;) that learns the representation similarity of positive and negative samples. Some recent studies (Xu et al 2022) apply mutual information for monolingual topic modeling and focus on the representations of documents. We share the same perspective of information theory but look into a different problem, cross-lingual topic modeling.…”