Topic Modelling Meets Deep Neural Networks: A Survey

Abstract

Topic modelling has been a successful technique for text analysis for almost twenty years. When topic modelling met deep neural networks, there emerged a new and increasingly popular research area, neural topic models, with nearly a hundred models developed and a wide range of applications in neural language understanding such as text generation, summarisation and language models. There is a need to summarise research developments and discuss open problems and future directions. In this paper, we provide a focused yet comprehensive overview of neural topic models for interested researchers in the AI community, so as to facilitate them to navigate and innovate in this fast-growing research area. To the best of our knowledge, ours is the first review on this specific topic.

Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

“…Like other TMs, the latent representation of the document is a distribution over K topics: θj ∈ Σ K , each element of which denotes the proportion of one topic in this document. Previous work shows that the data likelihood can be helpful to regularize the optimization of the a transport based loss (Frogner et al, 2015;Zhao et al, 2021). To amortize the computation of θ j and provide additional regularization, we introduce a regularized CT loss as…”

“…With the recent development in auto-encoding VI, originated from variational autoencoders (VAEs) (Kingma & Welling, 2014;Rezende et al, 2014), deep neural networks have been successfully used to develop neural topic models (NTMs) (Miao et al, 2016;Srivastava & Sutton, 2017;Burkhardt & Kramer, 2019;Zhang et al, 2018;Dieng et al, 2020;Zhao et al, 2021). The key advantage of NTMs is that approximate posterior inference can be carried out easily via a forward pass of the encoder network, without the need for expensive iterative inference scheme per test observation as in both Gibbs sampling and conventional VI.…”

“…Recently, NTMs (Zhang et al, 2018;Burkhardt & Kramer, 2019;Dieng et al, 2020;Wang et al, 2020;Duan et al, 2021b) have attracted significant research interests in topic modeling. Most existing NTMs can be regarded as extensions of BPTM like LDA within the VAEs framework (Zhao et al, 2021). In general, NTMs consist of an encoder network that maps the (normalized) BoW input x to its topic proportion θ, and a decoder network that generates x conditioned on the topics {φ k } K k=1 and proportion θ.…”

Neural Attention-Aware Hierarchical Topic Model

“…To our best knowledge, most state-of-the-art NTMs, like ours, are based on the VAE framework. For a detailed discussion of these NTMs, we refer readers to Zhao et al (2021a). Here, we only discuss the two lines of research that are most relevant to ours.…”

“…To address the above limitations, increasing effort has been made in leveraging deep neural networks (DNNs) for topic modelling, which leads to the so-called neural topic models (NTMs) (Zhao et al, 2021a). Most of these models follow the framework of variational auto-encoders (VAEs) (Kingma and Welling, 2014;Rezende et al, 2014) and adopt an encoder-decoder architecture, in which the encoder transforms the BoW data of each document into the corresponding documenttopical embeddings, and the decoder attempts to map these embeddings back to the same data.…”