TAN-NTM: Topic Attention Networks for Neural Topic Modeling

Abstract

Topic models have been widely used to learn text representations and gain insight into document corpora. To perform topic discovery, most existing neural models either take document bag-of-words (BoW) or sequence of tokens as input followed by variational inference and BoW reconstruction to learn topic-word distribution. However, leveraging topic-word distribution for learning better features during document encoding has not been explored much. To this end, we develop a framework TAN-NTM, which processes document as a sequence of tokens through a LSTM whose contextual outputs are attended in a topic-aware manner. We propose a novel attention mechanism which factors in topic-word distribution to enable the model to attend on relevant words that convey topic related cues. The output of topic attention module is then used to carry out variational inference. We perform extensive ablations and experiments resulting in ∼ 9 -15 percentage improvement over score of existing SOTA topic models in NPMI coherence on several benchmark datasets -20Newsgroups, Yelp Review Polarity and AGNews. Further, we show that our method learns better latent document-topic features compared to existing topic models through improvement on two downstream tasks: document classification and topic guided keyphrase generation.

FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition

“…Typically, a statistical topic model [15,4] captures topics in the form of latent variables with probability distributions over the entire vocabulary and performs approximate inference over document-topic and topic-word distributions through Variational Bayes [3]. However, such a learning paradigm requires an expensive iterative inference step performed on every document in a corpus [30]. The efficiency is boosted after the introduction of VAE-based (Variational AutoEncoder) neural topic model [2,49] because variational inference can be performed through a single forward pass [19].…”

A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification

“…We adopt a similar approach to (Lu et al, 2018) by modeling attention using topics. However, unlike the topic attention model (TAN), which uses a bag-of-words (BOW) model based on variational inference to align the topic space and word space with extracting meaningful topics (Panwar et al, 2021), we assume that these multiple attention heads represent multiple topics in terms of their semantics.…”

Topic Modelling Meets Deep Neural Networks: A Survey

“…At the similar period of time, [12] independently proposes an SNTM whose generative process is similar to [66], with an additional variable modelling stop words and several variants in the inference process. Recently, [44] proposes to use an LSTM with attentions as the encoder taking s as input, where the attention incorporates topical information with a context vector that is constructed by topic embeddings and document embeddings. [47] introduces an SNTM that is related to [12], where instead of marginalising out the discrete topic assignments, the paper proposes to generate them from an RNN model.…”