Sparsemax and Relaxed Wasserstein for Topic Sparsity

Abstract

Topic sparsity refers to the observation that individual documents usually focus on several salient topics instead of covering a wide variety of topics, and a real topic adopts a narrow range of terms instead of a wide coverage of the vocabulary. Understanding this topic sparsity is especially important for analyzing user-generated web content and social media, which are featured in the form of extremely short posts and discussions. As topic sparsity of individual documents in online social media increases, so does the difficulty of analyzing the online text sources using traditional methods.In this paper, we propose two novel neural models by providing sparse posterior distributions over topics based on the Gaussian sparsemax construction, enabling efficient training by stochastic backpropagation. We construct an inference network conditioned on the input data and infer the variational distribution with the relaxed Wasserstein (RW) divergence. Unlike existing works based on Gaussian softmax construction and Kullback-Leibler (KL) divergence, our approaches can identify latent topic sparsity with training stability, predictive performance, and topic coherence. Experiments on different genres of large text corpora have demonstrated the effectiveness of our models as they outperform both probabilistic and neural methods.

Topic Modelling Meets Deep Neural Networks: A Survey

“…Other than Dirichlet, [Miao et al, 2017] introduces a Gaussian softmax (GSM) function in the encoder: q z := softmax N (µ, diag K (σ 2 )) and [Silveira et al, 2018] proposes to use a logistic-normal mixture distribution for the prior of z. To further enhance the sparsity in z, [Lin et al, 2019] introduces to use the sparsemax function to replace the softmax in GSM.…”

Neural variational sparse topic model for sparse explainable text representation

“…It significantly improved the flexibility and efficiency of the original sparse topic model. Most recently, Lin et al [7] proposed Neural SparseMax Document and Topic Models, which utilized sparsemax to directly control the sparsity of the topics. It outperformed previous neural sparse topic methods such as [6] in quality and stability.…”

“…There are many researches devoting to combine deep learning techniques with topic models. In these models [5,4,6,7,8,9], the back propagation method is utilized to automatically update the parameters during the training, since the probabilistic mixtures in the generative process of traditional topic models are replaced by deep neural networks. Thus, they can be derived easily and extended flexibly with the neural network structures and the back propagation.…”

Lifelong Learning of Topics and Domain-Specific Word Embeddings

“…This ensures that each document covers limited topics (Chen et al, 2020b). The sparsity of topics encourages interpretable topics (Card et al, 2018), which corresponds with the tuition that a document usually focuses on several salient topics instead of covering a wide variety of topics (Lin et al, 2019). Although NMF has given sparseness to V t , a more direct control over such properties of the representation is still needed (Hoyer, 2004).…”

“…Considering that it is important to discover discriminative topics, we also adopt the topic uniqueness (TU) score (Nan et al, 2019) to measure the diversity of topics. In addition, the sparsity score of the document-topic distribution (TS-U) and the topic-word distribution (TS-V) proposed by Lin et al (2019) is further used to measure the topic sparsity quantitatively. Particularly, we use 1e-20 as the threshold to count the number of zero values in document-topic and topicword distributions.…”