Neural variational sparse topic model for sparse explainable text representation

Abstract

Texts are the major information carrier for internet users, from which learning the latent representations has important research and practical value. Neural topic models have been proposed and have great performance in extracting interpretable latent topics and representations of texts. However, there remain two major limitations: 1) these methods generally ignore the contextual information of texts and have limited feature representation ability due to the shallow feed-forward network architecture, 2) Sparsity of the representations in topic semantic space is ignored. To address these issues, in this paper, we propose a semantic reinforcement neural variational sparse topic model (SR-NSTM) towards explainable and sparse latent text representation learning. Compared with existing neural topic models, SR-NSTM models the generative process of texts with probabilistic distributions parameterized with neural networks and incorporates Bi-directional LSTM to embed contextual information at the document level. It achieves sparse posterior representations over documents and words with zero-mean Laplace distribution and topics with sparsemax. Moreover, we propose a supervised extension of SR-NSTM via adding the max-margin posterior regularization to tackle the supervised tasks. The neural variational inference method is utilized to learn our models efficiently. Experimental results on Web Snippets, 20Newsgroups, BBC, and Biomedical datasets demonstrate that the contextual information and revisiting generative process can improve the performance, leading to the competitive performance of our models in learning coherent topics and explainable sparse representations for texts.

Enhancing Topic Prediction Using Machine Learning Techniques and ConceptNet-based Cosine Similarity Measure

“…Q Xie et al 2021. [15] Utilised the neural networks to develop effective latent representations of text with prior sparse distribution using the semantic reinforcement neural variational sparse topic model (SR-NSTM) method. They also applied Bi-LSTM to take word sequence into account.…”

Knowledge Graph-Based Hierarchical Text Semantic Representation

“…Its purpose is to establish a mechanism for converting natural language with human intent into a structured form that is easy for computers to compare and measure. Researchers have conducted related research on text representation [5,6]. However, language processing tasks still face signifcant difculties because of the semantic variety of natural language texts.…”