Extracting nonlinear neural topics with neural variational bayes

Abstract

Recently, topic modeling has been upgraded by neural variational inference, which simultaneously allows the model structures deeper and proposes efficient update rules with the reparameterization trick. We formally call this recent new art as neural topic model. In this paper, we investigate a problem of neural topic models, where they formulate topic embeddings and measure the word weights within topics by linear transformation between topic and word embeddings, resulting in redundant and inaccurate topic representations.To solve this problem, we propose a novel neural topic model, namely Generative Model with Nonlinear Neural Topics (GMNNT). The insight of GMNNT is to replace the topic embeddings with neural networks of topics, named neural topic, so as to capture nonlinear relationships between words in the embedding space, enabling to induce more accurate topic representations. We derive the inference process of GMNNT under the framework of neural variational inference. Extensive empirical studies have been conducted on several widely used collections of documents, including datasets of both short texts and normal long texts. The experimental results validate that GMNNT can output more semantically coherent topics compared with traditional topic models and neural topic models.