Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

Abstract

A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffering from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations.

A Prototype-Oriented Clustering for Domain Shift with Source Privacy

“…For a complete picture of the field, readers may refer to the survey by Min et al (2018). We emphasize deep-clustering-based approaches, which attempt to learn the feature representation of the data while simultaneously discovering the underlying clusters: K-means Caron et al, 2018), information maximization (Menapace et al, 2020;Ji et al, 2019;Kim and Ha, 2021;Do et al, 2021), transport alignment (Asano et al, 2019;Caron et al, 2020;Wang et al, 2022), neighborhood-clustering (Xie et al, 2016;Huang et al, 2019;Dang et al, 2021), contrastive learning (Pan and Kang, 2021;Shen et al, 2021), probabilistic approaches Monnier et al, 2020;Falck et al, 2021;Manduchi et al, 2021), and kernel density (Yang and Li, 2021). These works primarily focus on clustering data for downstream tasks for a single domain, whereas our clustering algorithm is designed to cluster the data from multiple domains.…”

Editorial for the Topic on Micromachining for Advanced Biological Imaging

“…PCAE (Tu et al 2023) also proposes a flexible generation of the output by VAE, which shares a similar idea, and we focus on VQ embeddings as well. Attempts to include word embeddings, mostly using GloVe (Pennington, Socher, and Manning 2014), into generative (Petterson et al 2010;Dieng, Ruiz, and Blei 2020;Duan et al 2021) or non-generative (Wang et al 2022;Xu et al 2022;Tu et al 2023) topic modeling frameworks have also demonstrated successfully topic modeling performance. Moreover, utilizing pre-trained language models (PLMs) such as BERT (Devlin et al 2018), RoBERTa (Liu et al 2019), and XLNet (Yang et al 2019) has emerged as a new trend in mining topic models.…”