Fast Online EM for Big Topic Modeling

Abstract

The expectation-maximization (EM) algorithm can compute the maximum-likelihood (ML) or maximum a posterior (MAP) point estimate of the mixture models or latent variable models such as latent Dirichlet allocation (LDA), which has been one of the most popular probabilistic topic modeling methods in the past decade. However, batch EM has high time and space complexities to learn big LDA models from big data streams. In this paper, we present a fast online EM (FOEM) algorithm that infers the topic distribution from the previously unseen documents incrementally with constant memory requirements. Within the stochastic approximation framework, we show that FOEM can converge to the local stationary point of the LDA's likelihood function. By dynamic scheduling for the fast speed and parameter streaming for the low memory usage, FOEM is more efficient for some lifelong topic modeling tasks than the state-of-the-art online LDA algorithms to handle both big data and big models (aka, big topic modeling) on just a PC.Comment: 14 pages, 12 figures in IEEE Transactions on Knowledge and Data Engineering, 201

Latent topics resonance in scientific literature and commentaries: evidences from natural language processing approach

“…These paradigms include Expectation-Maximization (EM), online version ( e.g. , Zeng et al., 2016 ) and parallel version ( e.g. , Wang et al., 2015 ).…”

“…Based on CVB0 ( Teh et al., 2007 ), a stochastic algorithm (SCVB0) was developed to learn human-interpretable topics more accurately and more quickly, both on large and small datasets. SCVB0 has become a standard method whose performance is a benchmark ( Zeng et al., 2016 ).…”

Efficient integration of generative topic models into discriminative classifiers using robust probabilistic kernels

“…The time and memory complexities have been presented in many topic model publications [23,99,36,42,5,17]. Though the work of [99] provided the most extensive details about time and memory complexities when processing large collections under LDA.…”

“…The time and memory complexities have been presented in many topic model publications [23,99,36,42,5,17]. Though the work of [99] provided the most extensive details about time and memory complexities when processing large collections under LDA. Following the work in [99], for D documents containing each N words from a vocabulary of size V , in a particular class c, we obtain a D ⇥ V matrix where NN0 is the total number of nonzero elements in this document-word (sparse) matrix.…”

“…Though the work of [99] provided the most extensive details about time and memory complexities when processing large collections under LDA. Following the work in [99], for D documents containing each N words from a vocabulary of size V , in a particular class c, we obtain a D ⇥ V matrix where NN0 is the total number of nonzero elements in this document-word (sparse) matrix. During the formation of K topics, it involves placing a K + 1-dimensional variational distribution on every word leading to a K ⇥ NN0 matrix.…”