Larger Residuals, Less Work: Active Document Scheduling for Latent Dirichlet Allocation

Abstract

Abstract. Recently, there have been considerable advances in fast inference for latent Dirichlet allocation (LDA). In particular, stochastic optimization of the variational Bayes (VB) objective function with a natural gradient step was proved to converge and able to process massive document collections. To reduce noise in the gradient estimation, it considers multiple documents chosen uniformly at random. While it is widely recognized that the scheduling of documents in stochastic optimization may have significant consequences, this issue remains largely unexplored. In this work, we address this issue. Specifically, we propose residual LDA, a novel, easy-to-implement, LDA approach that schedules documents in an informed way. Intuitively, in each iteration, residual LDA actively selects documents that exert a disproportionately large influence on the current residual to compute the next update. On several real-world datasets, including 3M articles from Wikipedia, we demonstrate that residual LDA can handily analyze massive document collections and find topic models as good or better than those found with batch VB and randomly scheduled VB, and significantly faster.

Fast Online EM for Big Topic Modeling

“…Moreover, we show that the unified EM framework can explain recent LDA inference algorithms like VB [2], GS [5], CVB [7] and BP [8]. Experiments on four big data streams confirm that FOEM is significantly faster and more memoryefficient than the state-of-the-art online LDA algorithms including OGS [11], OVB [12], RVB [13], SOI [14] and SCVB [15]. We anticipate that the proposed FOEM can be also extended to compute ML or MAP estimate of other mixture models and latent variable models [30].…”

“…where −w, −d and −(w, d) denote all word indices except w, all document indices except d, and all word indices except {w, d}. After the E-step for each word, the M-step will update the sufficient statistics immediately by adding the updated responsibility µ w,d (k) (13) into (14), (15) and (16).…”

“…Recently, most online LDA algorithms are combinations of the stochastic optimization framework [22] with batch LDA algorithms like VB, GS, and CVB, e.g., online VB (OVB) [12], residual VB (RVB) [13], online GS (OGS) [11], sampled online inference (SOI) [14], and stochastic CVB (SCVB) [15]. However, these algorithms focus mainly on the big data problem but rarely on the big model problem.…”

More influence means less work

“…Then, we turned isLDA into a novel, easy-to-implement oLDA approach, called isoLDA, that scales well to massive and growing datasets by applying influence scheduling to randomly formed batches. Based on the results of the present paper, [8] have recently developed the first active LDA.…”

Towards big topic modeling

“…Thus, the higher communication rate leads to the larger communication cost in parallel online LDA algorithms. Therefore, it is nontrivial to reduce the communication complexity (5) for parallel online LDA algorithms [11], [12], [21], [27], [28] in order to achieve a better scalability performance. Moreover, not all parallel batch LDA algorithms based on MPA have been proved to converge to the local optimum of the LDA's objective function.…”