Bayesian Analysis of Dynamic Linear Topic Models

Abstract

In dynamic topic modeling, the proportional contribution of a topic to a document depends on the temporal dynamics of that topic's overall prevalence in the corpus. We extend the Dynamic Topic Model of Blei and Lafferty (2006) by explicitly modeling document-level topic proportions with covariates and dynamic structure that includes polynomial trends and periodicity. A Markov Chain Monte Carlo (MCMC) algorithm that utilizes Polya-Gamma data augmentation is developed for posterior inference. Conditional independencies in the model and sampling are made explicit, and our MCMC algorithm is parallelized where possible to allow for inference in large corpora. To address computational bottlenecks associated with Polya-Gamma sampling, we appeal to the Central Limit Theorem to develop a Gaussian approximation to the Polya-Gamma random variable. This approximation is fast and reliable for parameter values relevant in the text-mining domain. Our model and inference algorithm are validated with multiple simulation examples, and we consider the application of modeling trends in PubMed abstracts. We demonstrate that sharing information across documents is critical for accurately estimating document-specific topic proportions. We also show that explicitly modeling polynomial and periodic behavior improves our ability to predict topic prevalence at future time points.

Air Traffic Forecast in Post-Liberalization Context: A Dynamic Linear Models Approach

“…For this research, Dynamic Models (DMs) have been chosen. Unlike static models, some elements that participate in the construction of the model do not remain invariable but are considered as functions of time describing temporal trajectories (Glynn et al, 2019;Laine, 2019;McAlinn & West, 2019;Pole, West, & Harrison, 2018).…”

“…One of their advantages is that by using them one realizes that they are simpler models, yet powerful enough to adjust and forecast data and they may include explanatory variables in a simple way (Sargan & Bhargava, 1983;Ahn & Schmidt, 1995;Arellano & Bond, 1991;Arellano & Bover, 1995;Gelman et al, 2013;Kenkel, 2018). DLMs are defined under the following structure for each time t (Valencia & Correa, 2013;Bolstad, 2007;Glynn et al, 2019;Asparouhov, Hamaker, & Muthén, 2018)…”

Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions

“…These are increasingly common in areas such as consumer behavior in a range of socio-economic contexts, various natural and biological systems, and commercial and economic problems of analysis and forecasting of discrete outcomes (e.g. Cargnoni et al, 1997;Yelland, 2009;Terui and Ban, 2014;Chen and Lee, 2017;Aktekin et al, 2018;Glynn et al, 2019). Often there are questions of modeling simultaneously at different scales as well as of integrating information across series and scales (chapter 16 of Ferreira et al, 2006).…”

Spatio‐temporal models for big multinomial data using the conditional multivariate logit‐beta distribution

“…This leads to easy to sample from conjugate full‐conditional distributions (e.g. see Polson and Scott, ; Zhou et al, ; Chen et al, ; Polson et al, ; Linderman et al, ; Glynn et al, ) for use in a Gibbs sampler. Additionally, this Pólya–Gamma augmentation approach has been used to model dynamics (Blei and Lafferty, ), through the incorporation of a vector autoregressive model (Blei and Lafferty, ; Linderman et al ).…”

“…This leads to easy to sample from conjugate full-conditional distributions (e.g. see Polson and Scott, 2011;Zhou et al, 2012;Chen et al, 2013;Polson et al, 2013;Linderman et al, 2015;Glynn et al, 2018) for use in a Gibbs sampler. Additionally, this Pólya-Gamma augmentation approach has been used to model dynamics (Blei and Lafferty, 2006), through the incorporation of a vector autoregressive model (Blei and Lafferty, 2006;Linderman et al, 2015).The current state-of-the-art in the Bayesian hierarchical modeling literature requires one to augment the multinomial random vector with Pólya-Gamma random variables.…”