Variational Inference over Nonstationary Data Streams for Exponential Family Models

Abstract

In many modern data analysis problems, the available data is not static but, instead, comes in a streaming fashion. Performing Bayesian inference on a data stream is challenging for several reasons. First, it requires continuous model updating and the ability to handle a posterior distribution conditioned on an unbounded data set. Secondly, the underlying data distribution may drift from one time step to another, and the classic i.i.d. (independent and identically distributed), or data exchangeability assumption does not hold anymore. In this paper, we present an approximate Bayesian inference approach using variational methods that addresses these issues for conjugate exponential family models with latent variables. Our proposal makes use of a novel scheme based on hierarchical priors to explicitly model temporal changes of the model parameters. We show how this approach induces an exponential forgetting mechanism with adaptive forgetting rates. The method is able to capture the smoothness of the concept drift, ranging from no drift to abrupt drift. The proposed variational inference scheme maintains the computational efficiency of variational methods over conjugate models, which is critical in streaming settings. The approach is validated on four different domains (energy, finance, geolocation, and text) using four real-world data sets.

Probabilistic Models with Deep Neural Networks

“…While the model structure in Figure 1 at first sight can appear restrictive, it is in fact quite versatile, and many books contain entire sections devoted to LVMs [ 5 , 6 , 8 ]. For instance, LVMs include popular models like latent Dirichlet allocation (LDA) models used to uncover the hidden topics in a text corpora [ 40 ], mixture of Gaussian models to discover hidden clusters in data [ 5 ], probabilistic principal component analysis for dimensionality reduction [ 41 ], and models to capture drift in a data stream [ 42 , 43 ]. They have been used for knowledge extraction from GPS data [ 44 ], genetic data [ 45 ], graph data [ 46 ], and so on.…”

Adaptive infinite dropout for noisy and sparse data streams

“…In this paper, we focus on these challenges. Some recent studies (Broderick et al, 2013;Duc et al, 2017;Masegosa et al, 2017;McInerney et al, 2015;Tran et al, 2021; Van et al, 2022) have provided solutions to learning from data streams. Those methods enable Bayesian models, which are designed for static conditions, to work with data streams.…”

“…Those methods enable Bayesian models, which are designed for static conditions, to work with data streams. The recursive Bayesian approach (Ahn et al, 2019;Broderick et al, 2013;Duc et al, 2017;Masegosa et al, 2017;McInerney et al, 2015;Nguyen et al, 2018) has emerged as an effective solution and has been paid a great deal of attention by researchers. The main idea is that the learned posterior from a mini-batch is used as the prior in the next one.…”

“…This arguably cause several critical issues in the online Bayesian updates including poor uncertainty representation of the underlying data-generating distribution, and lack of the flexibility to adapt to the sudden changes in data streams. Hierarchical power prior (HPP) (Masegosa et al, 2017(Masegosa et al, , 2020 is more plastic to learn a new concept, however, it does not have any efficient way to work on noisy and sparse data. Other recursive Bayesian-based studies (Nguyen et al, 2021;Duc et al, 2017;Tran et al, 2021;Van et al, 2022) require external knowledge to deal with the challenges.…”

Non-stationary, online variational Bayesian learning, with circular variables

“…A form of batching is used by [13], who constrain the overall problem by assuming that the time series is piecewise stationary, and use Monte Carlo techniques to model the unknown size and number of segments. [14] also use batching, and explicitly model temporal changes in the system parameters using adaptive forgetting rates.…”