Quasi-Bayes Properties of a Procedure for Sequential Learning in Mixture Models

Abstract

Summary Bayesian methods are often optimal, yet increasing pressure for fast computations, especially with streaming data, brings renewed interest in faster, possibly suboptimal, solutions. The extent to which these algorithms approximate Bayesian solutions is a question of interest, but often unanswered. We propose a methodology to address this question in predictive settings, when the algorithm can be reinterpreted as a probabilistic predictive rule. We specifically develop the proposed methodology for a recursive procedure for on‐line learning in non‐parametric mixture models, which is often referred to as Newton's algorithm. This algorithm is simple and fast; however, its approximation properties are unclear. By reinterpreting it as a predictive rule, we can show that it underlies a statistical model which is, asymptotically, a Bayesian, exchangeable mixture model. In this sense, the recursive rule provides a quasi‐Bayes solution. Although the algorithm offers only a point estimate, our clean statistical formulation enables us to provide the asymptotic posterior distribution and asymptotic credible intervals for the mixing distribution. Moreover, it gives insights for tuning the parameters, as we illustrate in simulation studies, and paves the way to extensions in various directions. Beyond mixture models, our approach can be applied to other predictive algorithms.

Predictive Constructions Based on Measure-Valued Pólya Urn Processes

“…The resulting measure-valued urn process provides a predictive characterization of the law of an asymptotically exchangeable sequence of random variables, which corresponds to the observation process of an implied urn sampling scheme. In fact, the model ( 6)-( 7) fits into a line of recent research, which explores efficient predictive constructions for fast online prediction or approximately-Bayesian solutions, see [11,29,32] and references therein. To that end, one direction for future work is to generalize the functional relationship in (7) and/or, as one referee suggested, to consider finitely-additive measures, along the lines discussed in [33].…”

“…In fact, some of the predictive constructions in [11,29] can be framed in such a way. Theorem 1 extends to GMVPPs, provided that we condition all quantities on the parameter V. As a consequence, there exists a measurable function…”

“…Indeed, as new observations become available, predictions can be updated at a constant computational cost and with limited storage of information. If, in addition, (X n ) n≥1 is asymptotically exchangeable, then (4)-( 5) can provide a computationally simple approximation of an exchangeable scheme for Bayesian inference, along the lines in [11]. The recursive formula (5) allows us to interpret the dynamics of MVPPs in terms of an urn sampling scheme, as the name suggests.…”

Bayesian Predictive Inference Without a Prior

“…sequences have been introduced in [4] and [22] and then investigated in various papers; see e.g. [1], [2], [5], [6], [7], [8], [9], [11], [15], [18], [19].…”

Bayesian predictive inference without a prior

“…sequences have been introduced in [4] and [22] and then investigated in various papers; see e.g. [1], [2], [5], [6], [7], [8], [9], [11], [14], [17], [18].…”