Tractable Undirected Approximations for Graphical Models

Abstract

Graphical models provide a broad framework for probabilistic inference, with application to such diverse areas as speech recognition (Hidden Markov Models), medical diagnosis (Belief networks) and artificial intelligence (Boltzmann Machines). However, the computing time is typically exponential in the number of nodes in the graph. We present a general framework for a class of approximating models, based on the Kullback-Leibler divergence between an approximating graph and the original graph. We concentrate here on undirected approximations of both intractable directed and undirected graphical models. Simulation results on a small benchmark problem suggest that this method compares favourably against others previously reported in the literature.

Graphical Models, Exponential Families, and Variational Inference

“…More generally, we can consider classes of tractable distributions that incorporate additional structure. This structured mean field approach was first proposed by Saul and Jordan [209], and further developed by various researchers [10,259,120].…”

Variational approximations for categorical causal modeling with latent variables

“…The idea is that the computational complexity can be exponential in the maximal clique size of a junction tree of the approximating model. Jordan et al (1999) For approximate parameter estimation in (intractable) binary sigmoid belief networks using the logistic function, Barber and Wiegerinck (1999) and Wiegerinck and Barber (1999) have described variational approximations where the approximating conditional distributions are based on (tractable) sigmoid belief networks. In principle it is straightforward to extend this idea to models for polytomous categorical variables.…”

“…For some models, such as longitudinal factor analysis, this is not a problem, but for others, such as the longitudinal model of Figure 4, this is a problem (see section 5). Barber and Wiegerinck (1998) use Boltzmann machines as approximating joint (conditional) distributions in variational approximations for models for binary data. Although the principle extends quite easily to polytomous variables, for the time being we will stay with the binary case.…”

Approximate inference for medical diagnosis

“…To construct Q, one first has to define a tractable graphical structure for Q: Q(S) = D, ), Q(S')'I7r,),), [16,17,12]. The next step is to optimize the parameters of Q such that the Kullback-Leibler (KL) divergence between Q and PE, …”

“…Recently, variational methods for approximation are becoming increasingly popular [10,11,12]. An advantage of variational methods techniques is that they provide bounds on the quantity of interest in contrast to stochastic sampling methods which may yield unreliable results due to finite sampling times.…”