Exact or approximate inference in graphical models: why the choice is dictated by the treewidth, and how variable elimination can be exploited

Abstract

Probabilistic graphical models offer a powerful framework to account for the dependence structure between variables, which is represented as a graph. However, the dependence between variables may render inference tasks intractable. In this paper, we review techniques exploiting the graph structure for exact inference, borrowed from optimisation and computer science. They are built on the principle of variable elimination whose complexity is dictated in an intricate way by the order in which variables are eliminated. The so-called treewidth of the graph characterises this algorithmic complexity: low-treewidth graphs can be processed efficiently. The first point that we illustrate is therefore the idea that for inference in graphical models, the number of variables is not the limiting factor, and it is worth checking the width of several tree decompositions of the graph before resorting to the approximate method. We show how algorithms providing an upper bound of the treewidth can be exploited to derive a 'good' elimination order enabling to realise exact inference. The second point is that when the treewidth is too large, algorithms for approximate inference linked to the principle of variable elimination, such as loopy belief propagation and variational approaches, can lead to accurate results while being much less time consuming than Monte-Carlo approaches. We illustrate the techniques reviewed in this article on benchmarks of inference problems in genetic linkage analysis and computer vision, as well as on hidden variables restoration in coupled Hidden Markov Models.

Partition Function Estimation: A Quantitative Study

“…Tree Expectation Propagation represents factors with tree approximations using the expectation propagation framework, as opposed to LBP that represents each factor with a product of single node messages. The algorithm is a generalization of LBP since if the tree distribution approximation of factors has no edges, the results are identical to LBP [Qi and Minka, 2004].…”

“…Approximate [Jerrum and Sinclair, 1993;Darwiche, 1995;Gogate and Dechter, 2005;Georgiev et al, 2012;Ermon et al, 2013a;Ermon et al, 2013b;Kuck et al, 2018;Lee et al, 2019;Sharma et al, 2019] 3. Guarantee-less [Pearl, 1982;Yedidia et al, 2000;Minka, 2001;Dechter et al, 2002;Wiegerinck and Heskes, 2003;Qi and Minka, 2004;Eaton and Ghahramani, 2009;Liu and Ihler, 2011;Kuck et al, 2020] While the exact techniques return an accurate result, the approximate methods typically provide (ε, δ) guarantees such that the returned estimate is within ε factor of the true value with confidence of at least 1 − δ. Finally, the guarantee-less methods return estimates without any accuracy or confidence guarantees.…”

“…1. Message passing [Pearl, 1982;Yedidia et al, 2000;Minka, 2001;Wainwright et al, 2001;Dechter et al, 2002;Wiegerinck and Heskes, 2003;Qi and Minka, 2004;Eaton and Ghahramani, 2009] 2. Variable elimination [Dechter, 1996;Dechter, 1999;Liu and Ihler, 2011;Peyrard et al, 2019;Lee et al, 2019] 3.…”

“…Message passing [Pearl, 1982;Yedidia et al, 2000;Minka, 2001;Wainwright et al, 2001;Dechter et al, 2002;Wiegerinck and Heskes, 2003;Qi and Minka, 2004;Eaton and Ghahramani, 2009] 2. Variable elimination [Dechter, 1996;Dechter, 1999;Liu and Ihler, 2011;Peyrard et al, 2019;Lee et al, 2019] 3. Model counting [Darwiche, 2001a;Chavira et al, 2004;Chavira and Darwiche, 2007;Darwiche, 2011;Ermon et al, 2013a;Ermon et al, 2013b;Viricel et al, 2016;Lagniez and Marquis, 2017;Grover et al, 2018;Shu et al, 2018;Sharma et al, 2019;de Colnet and Meel, 2019;Wu et al, 2019;Lagniez and Marquis, 2019;Shu et al, 2019;Wu et al, 2020;Agrawal et al, 2020;Meel and Akshay, 2020;Soos et al, 2020;Zeng et al, 2020;Dudek et al, 2020] 4.…”

A variational method for parameter estimation in a logistic spatial regression

“…There's no general theory about variational estimators' properties, see e.g. Peyrard et al (2018), paragraph 6.3. However, they are known to be empirically accurate.…”