Bayesian Network Structure Learning with Integer Programming: Polytopes, Facets and Complexity

Abstract

The challenging task of learning structures of probabilistic graphical models is an important problem within modern AI research. Recent years have witnessed several major algorithmic advances in structure learning for Bayesian networks-arguably the most central class of graphical models-especially in what is known as the score-based setting. A successful generic approach to optimal Bayesian network structure learning (BNSL), based on integer programming (IP), is implemented in the gobnilp system. Despite the recent algorithmic advances, current understanding of foundational aspects underlying the IP based approach to BNSL is still somewhat lacking. Understanding fundamental aspects of cutting planes and the related separation problem is important not only from a purely theoretical perspective, but also since it holds out the promise of further improving the efficiency of state-of-the-art approaches to solving BNSL exactly. In this paper, we make several theoretical contributions towards these goals: (i) we study the computational complexity of the separation problem, proving that the problem is NP-hard; (ii) we formalise and analyse the relationship between three key polytopes underlying the IP-based approach to BNSL; (iii) we study the facets of the three polytopes both from the theoretical and practical perspective, providing, via exhaustive computation, a complete enumeration of facets for low-dimensional family-variable polytopes; and, furthermore, (iv) we establish a tight connection of the BNSL problem to the acyclic subgraph problem.

Learning Bayesian Networks with the Saiyan Algorithm

“…Other score-based approaches that put greater emphasis on pruning the search space of possible graphs, and some guarantee to return the graph that maximises a scoring function, include the Integer Programming (IP) methods by Cussens [16] and Cussens et al [17] that are based on the IP formulation of Bartlett and Cussens [18] and which form part of the GOBNILP system. Other relevant approaches include the Integer Linear Programming (ILP) bounded treewidth approach by Parviainen et al [19], the linear program acyclic approach by Jaakkola et al [20] that reduces search space based on various constraints, the special vector characteristic imset by Hemmecke et al [21], and the branch-and-bound (BnB) linear programming method by Peharz and Pernkopf [22] that maximises a discriminative score to offer an exact solution.…”

Knowledge-Based Regularization in Generative Modeling

“…It has often been discussed with the notion of graphical models [Koller and Friedman, 2009], which are built upon knowledge of conditional independence of features. We have also seen the advances in neural network techniques for generative modeling, such as variational autoencoders (VAEs) [Kingma and Welling, 2014] and generative adversarial networks (GANs) [Goodfellow et al, 2014].…”

“…A promising way to incorporate prior knowledge into general-purpose models is via regularization. For example, structured sparsity regularization [Huang et al, 2011] is known as a method for regularizing linear models with a rigorous theoretical background. In a related context, posterior regularization [Ganchev et al, 2010; has been discussed as a methodology to impose expectation constraints on learned distributions.…”

“…For example, structured sparsity regularization [Huang et al, 2011] is known as a method for regularizing linear models with a rigorous theoretical background. In a related context, posterior regularization [Ganchev et al, 2010; has been discussed as a methodology to impose expectation constraints on learned distributions. However, the applicability of the existing regularization methods is still limited in terms of target types of prior knowledge and base models.…”

A constraint-based genetic algorithm for optimizing neural network architectures for detection of loss of coolant accidents of nuclear power plants

“…SVMs are formulated as quadratic programming problems (a type of CSOPs) and are trained using constrained optimization algorithms such as the sequential minimal optimization algorithm (SMO). Cussens [21,22] and Bartlett [14] modeled the problem of optimizing the structures of Bayesian networks as an integer program (a type of CSOP) with acyclic constraints and solved the optimization model using the cutting plane method.…”