Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses

Abstract

We investigate the relationship between the structure of a discrete graphical model and the support of the inverse of a generalized covariance matrix. We show that for certain graph structures, the support of the inverse covariance matrix of indicator variables on the vertices of a graph reflects the conditional independence structure of the graph. Our work extends results that have previously been established only in the context of multivariate Gaussian graphical models, thereby addressing an open question about the significance of the inverse covariance matrix of a non-Gaussian distribution. The proof exploits a combination of ideas from the geometry of exponential families, junction tree theory and convex analysis. These populationlevel results have various consequences for graph selection methods, both known and novel, including a novel method for structure estimation for missing or corrupted observations. We provide nonasymptotic guarantees for such methods and illustrate the sharpness of these predictions via simulations. This is an electronic reprint of the original article published by the Institute of Mathematical Statistics in The Annals of Statistics, 2013, Vol. 41, No. 6, 3022-3049. This reprint differs from the original in pagination and typographic detail. 1 2 P.-L. LOH AND M. J. WAINWRIGHTpropensity to spread an infectious disease [28]. It is a classical corollary of the Hammersley-Clifford theorem [5,15,21] that zeros in the inverse covariance matrix of a multivariate Gaussian distribution indicate absent edges in the corresponding graphical model. This fact, combined with various types of statistical estimators suited to high dimensions, has been leveraged by many authors to recover the structure of a Gaussian graphical model when the edge set is sparse (see the papers [8,27,31,38] and the references therein). Recently, Liu et al. [23] and Liu, Lafferty and Wasserman [24] introduced the notion of a nonparanormal distribution, which generalizes the Gaussian distribution by allowing for monotonic univariate transformations, and argued that the same structural properties of the inverse covariance matrix carry over to the nonparanormal; see also the related work of Xue and Zou [37] on copula transformations. However, for non-Gaussian graphical models, the question of whether a general relationship exists between conditional independence and the structure of the inverse covariance matrix remains unresolved. In this paper, we establish a number of interesting links between covariance matrices and the edge structure of an underlying graph in the case of discrete-valued random variables. (Although we specialize our treatment to multinomial random variables due to their widespread applicability, several of our results have straightforward generalizations to other types of exponential families.) Instead of only analyzing the standard covariance matrix, we show that it is often fruitful to augment the usual covariance matrix with higher-order interaction terms. Our main result has an interesting corollary ...

Structural and parametric uncertainties in full Bayesian and graphical lasso based approaches: Beyond edge weights in psychological networks

“…This line of research was closely related to the investigation of the representational capabilities of probabilistic graphical models (PGMs), using undirected graphs (Markov networks, Markov Random Fields, MRFs) and directed acyclic graphs (Bayesian networks, BNs) [10]- [13]. The first lasso proposals to learn sparse probabilistic graphical models ("graphical lasso") assumed multivariate normal distributions [4], [5], which were extended toward the Bayesian framework [14], [15] and also toward non-Gaussian, discrete cases [16], [17]. Extensions for binary data resulting in binary pairwise Markov Random Fields (bPRMs), were also reported based on predictive approximations [18], [19].…”

“…Another open question is the remaining uncertainty at the level of model structures and in our case especially its implications for the effect strength parameters, although asymptotic consistency is proven for graphical lasso under the normality assumption [4] and sample complexity results (finite sample bounds) were reported [16], [20]. As an approximation, bootstrap methods were proposed, for its early application in the PGM field, see e.g.…”

Constructing Graphical Models via the Focused Information Criterion

“…While it would lead too far to construct a complete list of the existing work on Gaussian and 0/1 binary data for graph construction, we refer to some recent work of Yang et al (2012), Lee and Hastie (2012), Jalali et al (2010) and Loh and Wainwright (2012) who construct procedures oriented towards either situations where X is a discrete random variable, or situations where the distribution of X is a member of the more general exponential family of distributions. The above mentioned works are relevant to our case for several reasons.…”

“…This is the topic of Loh and Wainwright (2012). Unfortunately this property of having 0's on position (s,t) and (t, s) in a general inverse of a covariance matrix if an edge is missing between nodes s and t does not hold for general graphs.…”

Copula directed acyclic graphs

“…Stepping outside the multivariate normality assumption for graphical modeling is mainly addressed for undirected graphical models, see for example, Wainwright and Jordan (2008), Jalali et al (2010), Yang et al (2012), Lee and Hastie (2014), or Loh and Wainwright (2013) to name just a few, where most often one models each node assuming its distribution is a member of the more general exponential family.…”