Learning performance in inverse Ising problems with sparse teacher couplings

Abstract

We investigate the learning performance of the pseudolikelihood maximization method for inverse Ising problems. In the teacher–student scenario under the assumption that the teacher’s couplings are sparse and the student does not know the graphical structure, the learning curve and order parameters are assessed in the typical case using the replica and cavity methods from statistical mechanics. Our formulation is also applicable to a certain class of cost functions having locality; the standard likelihood does not belong to that class. The derived analytical formulas indicate that the perfect inference of the presence/absence of the teacher’s couplings is possible in the thermodynamic limit taking the number of spins N as infinity while keeping the dataset size M proportional to N, as long as α = M/N > 2. Meanwhile, the formulas also show that the estimated coupling values corresponding to the truly existing ones in the teacher tend to be overestimated in the absolute value, manifesting the presence of estimation bias. These results are considered to be exact in the thermodynamic limit on locally tree-like networks, such as the regular random or Erdős–Rényi graphs. Numerical simulation results fully support the theoretical predictions. Additional biases in the estimators on loopy graphs are also discussed.

On Model Selection Consistency of Lasso for High-Dimensional Ising Models

“…On the other hand, for sparse tree-like graph in the paramagnetic phase, the inverse covariance matrix C −1 can be computed from the Hessian of the Gibbs free energy [Abbara et al, 2020, Nguyen and Berg, 2012, Ricci-Tersenghi, 2012. Specifically, each element of the covariance matrix C = {C rt } r,t∈V can be expressed as…”

“…where the evaluations are performed at σ = 0 and m = arg min m A(m) (= 0 under the paramagnetic assumption). Consequently, the inverse covariance matrix of a tree-like graph G ∈ G p,d can be computed as [Abbara et al, 2020, Nguyen and Berg, 2012, Ricci-Tersenghi, 2012]…”

“…In the present case, the loss is quadratic and it is not difficult to show its model consistency, but in the general case it would become a nontrivial task. Nevertheless, in Abbara et al [2020] it has been shown by employing the tree structure of the generative Ising model that θ\r is model consistent for a wide class of loss functions corresponding to the single index model with the argument t( =r) θ * rt x r x t . Their "proof" is using the replica method and thus is not rigorous.…”

“…Consequently, one natural question is that, is it still possible to recover the structure of the Ising model using a misspecified loss function such as the quadratic loss? To the best of our knowledge, this problem has not been rigorously studied, though there are several recent non-rigorous works in the statistical physics literature [Abbara et al, 2020, Meng et al, 2020. In particular, it is shown in Meng et al [2021] that even in the case with a misspecified quadratic loss, the neighborhood-based least absolute shrinkage and selection operator (Lasso) [Tibshirani, 1996] (termed as 1 -regularized linear regression ( 1 -LinR) in Meng et al [2021]) has the same order of sample complexity as 1 -LogR for random regular (RR) graphs in the whole paramagnetic phase [Mezard and Montanari, 2009].…”

Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

“…The advent of massive data across various scientific disciplines has led to the widespread use of undirected graphical models, also known as Markov random fields (MRFs), as a tool for discovering and visualizing dependencies among covariates in multivariate data (Wainwright & Jordan, 2008). The Ising model, originally proposed in statistical physics, is one special class of binary MRFs with pairwise potentials and has been widely used in different domains such as image analysis, social networking, gene network analysis (Nguyen et al, 2017;Aurell & Ekeberg, 2012;Bachschmid-Romano & Opper, 2015;Berg, 2017;Bachschmid-Romano & Opper, 2017;Abbara et al, 2020). Among various applications, one fundamental problem of interest is called Ising model selection, which refers to recovering the underlying graph structure of the original Ising model from independent, identically distributed (i.i.d.)…”