Pairwise network information and nonlinear correlations

Abstract

Reconstructing the structural connectivity between interacting units from observed activity is a challenge across many different disciplines. The fundamental first step is to establish whether or to what extent the interactions between the units can be considered pairwise and, thus, can be modeled as an interaction network with simple links corresponding to pairwise interactions. In principle this can be determined by comparing the maximum entropy given the bivariate probability distributions to the true joint entropy. In many practical cases this is not an option since the bivariate distributions needed may not be reliably estimated, or the optimization is too computationally expensive. Here we present an approach that allows one to use mutual informations as a proxy for the bivariate probability distributions. This has the advantage of being less computationally expensive and easier to estimate. We achieve this by introducing a novel entropy maximization scheme that is based on conditioning on entropies and mutual informations. This renders our approach typically superior to other methods based on linear approximations. The advantages of the proposed method are documented using oscillator networks and a resting-state human brain network as generic relevant examples. Pairwise measures of dependence such as crosscorrelations (as measured by the Pearson correlation coefficient or covariance matrix) and mutual information are widely used to characterize the interactions within complex systems. They are a key ingredient to techniques such as principal component analysis, empirical orthogonal functions, and functional networks (networks inferred from dynamical time series) [1][2][3]. These techniques are widespread since they provide greatly simplified descriptions of complex systems, and allow for the analysis of what might otherwise be intractable problems [4]. In particular, functional networks have been widely applied in fields such as neuroscience [4,5], genetics [6], and cell physiology [7], as well as in climate research [1,8].In this paper we study how faithfully these measures alone can represent a given system. With the increasing use of functional networks this topic has received much attention recently, and many technical concerns have been brought to light dealing with the inference of these networks. Previous studies have shown that the estimates of the functional networks can be negatively affected by properties of the time series [9][10][11], as well as properties of the measure of association, e.g. crosscorrelations [12][13][14][15]. In this work however, we address a more fundamental question: How well do pairwise measurements represent a system?In principle this can be evaluated using a maximum entropy approach. The corresponding framework was first laid out in [16] and later applied in [17], where they assessed the rationale of only looking at the pairwise correlations between neurons. They examined how well the maximum entropy distribution, consistent with all the pairwise correlations described...

Network Inference and Maximum Entropy Estimation on Information Diagrams

“…The absence of techniques to efficiently calculate the maximum entropy in these cases is conspicuous; conditioning on the univariate entropies alone is equivalent to assuming the variables are independent, a widely used result, but a generalisation to a wider array of information-theoretic terms has not been forthcoming to the best of our knowledge. In 30 we introduced a method to address this issue using the set-theoretic formulation of information theory. Here, we extend our maximum entropy technique to continuous random variables and vastly expand the types of information-theoretic quantities one can condition on, compared to the univariate entropies and bivariate mutual informations discussed in ref.…”

“…Here, we extend our maximum entropy technique to continuous random variables and vastly expand the types of information-theoretic quantities one can condition on, compared to the univariate entropies and bivariate mutual informations discussed in ref. 30 .…”

“…However this does not seem to be a large issue in practice, as can be seen in our results in ref. 30 .…”

“…In order to overcome these difficulties we propose here a novel methodology that (i) explicitly takes into account nonlinear relationships, (ii) allows one to infer direct network connections, (iii) is much faster than other techniques for cardinalities greater than 2, and (iv) can be applied reliably even in the undersampled regime. It is based on the set-theoretic formulation of information theory and conditions on mutual informations 30 . Additionally, we use this methodology to construct a non-parametric estimate of connected informations, introduced in ref.…”

“…In the section Method we introduce our method of entropy maximization and show how one can vastly increase the types of information-theoretic quantities that one can condition on using the method we introduced in ref. 30 , as well as extend the method to continuous variables. Next, in the section Network Inference we show how to infer direct network connectivity using our method.…”