Multiscale Information Theory and the Marginal Utility of Information

Allen, Benjamin; Stacey, Blake C.; Bar‐Yam, Yaneer

doi:10.3390/e19060273

Cited by 39 publications

(48 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, since the I-diagrams are identical, so are the complexity profiles. The second profile is the marginal utility of information [56], which is a derivative of a linear programming problem whose constraints are given by the I-diagram, so here, again, they are identical. Finally, we have Schneidman et al's connected information [70], which comprise the differences in entropies of the maximum entropy distributions whose k-and k − 1-way marginals are fixed to match those of the distribution of interest.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multivariate Dependence beyond Shannon Information

James

Crutchfield

2017

Entropy

View full text Add to dashboard Cite

Accurately determining dependency structure is critical to understanding a complex system's organization. We recently showed that the transfer entropy fails in a key aspect of this-measuring information flow-due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that Shannon information measures (entropy and mutual information, in their conditional and multivariate forms) can fail to accurately ascertain multivariate dependencies due to their conflation of qualitatively different relations among variables. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful relationships among variables within joint probability distributions. We close by demonstrating that such distributions exist across an arbitrary set of variables.

show abstract

Section: Methodsmentioning

confidence: 99%

“…There are measures [20,22,44,[47][48][49][50][51][52][53] and expansions [54][55][56] purporting to measure or otherwise extract the complexity, magnitude or structure of dependencies within a multivariate distribution. Many of these techniques, including those just cited, are sums and differences of atoms in these information diagrams.…”

Section: Developmentmentioning

confidence: 99%

Multivariate Dependence beyond Shannon Information

James

Crutchfield

2017

Entropy

View full text Add to dashboard Cite

show abstract

“…It includes methods of studying joint distributions including information diagrams, connected informations (Schneidman et al 2003) (Amari 2001), marginal utility of information (Allen, Stacey, and Bar-Yam 2017), and the complexity profile(Y. Bar-Yam 2004).…”

mentioning

confidence: 99%

dit: a Python package for discrete information theory

James¹,

Ellison²,

Crutchfield³

2018

JOSS

View full text Add to dashboard Cite

Available at: Https://Github.com/Dit/Dit" n.d.) is a Python package for the study of discrete information theory. Information theory is a mathematical framework for the study of quantifying, compressing, and communicating random variables (Cover and Thomas 2006) (MacKay 2003) (Yeung 2008). More recently, information theory has been utilized within the physical and social sciences to quantify how different components of a system interact. dit is primarily concerned with this aspect of the theory.Information theory is a powerful extension to probability and statistics, quantifying dependencies among arbitrary random variables in a way tha tis consistent and comparable across systems and scales. Information theory was originally developed to quantify how quickly and reliably information could be transmitted across an arbitrary channel. The demands of modern, data-driven science have been coopting and extending these quantities and methods into unknown, multivariate settings where the interpretation and best practices are not known. For example, there are at least four reasonable multivariate generalizations of the mutual information, none of which inherit all the interpretations of the standard bivariate case. Which is best to use is context-dependent. dit implements a vast range of multivariate information measures in an effort to allow information practitioners to study how these various measures behave and interact in a variety of contexts. We hope that having all these measures and techniques implemented in one place will allow the development of robust techniques for the automated quantification of dependencies within a system and concrete interpretation of what those dependencies mean.dit implements the vast majority of information measure defined in the literature, including entropies (Shannon(Cover and

show abstract

“…The issue begins with three papers which deal with the foundational aspects of information processing in complex systems [3][4][5]. The study of Allen et al [3] describes two quantitative indices that summarize the structure of a complex system: (i) its complexity profile, based on the multivariate mutual information at a given scale or higher, and (ii) the marginal utility of information, characterizing the extent to which a system can be described using limited amounts of information.…”

mentioning

confidence: 99%

“…The study of Allen et al [3] describes two quantitative indices that summarize the structure of a complex system: (i) its complexity profile, based on the multivariate mutual information at a given scale or higher, and (ii) the marginal utility of information, characterizing the extent to which a system can be described using limited amounts of information. Information is understood to have a scale equal to the multiplicity (or redundancy) at which it arises, and so the analysis shows how these indices capture the multi-scale structure of complex systems.…”

mentioning

confidence: 99%

Complexity, Criticality and Computation

Prokopenko

2017

Entropy

View full text Add to dashboard Cite

What makes a system "complex"? Is it merely the number of components it integrates, a nonlinear nature of the dependencies and feedbacks among its parts, or an unpredictable behavior it exhibits over time? The term "complexity" was initially applied generically to express the lack of predictability, reflecting on the self-organization of a synergistic macroscopic behavior out of interactions between the constituent microscopic parts, and the emergence of global patterns. Without a doubt, by now the concept has acquired a fairly definitive meaning, describing a distinct field of research and education and a new approach to science and engineering. There are abundant examples showing that the enterprise of Complex Systems, having achieved a substantial level of maturity, reaches back into our everyday lives, revealing patterns of complexity that should be considered without employing a reductionist logic [1].Similarly, the idea of criticality was originally motivated by studies of various crises and disruptive events, as well as sensitivities to initial conditions, but over time has developed into a precise field: critical dynamics. Research into critical dynamics is typically focused on the behavior of dynamical spatiotemporal systems during phase transitions where scale invariance prevails and symmetries break. Crucially, such behavior can be understood in terms of the control and order parameters. For instance, a second-order phase transition in a ferromagnetic system, separating two qualitatively different behaviors, can be reached by controlling the temperature parameter: the "disordered" and isotropic (symmetric) high-temperature phase is characterized by the absence of net magnetization, while the "ordered" and anisotropic (less symmetric) low-temperature phase can be described by an order parameter, the net magnetization vector defining a preferred direction in space. Critical phenomena have become associated with the physics of critical points, such as fractal behavior, the divergence of the correlation length, power-law divergences (e.g., the divergence of the magnetic susceptibility in the ferromagnetic phase transition), universality of relevant critical exponents, and so on. Now, these precise theoretical notions begin to reconnect with their motivating applied studies of crisis modeling, forecasting, and response. There is a growing awareness that complexity is strongly related to criticality, and many examples of self-organizing complex systems can be found in applications managing complexity specifically at critical regimes.A similar loop originating in practical studies, maturing to an exact science with precise but narrow definitions, and then reaching back to applied scenarios, can be seen in the realm of distributed computation. These days, complex systems can be viewed as distributed information-processing systems, in the domains ranging from systems biology and artificial life to computational neuroscience, digital circuitry, and transport networks [2]. Consciousness emerging from neuronal ac...

show abstract

Multiscale Information Theory and the Marginal Utility of Information

Cited by 39 publications

References 95 publications

Multivariate Dependence beyond Shannon Information

Multivariate Dependence beyond Shannon Information

dit: a Python package for discrete information theory

Complexity, Criticality and Computation

Contact Info

Product

Resources

About