dit: a Python package for discrete information theory

James, Ryan G.; Ellison, Christopher J.; Crutchfield, James P.

doi:10.21105/joss.00738

Cited by 55 publications

(64 citation statements)

References 34 publications

(16 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The heuristics include using the Möbius inversion on a subset of the lattice as a linear constraint. Several techniques such as this are implemented in the Python information theory package dit [40].…”

Section: Quantifying Unique Informationmentioning

confidence: 99%

See 1 more Smart Citation

Unique information via dependency constraints

James¹,

Emenheiser²,

Crutchfield³

2018

J. Phys. A: Math. Theor.

Self Cite

111

View full text Add to dashboard Cite

The partial information decomposition (PID) is perhaps the leading proposal for resolving information shared between a set of sources and a target into redundant, synergistic, and unique constituents. Unfortunately, the PID framework has been hindered by a lack of a generally agreed-upon, multivariate method of quantifying the constituents. Here, we take a step toward rectifying this by developing a decomposition based on a new method that quantifies unique information. We first develop a broadly applicable method-the dependency decomposition-that delineates how statistical dependencies influence the structure of a joint distribution. The dependency decomposition then allows us to define a measure of the information about a target that can be uniquely attributed to a particular source as the least amount which the source-target statistical dependency can influence the information shared between the sources and the target. The result is the first measure that satisfies the core axioms of the PID framework while not satisfying the Blackwell relation, which depends on a particular interpretation of how the variables are related. This makes a key step forward to a practical PID. PACS numbers: 05.45.-a 89.75.Kd 89.70.+c 02.50.-r

show abstract

Section: Quantifying Unique Informationmentioning

confidence: 99%

“…That is, each channel "pushes" Pr (Y = 0) 4 /3 of the way from '0' toward 4 Although, it is not hard to find a simpler example. See the dit [40] documentation for another: http://docs.dit.io/en/latest/measures/ pid.html#and-are-distinct.…”

Section: Quantifying Unique Informationmentioning

confidence: 99%

Unique information via dependency constraints

James¹,

Emenheiser²,

Crutchfield³

2018

J. Phys. A: Math. Theor.

Self Cite

111

View full text Add to dashboard Cite

show abstract

“…We provide a Jupyter notebook [86] making use of dit ("Discrete Information Theory") [87], an open source Python package that readily calculates these quantities.…”

Section: Appendix a A Python Discrete Information Packagementioning

confidence: 99%

Multivariate Dependence beyond Shannon Information

James

Crutchfield

2017

Entropy

View full text Add to dashboard Cite

Accurately determining dependency structure is critical to understanding a complex system's organization. We recently showed that the transfer entropy fails in a key aspect of this-measuring information flow-due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that Shannon information measures (entropy and mutual information, in their conditional and multivariate forms) can fail to accurately ascertain multivariate dependencies due to their conflation of qualitatively different relations among variables. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful relationships among variables within joint probability distributions. We close by demonstrating that such distributions exist across an arbitrary set of variables.

show abstract

“…The value of M(y) can be understood as the additional scale units that can be described by an additional bit of information, given that the first y bits have been optimally utilized. Code for computing the MUI has been developed and is available online [69].…”

Section: Marginal Utility Of Informationmentioning

confidence: 99%

Multiscale Information Theory and the Marginal Utility of Information

Allen

Stacey

Bar‐Yam

2017

Entropy

View full text Add to dashboard Cite

Complex systems display behavior at a range of scales. Large-scale behaviors can emerge from the correlated or dependent behavior of individual small-scale components. To capture this observation in a rigorous and general way, we introduce a formalism for multiscale information theory. Dependent behavior among system components results in overlapping or shared information. A system's structure is revealed in the sharing of information across the system's dependencies, each of which has an associated scale. Counting information according to its scale yields the quantity of scale-weighted information, which is conserved when a system is reorganized. In the interest of flexibility we allow information to be quantified using any function that satisfies two basic axioms. Shannon information and vector space dimension are examples. We discuss two quantitative indices that summarize system structure: an existing index, the complexity profile, and a new index, the marginal utility of information. Using simple examples, we show how these indices capture the multiscale structure of complex systems in a quantitative way.

show abstract

dit: a Python package for discrete information theory

Cited by 55 publications

References 34 publications

Unique information via dependency constraints

Unique information via dependency constraints

Multivariate Dependence beyond Shannon Information

Multiscale Information Theory and the Marginal Utility of Information

Contact Info

Product

Resources

About