Quantifying Redundant Information in Predicting a Target Random Variable

Griffith, Virgil; Ho, Tracey

doi:10.3390/e17074644

Cited by 26 publications

(38 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Griffith et al [13] suggest an alternative measure motivated by zero-error information, which again formulates an optimisation problem (here maximisation of mutual information) over a family of distributions (here distributions Q which are a function of each predictor so that H(Q|X i ) = 0). Griffith and Ho [16] extend this approach by modifying the optimisation constraint to be H(Q|X i ) = H(Q|X i , Y).…”

Section: Other Redundancy Measuresmentioning

confidence: 99%

See 1 more Smart Citation

Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal

Ince

2017

Entropy

147

303

View full text Add to dashboard Cite

Abstract:The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables X i . It can be thought of as quantifying overlapping information content or similarities in the representation of S between the X i . We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.

show abstract

Section: Other Redundancy Measuresmentioning

confidence: 99%

“…It can therefore overstate the amount of redundancy in a particular set of variables. Several studies have noted this point and suggested alternative approaches [10][11][12][13][14][15][16].…”

Section: Introductionmentioning

confidence: 99%

Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal

Ince

2017

Entropy

147

303

View full text Add to dashboard Cite

show abstract

“…To do that, we need to specify one of the four variables in the system, typically by providing an expression to calculate either R or S. There are a number of proposals in the literature [28][29][30][31], but at the time of writing there is no consensus on any one candidate.…”

Section: Non-negative Decomposition Of Multivariate Informationmentioning

confidence: 99%

The Partial Information Decomposition of Generative Neural Network Models

Tax¹,

Mediano

Shanahan

2017

Entropy

View full text Add to dashboard Cite

In this work we study the distributed representations learnt by generative neural network models. In particular, we investigate the properties of redundant and synergistic information that groups of hidden neurons contain about the target variable. To this end, we use an emerging branch of information theory called partial information decomposition (PID) and track the informational properties of the neurons through training. We find two differentiated phases during the training process: a first short phase in which the neurons learn redundant information about the target, and a second phase in which neurons start specialising and each of them learns unique information about the target. We also find that in smaller networks individual neurons learn more specific information about certain features of the input, suggesting that learning pressure can encourage disentangled representations.

show abstract

“…From equations , it is evident that an additional formula for any single S , U , or R component is required for the computation of all others, since the mutual information values are directly computable from the pdf s. To this end, various methods of computing U [ Bertschinger et al ., ], S [ Griffith and Koch , ; Olbrich et al ., ], and R [ Williams and Beer , ; Harder et al ., ; Griffith and Ho , ] components have been proposed, although there is no universal agreement on the appropriate method. The most common approach is to first compute R , and several proposed measures have been shown to reduce to simply the minimum shared information between the target X tar and either source as follows [ Barrett , ]:

R_{M M I} = \min [I (X_{s 1}; X_{t a r}), I (X_{s 2}; X_{t a r})]

where MMI denotes “minimum mutual information.” In a cardiovascular application, Faes et al .…”

Section: Information Partitioning Into Synergistic Unique and Redunmentioning

confidence: 99%

“…[] used this metric to show that heart period, arterial pressure, and respiration flow are intertwined elements that control heart rate together, and that there is higher R between subsystems during fast‐paced breathing. However, this formulation for R actually represents a maximum bound for redundancy, since it assumes that all information provided by the weaker source is redundant [ Barrett , ; Griffith and Ho , ]. The use of

R_{M M I}

as a redundancy metric greatly decreases the number of ways in which information can be partitioned into synergistic, unique, and redundant components.…”

Section: Information Partitioning Into Synergistic Unique and Redunmentioning

confidence: 99%

Temporal information partitioning: Characterizing synergy, uniqueness, and redundancy in interacting environmental variables

Goodwell

Kumar

2017

Water Resources Research

113

View full text Add to dashboard Cite

Information theoretic measures can be used to identify nonlinear interactions between source and target variables through reductions in uncertainty. In information partitioning, multivariate mutual information is decomposed into synergistic, unique, and redundant components. Synergy is information shared only when sources influence a target together, uniqueness is information only provided by one source, and redundancy is overlapping shared information from multiple sources. While this partitioning has been applied to provide insights into complex dependencies, several proposed partitioning methods overestimate redundant information and omit a component of unique information because they do not account for source dependencies. Additionally, information partitioning has only been applied to time‐series data in a limited context, using basic pdf estimation techniques or a Gaussian assumption. We develop a Rescaled Redundancy measure (Rs) to solve the source dependency issue, and present Gaussian, autoregressive, and chaotic test cases to demonstrate its advantages over existing techniques in the presence of noise, various source correlations, and different types of interactions. This study constitutes the first rigorous application of information partitioning to environmental time‐series data, and addresses how noise, pdf estimation technique, or source dependencies can influence detected measures. We illustrate how our techniques can unravel the complex nature of forcing and feedback within an ecohydrologic system with an application to 1 min environmental signals of air temperature, relative humidity, and windspeed. The methods presented here are applicable to the study of a broad range of complex systems composed of interacting variables.

show abstract

Quantifying Redundant Information in Predicting a Target Random Variable

Cited by 26 publications

References 17 publications

Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal

Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal

The Partial Information Decomposition of Generative Neural Network Models

Temporal information partitioning: Characterizing synergy, uniqueness, and redundancy in interacting environmental variables

Contact Info

Product

Resources

About