Intersection Information Based on Common Randomness

Abstract

The introduction of the partial information decomposition generated a flurry of proposals for defining an intersection information that quantifies how much of "the same information" two or more random variables specify about a target random variable. As of yet, none is wholly satisfactory. A palatable measure of intersection information would provide a principled way to quantify slippery concepts, such as synergy. Here, we introduce an intersection information measure based on the Gács-Körner common random variable that is the first to satisfy the coveted target monotonicity property. Our measure is imperfect, too, and we suggest directions for improvement.

Quantifying Redundant Information in Predicting a Target Random Variable

“…As discussed in [12], I ∩ measures fail (LP) if and only if they are too strict a measure of redundant information. Loosening the constraints on I ∧ yields I α and achieves a nonnegative decomposition on example IMPERFECTRDN.…”

“…which reduces to a simple expression in [12]. Example IMPERFECTRDN highlights the foremost shortcoming of I ∧ ; I ∧ does not detect "imperfect" or "lossy" correlations between X 1 and X 2 .…”

“…There are a number of intuitive properties, proposed in [5,[9][10][11][12][13], that are considered desirable for the intersection information measure I ∩ to satisfy:…”

“…However, it has been challenging [9][10][11][12] to come up with precise information-theoretic definitions of synergy and redundancy that are consistent with all intuitively desired properties.…”

On Extractable Shared Information

“…Indeed, it is easy to derive Inequality (8) from the identity axiom and from the assumption that SI is left monotonic. Although Inequality (8) may not seem counterintuitive at first sight, none of the information decompositions proposed so far satisfy this property (the function I from [12] satisfies left monotonicity and has been proposed as a measure of shared information, but it does not lead to a nonnegative information decomposition).…”