Exact common information

Abstract

This paper introduces the notion of exact common information, which is the minimum description length of the common randomness needed for the exact distributed generation of two correlated random variables (X, Y ). We introduce the quantity G(X; Y ) = minX→W →Y H(W ) as a natural bound on the exact common information and study its properties and computation. We then introduce the exact common information rate, which is the minimum description rate of the common randomness for the exact generation of a 2-DMS (X, Y ). We give a multiletter characterization for it as the limit G(X; Y ) = limn→∞(1/n)G(X n ; Y n ). While in general G(X; Y ) is greater than or equal to the Wyner common information, we show that they are equal for the Symmetric Binary Erasure Source. We do not know, however, if the exact common information rate has a single letter characterization in general.

Distributed Simulation of Continuous Random Variables

“…, X n ) follows a prescribed distribution exactly. The distributed randomness simulation problem is to find the common randomness W * with the minimum average description length R * , referred to in [1] as the exact common information between X n , and the scheme that achieves this exact common information. Since W can be represented by an optimal prefix-free code, e.g., a Huffman code or the code in [2] if the alphabet is infinite, the average description length R * can be upper bounded as H(W ) ≤ R * < H(W ) + 1, whereW minimizes H(W ).…”

“…Hence in this paper we will focus on investigating W that minimizes H(W ) instead of R * . The above setting was introduced in [1] for two discrete random variables and the minimum entropy of W , referred to as the common entropy, is given by G(X 1 ; X 2 ) = min…”

“…[3]. Computing G(X 1 ; X 2 ), even for moderate size random variable alphabets, can be computationally difficult since it involves minimizing a concave function over a non-convex set; see [1] for some cases where G can be computed and for some properties that can be exploited to compute it. Hence the main difficulty in constructing a scheme that achieves G (within 1-bit) for a given (X 1 , X 2 ) distribution is finding the optimal common randomness W that achieves it.…”

Quantifying Redundant Information in Predicting a Target Random Variable

“…In these settings, we are concerned with the statistics of how two (or more) random variables X 1 , X 2 , called predictors, jointly or separately specify/predict another random variable Y , called a target random variable. This focus on a target random variable is in contrast to Shannon's mutual information which quantifies statistical dependence between two random variables, and various notions of common information, e.g., [6][7][8].…”

dit: a Python package for discrete information theory

“…dit implements the vast majority of information measure defined in the literature, including entropies (Shannon (Cover and Thomas 2006), Renyi, Tsallis), multivariate mutual informations (co-information (Bell 2003) (McGill 1954), total correlation (Watanabe 1960), dual total correlation(Te Sun 1980) (Han 1975) (Abdallah and Plumbley 2012), CAEKL mutual information (Chan et al 2015)), common informations (Gács-Körner(Gács and Körner 1973) (Tyagi, Narayan, and Gupta 2011), Wyner (Wyner 1975)(W. Liu, Xu, and Chen 2010), exact (Kumar, Li, and El Gamal 2014), functional, minimal sufficient statistic), and channel capacity (Cover and Thomas 2006). It includes methods of studying joint distributions including information diagrams, connected informations (Schneidman et al 2003) (Amari 2001), marginal utility of information (Allen, Stacey, and Bar-Yam 2017), and the complexity profile(Y.…”