Strong functional representation lemma and applications to coding theorems

Li, Cheuk Ting; Gamal, Abbas El

doi:10.1109/isit.2017.8006596

“…where • (37) follows from(36) by the definition of the operator half; • (38) follows from (37) by Lemma 6; • (39) follows from (38) by the induction hypotheses (34)-(35) ; • (40) follows from (39) by observing that the components of m (i−j) coincide with the components of the array M output by algorithm MIN-ENTROPY-JOINT-DISTRIBUTION executed on the distributions m (i−j1) and m (j2−j)…”

mentioning

confidence: 99%

Minimum-Entropy Couplings and Their Applications

Cicalese

¹

,

Gargano

²

,

Vaccaro

³

2019

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Given two discrete random variables X and Y, with probability distributions p = (p1, . . . , pn) and q = (q1, . . . , qm), respectively, denote by C(p, q) the set of all couplings of p and q, that is, the set of all bivariate probability distributions that have p and q as marginals. In this paper, we study the problem of finding a joint probability distribution in C(p, q) of minimum entropy (equivalently, a coupling that maximizes the mutual information between X and Y ), and we discuss several situations where the need for this kind of optimization naturally arises. Since the optimization problem is known to be NP-hard, we give an efficient algorithm to find a joint probability distribution in C(p, q) with entropy exceeding the minimum possible at most by 1 bit, thus providing an approximation algorithm with an additive gap of at most 1 bit. Leveraging on this algorithm, we extend our result to the problem of finding a minimum-entropy joint distribution of arbitrary k ≥ 2 discrete random variables X1, . . . , X k , consistent with the known k marginal distributions of the individual random variables X1, . . . , X k . In this case, our algorithm has an additive gap of at most log k from optimum. We also discuss several related applications of our findings and extensions of our results to entropies different from the Shannon entropy.

show abstract

“…This can be considered as a causal scheme on the source sequence (X with rate tuple lR(w) and symbol error probability P e (l). Hence by (9) and (10) in the proof of Theorem 1, R(w) + log (|X | · |Y|) P e (l) · 1 ∈ (1/l)R(X l ; Y l ).…”

Section: Proof Of Theoremmentioning

confidence: 87%

“…The asymmetric private interaction information is the opposite of excess functional information defined in [10] in which I(Y ; U ) is maximized instead. Another operational meaning of G PNI is the generation of random variables with a privacy constraint.…”

Section: Maximal Interaction Information Is Defined Asmentioning

confidence: 99%

“…To achieve R(p 1 ), R(p 2 ), R(p 3 ) and R(p 4 ), apply the causal scheme in Theorem 1 on U ← ∅, U ← X, U ← Y and U ← (X, Y ), respectively. To achieve R(p 5 ), applying the strong functional representation lemma [10], there exists V n ⊥ ⊥ Y n such that H(X n |Y n , V n ) = 0 and I(V n ; Y n |X n ) ≤ ǫn/2 for n large enough. We then apply the causal scheme on X ← X n , Y ← Y n and U ← V n .…”

Section: Proof Of Theoremmentioning

confidence: 99%

“…As a consequence and of particular interest, the extreme points of the rate region R (and its equivalent mutual information region I XY ) for the EGW system include, in addition to the aforementioned extreme points of the Gray-Wyner system, the Körner graph entropy [8], privacy funnel [9] and excess functional information [10], as well as three new quantities with interesting operational meaning, which we refer to as the maximal interaction information, the asymmetric private interaction information, and the symmetric private interaction information. These extreme points can be cast as maximizations of the interaction information [11] I(X; Y |U ) − I(X; Y ) under various constraints.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Extended Gray–Wyner System With Complementary Causal Side Information

Li

¹

,

Gamal

²

2018

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

We establish the rate region of an extended Gray-Wyner system for 2-DMS (X, Y ) with two additional decoders having complementary causal side information. This extension is interesting because in addition to the operationally significant extreme points of the Gray-Wyner rate region, which include Wyner's common information, Gács-Körner common information and information bottleneck, the rate region for the extended system also includes the Körner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest, as extreme points. To simplify the investigation of the 5-dimensional rate region of the extended Gray-Wyner system, we establish an equivalence of this region to a 3-dimensional mutual information region that consists of the set of all triples of the form (I(X; U ), I(Y ; U ), I(X, Y ; U )) for some p U |X,Y . We further show that projections of this mutual information region yield the rate regions for many settings involving a 2-DMS, including lossless source coding with causal side information, distributed channel synthesis, and lossless source coding with a helper. Index TermsGray-Wyner system, side information, complementary delivery, Körner graph entropy, privacy funnel. I. INTRODUCTIONThe lossless Gray-Wyner system [1] is a multi-terminal source coding setting for two discrete memoryless source (2-DMS) (X, Y ) with one encoder and two decoders. This setup draws some of its significance from providing operational interpretation for several information theoretic quantities of interest, namely Wyner's common information [2], the Gács-Körner common information [3], the necessary conditional entropy [4], and the information bottleneck [5].In this paper, we consider an extension of the Gray-Wyner system (henceforth called the EGW system), which includes two new individual descriptions and two decoders with causal side information as depicted in Figure 1. The encoder maps sequences from a 2-DMS (X, Y ) into five indices M i ∈ [1 : 2 nRi ], i = 0, . . . , 4. Decoders 1 and 2 correspond to those of the Gray-Wyner system, that is, decoder 1 recovers X n from (M 0 , M 1 ) and decoder 2 recovers Y n from (M 0 , M 2 ). At time i ∈ [1 : n], decoder 3 recovers X i causally from (M 0 , M 3 , Y i ) and decoder 4 similarly recovers Y i causally from (M 0 , M 4 , X i ). Note that decoders 3 and 4 correspond to those of the complementary delivery setup studied in [6], [7] with causal (instead of noncausal) side information and with two additional private indices M 3 and M 4 . This extended Gray-Wyner system setup is lossless, that is, the decoders recover their respective source sequences with probability of error that vanishes as n approaches infinity. The rate region R of the EGW system is defined in the usual way as the closure of the set of achievable rate tuplesThe first contribution of this paper is to establish the rate region of the EGW system. Moreover, to simplify the study of this rate region and its extreme points, we show that it is equivale...

show abstract

Extended Gray-Wyner system with complementary causal side information

Li

¹

,

Gamal

²

2017

2017 IEEE International Symposium on Information Theory (ISIT)

Self Cite

View full text Add to dashboard Cite

We establish the rate region of an extended Gray-Wyner system for 2-DMS (X, Y ) with two additional decoders having complementary causal side information. This extension is interesting because in addition to the operationally significant extreme points of the Gray-Wyner rate region, which include Wyner's common information, Gács-Körner common information and information bottleneck, the rate region for the extended system also includes the Körner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest, as extreme points. To simplify the investigation of the 5-dimensional rate region of the extended Gray-Wyner system, we establish an equivalence of this region to a 3-dimensional mutual information region that consists of the set of all triples of the form (I(X; U ), I(Y ; U ), I(X, Y ; U )) for some p U |X,Y . We further show that projections of this mutual information region yield the rate regions for many settings involving a 2-DMS, including lossless source coding with causal side information, distributed channel synthesis, and lossless source coding with a helper. Index TermsGray-Wyner system, side information, complementary delivery, Körner graph entropy, privacy funnel. I. INTRODUCTIONThe lossless Gray-Wyner system [1] is a multi-terminal source coding setting for two discrete memoryless source (2-DMS) (X, Y ) with one encoder and two decoders. This setup draws some of its significance from providing operational interpretation for several information theoretic quantities of interest, namely Wyner's common information [2], the Gács-Körner common information [3], the necessary conditional entropy [4], and the information bottleneck [5].In this paper, we consider an extension of the Gray-Wyner system (henceforth called the EGW system), which includes two new individual descriptions and two decoders with causal side information as depicted in Figure 1. The encoder maps sequences from a 2-DMS (X, Y ) into five indices M i ∈ [1 : 2 nRi ], i = 0, . . . , 4. Decoders 1 and 2 correspond to those of the Gray-Wyner system, that is, decoder 1 recovers X n from (M 0 , M 1 ) and decoder 2 recovers Y n from (M 0 , M 2 ). At time i ∈ [1 : n], decoder 3 recovers X i causally from (M 0 , M 3 , Y i ) and decoder 4 similarly recovers Y i causally from (M 0 , M 4 , X i ). Note that decoders 3 and 4 correspond to those of the complementary delivery setup studied in [6], [7] with causal (instead of noncausal) side information and with two additional private indices M 3 and M 4 . This extended Gray-Wyner system setup is lossless, that is, the decoders recover their respective source sequences with probability of error that vanishes as n approaches infinity. The rate region R of the EGW system is defined in the usual way as the closure of the set of achievable rate tuplesThe first contribution of this paper is to establish the rate region of the EGW system. Moreover, to simplify the study of this rate region and its extreme points, we show that it is equivale...

show abstract

Strong functional representation lemma and applications to coding theorems

Cited by 23 publications

References 26 publications

Minimum-Entropy Couplings and Their Applications

Minimum-Entropy Couplings and Their Applications

Extended Gray–Wyner System With Complementary Causal Side Information

Extended Gray-Wyner system with complementary causal side information

Contact Info

Product

Resources

About