Spreading of Sets in Product Spaces and Hypercontraction of the Markov Operator

Ahlswede, Rudolf; Gács, Péter

doi:10.1214/aop/1176995937

Cited by 151 publications

(275 citation statements)

References 0 publications

Supporting

Mentioning

268

Contrasting

Order By: Relevance

“…Perhaps worth noting that similar intuition can be drawn for the two-party gap hamming distance problem but there we don't have a completely information theoretic proof of the linear lower bound [CR11]. We will be using the strong data processing inequality for the binary symmetric channel first proven by [AG76]. it studies how information decays on a binary symmetric channel.…”

Section: Distributed Gap Majoritymentioning

confidence: 99%

Communication lower bounds for statistical estimation problems via a distributed data processing inequality

Braverman

Garg

et al. 2016

Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing

111

125

View full text Add to dashboard Cite

We study the tradeoff between the statistical error and communication cost of distributed statistical estimation problems in high dimensions. In the distributed sparse Gaussian mean estimation problem, each of the m machines receives n data points from a d-dimensional Gaussian distribution with unknown mean θ which is promised to be k-sparse. The machines communicate by message passing and aim to estimate the mean θ. We provide a tight (up to logarithmic factors) tradeoff between the estimation error and the number of bits communicated between the machines. This directly leads to a lower bound for the distributed sparse linear regression problem: to achieve the statistical minimax error, the total communication is at least Ω(min{n, d}m), where n is the number of observations that each machine receives and d is the ambient dimension. These lower results improve upon [Sha14, SD15] by allowing multi-round iterative communication model. We also give the first optimal simultaneous protocol in the dense case for mean estimation.As our main technique, we prove a distributed data processing inequality, as a generalization of usual data processing inequalities, which might be of independent interest and useful for other problems.

show abstract

Section: Distributed Gap Majoritymentioning

confidence: 99%

Communication lower bounds for statistical estimation problems via a distributed data processing inequality

Braverman

Garg

et al. 2016

Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing

111

125

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 90%

“…Such inequalities have apparently been first discovered by Ahlswede and Gács in a landmark paper [AG76]. Among the work predating [AG76] and extending it we mention [Dob56,Sar62,CIR + 93]. Notable connections include topics ranging from existence and uniqueness of Gibbs measures and log-Sobolev inequalities to performance limits of noisy circuits.…”

Section: Introductionmentioning

confidence: 99%

Strong Data Processing Inequalities for Input Constrained Additive Noise Channels

Calmon

Polyanskiy

2018

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

This paper quantifies the intuitive observation that adding noise reduces available information by means of non-linear strong data processing inequalities. Consider the random variables W → X → Y forming a Markov chain, where Y = X +Z with X and Z real-valued, independent and X bounded in L p -norm. It is shown that I(W ; Y ) ≤ F I (I(W ; X)) with F I (t) < t whenever t > 0, if and only if Z has a density whose support is not disjoint from any translate of itself.A related question is to characterize for what couplings (W, X) the mutual information I(W ; Y ) is close to maximum possible. To that end we show that in order to saturate the channel, i.e. for I(W ; Y ) to approach capacity, it is mandatory that I(W ; X) → ∞ (under suitable conditions on the channel). A key ingredient for this result is a deconvolution lemma which shows that post-convolution total variation distance bounds the pre-convolution KolmogorovSmirnov distance.Explicit bounds are provided for the special case of the additive Gaussian noise channel with quadratic cost constraint. These bounds are shown to be order-optimal. For this case simplified proofs are provided leveraging Gaussian-specific tools such as the connection between information and estimation (I-MMSE) and Talagrand's information-transportation inequality.

show abstract

“…It is shown in [33] that its slope is given by the chordal slope of the hypercontractivity of Markov operator [34] …”

Section: Necessary Conditional Entropymentioning

confidence: 99%

Extended Gray-Wyner system with complementary causal side information

Gamal

2017

2017 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We establish the rate region of an extended Gray-Wyner system for 2-DMS (X, Y ) with two additional decoders having complementary causal side information. This extension is interesting because in addition to the operationally significant extreme points of the Gray-Wyner rate region, which include Wyner's common information, Gács-Körner common information and information bottleneck, the rate region for the extended system also includes the Körner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest, as extreme points. To simplify the investigation of the 5-dimensional rate region of the extended Gray-Wyner system, we establish an equivalence of this region to a 3-dimensional mutual information region that consists of the set of all triples of the form (I(X; U ), I(Y ; U ), I(X, Y ; U )) for some p U |X,Y . We further show that projections of this mutual information region yield the rate regions for many settings involving a 2-DMS, including lossless source coding with causal side information, distributed channel synthesis, and lossless source coding with a helper. Index TermsGray-Wyner system, side information, complementary delivery, Körner graph entropy, privacy funnel. I. INTRODUCTIONThe lossless Gray-Wyner system [1] is a multi-terminal source coding setting for two discrete memoryless source (2-DMS) (X, Y ) with one encoder and two decoders. This setup draws some of its significance from providing operational interpretation for several information theoretic quantities of interest, namely Wyner's common information [2], the Gács-Körner common information [3], the necessary conditional entropy [4], and the information bottleneck [5].In this paper, we consider an extension of the Gray-Wyner system (henceforth called the EGW system), which includes two new individual descriptions and two decoders with causal side information as depicted in Figure 1. The encoder maps sequences from a 2-DMS (X, Y ) into five indices M i ∈ [1 : 2 nRi ], i = 0, . . . , 4. Decoders 1 and 2 correspond to those of the Gray-Wyner system, that is, decoder 1 recovers X n from (M 0 , M 1 ) and decoder 2 recovers Y n from (M 0 , M 2 ). At time i ∈ [1 : n], decoder 3 recovers X i causally from (M 0 , M 3 , Y i ) and decoder 4 similarly recovers Y i causally from (M 0 , M 4 , X i ). Note that decoders 3 and 4 correspond to those of the complementary delivery setup studied in [6], [7] with causal (instead of noncausal) side information and with two additional private indices M 3 and M 4 . This extended Gray-Wyner system setup is lossless, that is, the decoders recover their respective source sequences with probability of error that vanishes as n approaches infinity. The rate region R of the EGW system is defined in the usual way as the closure of the set of achievable rate tuplesThe first contribution of this paper is to establish the rate region of the EGW system. Moreover, to simplify the study of this rate region and its extreme points, we show that it is equivale...

show abstract

Spreading of Sets in Product Spaces and Hypercontraction of the Markov Operator

Cited by 151 publications

References 0 publications

Communication lower bounds for statistical estimation problems via a distributed data processing inequality

Communication lower bounds for statistical estimation problems via a distributed data processing inequality

Strong Data Processing Inequalities for Input Constrained Additive Noise Channels

Extended Gray-Wyner system with complementary causal side information

Contact Info

Product

Resources

About