Wyner-Ziv compression is (almost) optimal for distributed optimization

Fundamental Limits of Distributed Optimization over Multiple Access Channel

“…where for i ∈ [d], Y (i) satisfies ( 7) and the expectation is taken over all the randomness in the set up. We now recall a lemma from [14] that upper bounds the convergence rate of Algorithm…”

“…b) The Wyner-Ziv estimate: The clients in C 2 use a Wyner-Ziv estimator boosted DAQ from [14] to construct the final estimate, while those in C 1 tranmit 0 in all channel uses. The boosted DAQ uses the idea of correlated sampling between the input and the side information to reduce quantization error.…”

“…For our lower bounds, we use affine functions as difficult functions which are 0-smooth and are admissible in the class of L-smooth functions. These functions are considered in the same spirit as in showing the lower bounds for convex, smooth optimization under communication constraints [14]. We consider the domain X {x ∈ R d : x ∞ ≤ D/(2 √ d}, and consider the following class of functions on X : For v ∈ {−1, 1} d , let The parameter δ > 0 is to be chosen later.…”

“…where the first equality uses the unitary property of R, the second uses the fact that boosted DAQ yields an unbiased estimate (see Lemma VI.1), the first inequality follows from Jensen's inequality, and the last line uses the bound in (14). Finally, we set δ = c 2 K 2 N .…”

“…Our work is closely related to [13] and [14]. [13], too, studies fundamental limits of over-theair optimization, but they do so in the single client setting and when the communication channel is the more straightforward additive Gaussian noise channel.…”