Communication lower bounds for statistical estimation problems via a distributed data processing inequality

Abstract

We study the tradeoff between the statistical error and communication cost of distributed statistical estimation problems in high dimensions. In the distributed sparse Gaussian mean estimation problem, each of the m machines receives n data points from a d-dimensional Gaussian distribution with unknown mean θ which is promised to be k-sparse. The machines communicate by message passing and aim to estimate the mean θ. We provide a tight (up to logarithmic factors) tradeoff between the estimation error and the number of bits communicated between the machines. This directly leads to a lower bound for the distributed sparse linear regression problem: to achieve the statistical minimax error, the total communication is at least Ω(min{n, d}m), where n is the number of observations that each machine receives and d is the ambient dimension. These lower results improve upon [Sha14, SD15] by allowing multi-round iterative communication model. We also give the first optimal simultaneous protocol in the dense case for mean estimation.As our main technique, we prove a distributed data processing inequality, as a generalization of usual data processing inequalities, which might be of independent interest and useful for other problems.

The Role of Interactivity in Local Differential Privacy

“…More recently, Duchi and Rogers [16] showed how to combine the above analogue of Assouad's method with techniques from information complexity [8,22] to prove lower bounds for estimation problems that apply to a restricted class of fully interactive locally private protocols. A corollary of their lower bounds is that several known noninteractive algorithms are optimal minimax estimators within the class they consider.…”

“…Let h 2 denote the square of the Hellinger distance, h 2 (f, g) = 1 − X f (x)g(x)dx. We begin with Lemma 5.4, originally proven as Lemma 2 in Braverman et al [8].…”

Distributed Estimation of Gaussian Correlations

“…Xu and Raginsky [16] provided lower bounds on the risk in a distributed Bayesian estimation setting with noisy channels between the data collection terminals and the estimation entity. Braverman et al [17] provided lower bounds for some high dimensional distributed estimation problems, again when the samples of all terminals are from the same distribution, e.g. for distributed estimation of the multivariate Guassian mean when it is known to be sparse.…”

Randomized Distributed Mean Estimation: Accuracy vs. Communication

“…The distributed mean estimation problem was recently studied in a statistical framework where it is assumed that the vectors X i are independent and identicaly distributed samples from some specific underlying distribution. In such a setup, the goal is to estimate the true mean of the underlying distribution [14,13,2,1]. These works formulate lower and upper bounds on the communication cost needed to achieve the minimax optimal estimation error.…”