Differential privacy and robust statistics in high dimensions

Abstract

We introduce a universal framework for characterizing the statistical efficiency of a statistical estimation problem with differential privacy guarantees. Our framework, which we call High-dimensional Propose-Test-Release (HPTR), builds upon three crucial components: the exponential mechanism from [MT07], robust statistics, and the Propose-Test-Release mechanism from [DL09]. Gluing all these together is the concept of resilience, which is central to robust statistical estimation. Resilience guides the design of the algorithm, the sensitivity analysis, and the success probability analysis of the test step in Propose-Test-Release. The key insight is that if we design an exponential mechanism that accesses the data only via one-dimensional robust statistics, then the resulting local sensitivity can be dramatically reduced. Using resilience, we can provide tight local sensitivity bounds. These tight bounds readily translate into near-optimal utility guarantees in several cases. We give a general recipe for applying HPTR to a given instance of a statistical estimation problem and demonstrate it on canonical problems of mean estimation, linear regression, covariance estimation, and principal component analysis. We introduce a general utility analysis technique that proves that HPTR nearly achieves the optimal sample complexity under several scenarios studied in the literature.

DP-PCA: Statistically Optimal and Differentially Private PCA

“…Assumption A.4 can be relaxed to heavy-tail bounds with bounded k-th moment on A i , in which case we expect the second term in Eq. ( 10) to scale as O(d( log(1/δ)/εn) 1−1/k ), drawing analogy from a similar trend in a computationally inefficient DP-PCA without spectral gap [56,Corollary 6.10]. When a fraction of data is corrupted, recent advances in [74,51,40] provide optimal algorithms for PCA.…”

“…When a fraction of data is corrupted, recent advances in [74,51,40] provide optimal algorithms for PCA. However, existing approach of [56] for robust and private PCA is computationally intractable. Borrowing ideas from robust and private mean estimation in [55], one can design an efficient algorithm, but at the cost of sub-optimal sample complexity.…”

“…One caveat is that the lower bound assumes pure-DP with δ = 0. We do not yet have a lower bound technique for approximate DP that is tight, and all known approximate DP lower bounds have gaps to achievable upper bounds in its dependence in log(1/δ), e.g., [6,56]. We provide a proof in Appendix C.1.…”

“…[49] studied private mean estimation from Gaussian sample, and obtained an optimal error rate. There has been a lot of recent interests on private mean estimation under various assumptions, including mean and covariance joint estimation [43,9], heavy-tailed mean estimation [47], mean estimation for general distributions [31,66], distribution adaptive mean estimation [13], estimation for unbounded distribution parameters [46], mean estimation under pure differential privacy [37], local differential privacy [19,20,33,41,54], user-level differential privacy [27,27], Mahalanobis distance [11,56] and robust and differentially private mean estimation [55,52,56].…”

Efficient Mean Estimation with Pure Differential Privacy via a Sum-of-Squares Exponential Mechanism

“…However, none of their results give computationally efficient estimators for pure DP. A simultaneous and independent work of [LKO21] demonstrates an interesting connection between resilience [SCV18] and private estimation. They exploit this connection to design robust and private algorithms for a variety of settings, including mean estimation, covariance estimation, PCA, and more.…”