Privacy-Preserving Parametric Inference: A Case for Robust Statistics

Abstract

Differential privacy is a cryptographically-motivated approach to privacy that has become a very active field of research over the last decade in theoretical computer science and machine learning. In this paradigm one assumes there is a trusted curator who holds the data of individuals in a database and the goal of privacy is to simultaneously protect individual data while allowing the release of global characteristics of the database. In this setting we introduce a general framework for parametric inference with differential privacy guarantees. We first obtain differentially private estimators based on bounded influence M-estimators by leveraging their gross-error sensitivity in the calibration of a noise term added to them in order to ensure privacy. We then show how a similar construction can also be applied to construct differentially private test statistics analogous to the Wald, score and likelihood ratio tests. We provide statistical guarantees for all our proposals via an asymptotic analysis. An interesting consequence of our results is to further clarify the connection between differential privacy and robust statistics. In particular, we demonstrate that differential privacy is a weaker stability requirement than infinitesimal robustness, and show that robust M-estimators can be easily randomized in order to guarantee both differential privacy and robustness towards the presence of contaminated data. We illustrate our results both on simulated and real data.

High-Dimensional Differentially-Private EM Algorithm: Methods and Near-Optimal Statistical Guarantees

“…. Then we find that actually ||W t || 2 2 /σ 2 follows a chi-square distribution χ 2 (d). By the concentration of chi-square distribution, there exists constants c 0 , c 1 , c 2 , such that:…”

“…The trade-off between statistical accuracy and privacy is one of the fundamental topics in differential privacy. In the low-dimensional setting, there are various works focusing on this trade-off, including mean estimation [28,52,8,34,10,36], confidence intervals of Gaussian mean [37] and binomial mean [4], linear regression [55,10], generalized linear models [51,11,50], principal component analysis [30,12], convex empirical risk minimization [7], and robust M-estimators [2,3].…”

Differential privacy and robust statistics in high dimensions

“…Note that, in general, computing smooth sensitivity is also computationally inefficient with an exception of [AM21]. Using smooth sensitivity, [Lei11, Smi11, CH12, AM21] leverage robust Mestimators for differentially private estimation and inference.…”

Perturbed M-Estimation: A Further Investigation of Robust Statistics for Differential Privacy

“…For example, after defining the above robustness measures, Chaudhuri and Hsu (2012) use the notion of GES to deliver convergence rates for differentially private statistical estimators while Avella-Medina (2019) uses this measure to calibrate the additive noise to deliver differentially private M-Estimators. In the next sections we explore another approach, suggested but not studied in Chaudhuri and Hsu (2012) and Avella-Medina (2019), where we investigate the use of bounded M-Estimation for differentially private estimation and prediction using the OPM. While empirical risk minimization, that objective perturbation is built on, can be classified as M-Estimation, it is not straightforward to integrate the standard bounded functions for M-estimation as is.…”