Efficient Privacy-Preserving Stochastic Nonconvex Optimization

Abstract

While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains under challenging. In this paper, we study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. To achieve both efficiency and strong privacy guarantees with efficiency, we propose a differentially-private stochastic gradient descent algorithm for nonconvex ERM, and provide a tight analysis of its privacy and utility guarantees, as well as its gradient complexity. We show that our proposed algorithm can substantially reduce gradient complexity while matching the best-known utility guarantee obtained by Wang et al. (2017). We extend our algorithm to the distributed setting using secure multi-party computation, and show that it is possible for a distributed algorithm to match the privacy and utility guarantees of a centralized algorithm in this setting. Our experiments on benchmark nonconvex ERM problems and real datasets demonstrate superior performance in terms of both training time and utility gains compared with previous differentially-private methods using the same privacy budgets.

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

“…Algorithms such as output-perturbation that perturbs the output of a non-DP algorithm, objective function perturbation that perturbs the objective function [7] and gradient perturbation that adds noise to the gradient in gradient descent algorithms [4,28] have been proposed to solve DP-ERM. We mainly discuss those algorithms that are most related to our problem, i.e., gradient perturbation [2,4,6,30,31,32,37]. Most DP gradient-based algorithms focus on minimizing the convex loss and aim to achieve optimal empirical and population risk bounds under privacy.…”

“…Recently, DP algorithms have been studied for non-convex loss functions [30,31,32,37]. Since finding the global minimum for non-convex functions is NP-hard, the utility of a DP algorithm is typically measured by the 2 -norm of the gradient.…”

“…A natural approach toward solving the problem stated in (1) is Differentially Private Empirical Risk Minimization (DP-ERM) [4,31,32,34], which finds w priv by minimizing the empirical risk:…”

“…For DP-ERM with non-convex loss function, the utility of the private minimizer w priv is usually measured by the 2 norm of the empirical gradient, i.e., ∇ f (w priv ) [30,31,37]. Recent work [30,31,32] solve non-convex DP-ERM by DP gradient descent and DP stochastic variance reduced gradient (SVRG) and provide O 4 √ p/ √ n bound on the 2 norm of the empirical gradient over p-dimensional space. Built on the empirical risk results, the standard approach for deriving bounds on the population loss is the uniform convergence of the empirical gradient to the population gradient, namely an upper bound on sup w ∇f (w) − ∇ f (w) .…”

Differentially Private ADMM Algorithms for Machine Learning

“…where each f i : R d → R is a smooth convex loss function, and g : R d → R is a simple convex (non)-smooth regularizer such as the 1 -norm or 2 -norm regularizer. Many differential privacy algorithms have been proposed to deal with ERM problems, such as DP-SGD [18], DP-SVRG [19], and DP-SRGD [20]. Therefore, this paper mainly considers the generalized ERM problem with more complex regularizers (e.g., g(x) = λ Ax 1 with a given matrix A and a regularized parameter λ > 0), such as graph-guided fused Lasso [21], generalized Lasso [22] and graph-guided support vector machine (SVM) [23].…”