Differential privacy for sparse classification learning

Abstract

In this paper, we present a differential privacy version of convex and nonconvex sparse classification approach. Based on alternating direction method of multiplier (ADMM) algorithm, we transform the solving of sparse problem into the multistep iteration process. Then we add exponential noise to stable steps to achieve privacy protection. By the property of the post-processing holding of differential privacy, the proposed approach satisfies the −differential privacy even when the original problem is unstable. Furthermore, we present the theoretical privacy bound of the differential privacy classification algorithm. Specifically, the privacy bound of our algorithm is controlled by the algorithm iteration number, the privacy parameter, the parameter of loss function, ADMM pre-selected parameter, and the data size. Finally we apply our framework to logistic regression with L1 regularizer and logistic regression with L 1/2 regularizer. Numerical studies demonstrate that our method is both effective and efficient which performs well in sensitive data analysis.

Rényi Differentially Private ADMM for Non-Smooth Regularized Optimization

“…In addition to its original version, many variations has been presented, such as [23,24] and [12]. Several ADMM based differentially private algorithms have been presented, for example, [25] applied objective perturbation technique on the original ADMM problem, [26] and [27] applied output and objective perturbation technique, and [28] applied gradient perturbation technique on ADMM-based algorithms in distributed settings.…”

“…[35] and [36] has shown that L 1 regularized classification has good performance in feature selection. Limited to the assumption on the loss function, many differentially private ERM algorithms cannot be directly applied on L 1 regularized classification, with a few exceptions such as [4,25], and [28].…”

“…Many differentially private ERM algorithms cannot be applied to L 1 regularized classification, such as ObjPert [1], [2], OutPert [40], PVP and DVP [5], PSGD [16], and RSGD [7]. Therefore, we compare our proposed algorithms with these baselines: DP-SGD [4], DP-ADMM [25], ADMM-objP [28], and Non-Private approach.…”

“…To test the performance on feature selection, we created a synthetic dataset with many irrelevant features, using similar strategy as in [25]. To be specific, we generate a 100-dimension data s i ∼ N (0 100 , Σ) where Σ i,j = 0.5 |i−j| .…”

“…Our method differed from theirs for the following aspects: (i) DP-ADMM is used for distributed learning, so that the training objective is assigned into multiple parties each holding a portion of the data, instead in ssADMM it is the data dependent loss and regularization that are separated; (ii) in DP-ADMM, each party is perturbing full gradient and transmit to the center, so that there is no privacy amplification effect, therefore although both algorithms solve optimization approximately, their privacy loss is higher than ours at each step. Our methods differ from the ADMM-objP method (DPLL in [25]) for the following aspect: (i) ADMM-objP perturb the training objective at each iteration, and use full gradient descent multiple times to acquire exact solution at each iteration, which is not as efficient as ours, since our method only access a portion of data once at each step; (ii) ADMM-objP guarantees privacy only if exact solution is acquired at each step, therefore the privacy guarantee is only theoretically true. The privacy guarantee of ssADMM is given by Theorem 1.…”