Recycled ADMM: Improving the Privacy and Accuracy of Distributed Algorithms

Abstract

Alternating direction method of multiplier (ADMM) is a powerful method to solve decentralized convex optimization problems. In distributed settings, each node performs computation with its local data and the local results are exchanged among neighboring nodes in an iterative fashion. During this iterative process the leakage of data privacy arises and can accumulate significantly over many iterations, making it difficult to balance the privacy-accuracy tradeoff. We propose Recycled ADMM (R-ADMM), where a linear approximation is applied to every even iteration, its solution directly calculated using only results from the previous, odd iteration. It turns out that under such a scheme, half of the updates incur no privacy loss and require much less computation compared to the conventional ADMM. Moreover, R-ADMM can be further modified (MR-ADMM) such that each node independently determines its own penalty parameter over iterations. We obtain a sufficient condition for the convergence of both algorithms and provide the privacy analysis based on objective perturbation. It can be shown that the privacyaccuracy tradeoff can be improved significantly compared with conventional ADMM.

DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

“…Therefore, distributed machine learning helps to reduce computational burden and improves both robustness and scalability of data processing. As pointed out in recent studies [1], [2], existing approaches to decentralizing an optimization problem mainly consist of subgradient-based algorithms [3], [4], alternating direction method of multipliers (ADMM) based algorithms [5]- [8], and composite of sub-gradient descent and ADMM [9]. It has been shown that ADMM-based algorithms can converge at the rate of O(1/t) while subgradient-based algorithms typically converge at the rate of O(1/ √ t), where t is the number of iterations [10].…”

Recycled ADMM: Improve Privacy and Accuracy with Less Computation in Distributed Algorithms

“…D. Private ADMM [26] & Private M-ADMM [27] In private ADMM [26], the noise is added either to the updated primal variable before broadcasting to its neighbors (primal variable perturbation), or to the dual variable before updating its primal variable using (8) (dual variable perturbation). The privacy property is only evaluated for a single node and a single iteration, both methods cannot balance the privacy-utility tradeoff very well if consider the total privacy loss.…”

“…Following the same pre-processing steps as in [27], the final data includes 45,223 individuals, each represented as a 105-dimensional vector of norm at most 1. We will use as loss function the logistic loss L (z) = log(1 + exp(−z)), with |L | ≤ 1 and L ≤ c 1 = 1 4 .…”

“…We next inspect the accuracy and privacy of the private R-ADMM (Algorithm 2) and compare it with the private (conventional) ADMM using dual variable perturbation (DVP) [26] and the private M-ADMM using penalty perturbation (PP) [27]. In the set of experiments, we fix γ = 0.2, η = 1 in private R-ADMM and set the noise parameter α i (k) = α, ∀i, k. The noise parameters of conventional ADMM and M-ADMM are also chosen respectively such that they have almost the same total privacy loss bounds.…”

Rényi Differentially Private ADMM for Non-Smooth Regularized Optimization

“…In addition to its original version, many variations has been presented, such as [23,24] and [12]. Several ADMM based differentially private algorithms have been presented, for example, [25] applied objective perturbation technique on the original ADMM problem, [26] and [27] applied output and objective perturbation technique, and [28] applied gradient perturbation technique on ADMM-based algorithms in distributed settings.…”