Differentially Private Accelerated Optimization Algorithms

Kuru, Nurdan; Birbil, Ş. İlker; Gürbüzbalaban, Mert; Yıldırım, Sinan

doi:10.1137/20m1355847

Cited by 9 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For further reducing computational cost, applying variance reduction technique (Johnson & Zhang, 2013;Defazio et al, 2014;Nguyen et al, 2017;Cutkosky & Orabona, 2019) has been studied (Wang et al, 2019c;Bassily et al, 2019;Asi et al, 2021). Also, Kuru et al (2022) has combined DP-GD with Nesterov's acceleration to improve the computational efficiency. On the other side, several works have developed more communication efficient DP algorithms than DP-GD in distributed learning.…”

Section: Related Workmentioning

confidence: 99%

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

Murata¹,

Suzuki²

2023

Preprint

View full text Add to dashboard Cite

Differential private optimization for nonconvex smooth objective is considered. In the previous work, the best known utility bound is O( √ d/(nε DP )) in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where n is the sample size, d is the problem dimensionality and ε DP is the differential privacy parameter. To improve the best known utility bound, we propose a new differential private optimization framework called DIFF2 (DIFFerential private optimization via gradient DIFFerences) that constructs a differential private global gradient estimator with possibly quite small variance based on communicated gradient differences rather than gradients themselves. It is shown that DIFF2 with a gradient descent subroutine achieves the utility of O(d 2/3 /(nε DP ) 4/3 ), which can be significantly better than the previous one in terms of the dependence on the sample size n. To the best of our knowledge, this is the first fundamental result to improve the standard utility O( √ d/(nε DP )) for nonconvex objectives. Additionally, a more computational and communication efficient subroutine is combined with DIFF2 and its theoretical analysis is also given. Numerical experiments are conducted to validate the superiority of DIFF2 framework.

show abstract

Section: Related Workmentioning

confidence: 99%

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

Murata¹,

Suzuki²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…[1] proposes a differentially private stochastic gradient descent algorithm DPSGD and presents the idea of moment accounting.. Subsequently, it has been studied in depth in a series of works ( [2], [3], [4], [5]) applying DP to other optimisation algorithms such as DP-AdaGrad, DP-SVRG, and ApolySFW. From a theoretical point of view, [6] and [7] analysed the clipping SGD of convergence.…”

Section: Related Workmentioning

confidence: 99%

Adaptive differential private in deep learning

Hao¹,

Lu²,

Luo³

2023

International Conference on Cloud Computing, Performance Computing, and Deep Learning (CCPCDL 2023)

View full text Add to dashboard Cite

“…Many works in the data privacy literature do not mainly focus on regression but are motivated by or can be applied to regression. As an example, differentially private empirical risk minimisation (Chaudhuri et al, 2009;Bassily et al, 2014;Abadi et al, 2016;Kuru et al, 2022) can be applied to regression once it is cast as a data-driven optimisation problem. Many general-purpose Bayesian differentially private estimation methods can also be used in regression problems.…”

Section: Introductionmentioning

confidence: 99%

Differentially Private Distributed Bayesian Linear Regression with MCMC

Alparslan¹,

Yıldırım²,

Birbil³

2023

Preprint

View full text Add to dashboard Cite

We propose a novel Bayesian inference framework for distributed differentially private linear regression. We consider a distributed setting where multiple parties hold parts of the data and share certain summary statistics of their portions in privacy-preserving noise. We develop a novel generative statistical model for privately shared statistics, which exploits a useful distributional relation between the summary statistics of linear regression. Bayesian estimation of the regression coefficients is conducted mainly using Markov chain Monte Carlo algorithms, while we also provide a fast version to perform Bayesian estimation in one iteration. The proposed methods have computational advantages over their competitors. We provide numerical results on both real and simulated data, which demonstrate that the proposed algorithms provide well-rounded estimation and prediction.

show abstract

Differentially Private Accelerated Optimization Algorithms

Cited by 9 publications

References 12 publications

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

Adaptive differential private in deep learning

Differentially Private Distributed Bayesian Linear Regression with MCMC

Contact Info

Product

Resources

About