Shuffle Private Stochastic Convex Optimization

Cheu, Albert; Joseph, Matthew; Mao, Jieming; Peng, Baoxiang

doi:10.48550/arxiv.2106.09805

Cited by 5 publications

(20 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The algorithm returns a uniformly random iterate from tw t r`1 u r"0,¨¨¨,E´1;t"0,¨¨¨Q´1 . SDP Prox-SVRG (Algorithm 7 in Appendix D.4) follows the same structure, but with Gaussian noise replaced by the protocol of [CJMP21].…”

Section: Noisy Distributed Prox-pl-svrg For Federated Ermmentioning

confidence: 99%

“…In this section, we recall the shuffle private vector summation protocol P vec of [CJMP21], and its privacy and utility guarantee. As our first building block, we will need the scalar summation protocol, Algorithm 4.…”

Section: Shuffle Privacy Building Blocksmentioning

confidence: 99%

“…In the SDP versions of our algorithms, clients and shuffler use the private vector summation protocol, P vec , of [CJMP21]: see Appendix C for details. The idea of P vec is: in each round, clients send binomial-noised, discretized stochastic gradients to the shuffler; the shuffler randomly permutes these noisy gradients, concealing client identities and amplifying privacy; the server aggregates and re-scales the shuffled noisy gradients, and updates the model.…”

mentioning

confidence: 99%

“…The idea of P vec is: in each round, clients send binomial-noised, discretized stochastic gradients to the shuffler; the shuffler randomly permutes these noisy gradients, concealing client identities and amplifying privacy; the server aggregates and re-scales the shuffled noisy gradients, and updates the model. In Section 3, we will usually describe the LDP variation of each algorithm in more detail and defer the SDP version to the appendix; however, the only difference between the two algorithms is that the Gaussian mechanism is replaced with the protocol of [CJMP21].…”

mentioning

confidence: 99%

“…Privacy of the LDP/SDP proximal gradient methods follows from parallel composition [McS09] and the privacy guarantees of the Gaussian mechanism/vector summation protocol [DR14,CJMP21], together with the post-processing property [DR14]. The main idea of the excess loss proofs is to view each noisy proximal evaluation (line 9 in Algorithm 1) as an execution of objective perturbation [CMS11].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Private Non-Convex Federated Learning Without a Trusted Server

Lowy¹,

Ghafelebashi²,

Razaviyayn³

2022

Preprint

View full text Add to dashboard Cite

We study differentially private (DP) federated learning (FL) with non-convex loss functions and heterogeneous (non-i.i.d.) client data in the absence of a trusted server, both with and without a secure "shuffler" to anonymize client reports. We propose novel algorithms that satisfy local differential privacy (LDP) at the client level and shuffle differential privacy (SDP) for three classes of Lipschitz continuous loss functions: First, we consider losses satisfying the Proximal Polyak-Łojasiewicz (PL) inequality, which is an extension of the classical PL condition to the constrained setting. Prior works studying DP PL optimization only consider the unconstrained problem with Lipschitz loss functions, which rules out many interesting practical losses, such as strongly convex, least squares, and regularized logistic regression. However, by analyzing the proximal PL scenario, we permit such losses which are Lipschitz on a restricted parameter domain. We propose LDP and SDP algorithms that nearly attain the optimal strongly convex, homogeneous (i.i.d.) rates. Second, we provide the first DP algorithms for non-convex/non-smooth loss functions. Third, we specialize our analysis to smooth, unconstrained non-convex FL. Our bounds improve on the state-of-the-art, even in the special case of a single client, and match the non-private lower bound in certain practical parameter regimes. Numerical experiments show that our algorithm yields better accuracy than baselines for most privacy levels.

show abstract

Section: Noisy Distributed Prox-pl-svrg For Federated Ermmentioning

confidence: 99%

Section: Shuffle Privacy Building Blocksmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Private Non-Convex Federated Learning Without a Trusted Server

Lowy¹,

Ghafelebashi²,

Razaviyayn³

2022

Preprint

View full text Add to dashboard Cite

show abstract

(Private) Kernelized Bandits with Distributed Biased Feedback

Zhou

2023

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

We study kernelized bandits with distributed biased feedback. This problem is motivated by several real-world applications (such as dynamic pricing, cellular network configuration, and policy making), where users from a large population contribute to the reward of the action chosen by a central entity, but it is difficult to collect feedback from all users. Instead, only biased feedback (due to user heterogeneity) from a subset of users may be available. In addition to such biased feedback, we are also faced with two practical challenges due to communication cost and computation complexity. To tackle these challenges, we carefully design a new distributed phase-then-batch-based elimination (DPBE) algorithm, which samples users in phases for collecting feedback to reduce the bias and employs maximum variance reduction to select actions in batches within each phase. By properly choosing the phase length, the batch size, and the confidence width used for eliminating suboptimal actions, we show that DPBE achieves a sublinear regret of Õ(T1-α/2 +γTT), where α∈ (0,1) is the user-sampling parameter one can tune. Moreover, DPBE can significantly reduce both communication cost and computation complexity in distributed kernelized bandits, compared to some variants of the state-of-the-art algorithms (originally developed for standard kernelized bandits). Furthermore, by incorporating various differential privacy models, we generalize DPBE to provide privacy guarantees for users participating in the distributed learning process. The algorithm design, analyses, and numerical experiments are provided in the full version of this paper [4].

show abstract

Distributed Differential Privacy in Multi-Armed Bandits

Chowdhury¹,

Zhou²

2022

Preprint

View full text Add to dashboard Cite

We consider the standard K-armed bandit problem under a distributed trust model of differential privacy (DP), which enables to guarantee privacy without a trustworthy server. Under this trust model, previous work largely focus on achieving privacy using a shuffle protocol, where a batch of users data are randomly permuted before sending to a central server. This protocol achieves (ε, δ) or approximate-DP guarantee by sacrificing an additional additive OIn contrast, the optimal privacy cost for achieving a stronger (ε, 0) or pure-DP guarantee under the widely used central trust model is only Θ K log T ε , where, however, a trusted server is required. In this work, we aim to obtain a pure-DP guarantee under distributed trust model while sacrificing no more regret than that under central trust model. We achieve this by designing a generic bandit algorithm based on successive arm elimination, where privacy is guaranteed by corrupting rewards with an equivalent discrete Laplace noise ensured by a secure computation protocol. We also show that our algorithm, when instantiated with Skellam noise and the secure protocol, ensures Rényi differential privacy -a stronger notion than approximate DP -under distributed trust model with a privacy cost of O K √ log T ε .

show abstract

Shuffle Private Stochastic Convex Optimization

Cited by 5 publications

References 20 publications

Private Non-Convex Federated Learning Without a Trusted Server

Private Non-Convex Federated Learning Without a Trusted Server

(Private) Kernelized Bandits with Distributed Biased Feedback

Distributed Differential Privacy in Multi-Armed Bandits

Contact Info

Product

Resources

About