Private Convex Optimization via Exponential Mechanism

Gopi, Sivakanth; Lee, Yin Tat; Liu, Daogao

doi:10.48550/arxiv.2203.00263

Cited by 4 publications

(16 citation statements)

References 13 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This result holds even when F is a distribution over G-Lipschitz functions, and we only have sample access to this distribution. This extends a similar implementation of the marginal sampler required by [LST21b] for log-Lipschitz densities in the ℓ 2 norm, given by [GLL22]. The remaining complexity of the marginal sampling depends on the structure of the chosen ϕ and X , but is independent of F ; we give a discussion of this aspect of our sampler in Sections 5.3 and 6.…”

Section: Our Resultsmentioning

confidence: 75%

“…π(x | y) or π(y | x), mixes rapidly. We give an extended discussion on recent activity on designing and harnessing proximal samplers building upon [LST21b] in Section 1.3, but mention that instantiations of the framework have resulted in state-of-the-art runtimes for many structured density families [CCSW22,LC22,GLL22]. Motivated by the success of proximal methods in the Euclidean setting, one goal of our work is to extend this technique to non-Euclidean geometries.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler

Gopi¹,

Lee²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

The development of efficient sampling algorithms catering to non-Euclidean geometries has been a challenging endeavor, as discretization techniques which succeed in the Euclidean setting do not readily carry over to more general settings. We develop a non-Euclidean analog of the recent proximal sampler of [LST21b], which naturally induces regularization by an object known as the log-Laplace transform (LLT) of a density. We prove new mathematical properties (with an algorithmic flavor) of the LLT, such as strong convexity-smoothness duality and an isoperimetric inequality, which are used to prove a mixing time on our proximal sampler matching [LST21b] under a warm start. As our main application, we show our warm-started sampler improves the value oracle complexity of differentially private convex optimization in ℓ p and Schatten-p norms for p ∈ [1, 2] to match the Euclidean setting [GLL22], while retaining state-of-the-art excess risk bounds [GLL + 23]. We find our investigation of the LLT to be a promising proof-of-concept of its utility as a tool for designing samplers, and outline directions for future exploration.

show abstract

Section: Our Resultsmentioning

confidence: 75%

Section: Introductionmentioning

confidence: 99%

Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler

Gopi¹,

Lee²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In a concurrent, independent, and complementary work on convex losses, Gopi, Lee, and Liu [GLL22] showed that the stationary distribution of a Metropolis-Hastings style process provides the optimal algorithm both for DP-SCO and DP-ERM under (ε, δ)-DP. It is an interesting open question to understand the necessity of the two-phase utility analysis of DP-Langevin diffusion style algorithms.…”

Section: Our Results At a Glancementioning

confidence: 99%

On the Universality of Langevin Diffusion for Private Euclidean (Convex) Optimization

Garg¹,

Thakurta²,

Upadhyay³

2022

Preprint

View full text Add to dashboard Cite

In this paper we revisit the problem of differentially private empirical risk minimization (DP-ERM) and differentially private stochastic convex optimization (DP-SCO). We show that a well-studied continuous time algorithm from statistical physics, called Langevin diffusion (LD), simultaneously provides optimal privacy/utility trade-offs for both DP-ERM and DP-SCO, under ε-DP, and (ε, δ)-DP. Using the uniform stability properties of LD, we provide the optimal excess population risk guarantee for 2 -Lipschitz convex losses under ε-DP (even up to log(n) factors), thus improving on [ALD21].Along the way, we provide various technical tools, which can be of independent interest: i) A new Rényi divergence bound for LD, when run on loss functions over two neighboring data sets, ii) Excess empirical risk bounds for last-iterate LD, analogous to that of Shamir and Zhang [SZ13] for noisy stochastic gradient descent (SGD), and iii) A two phase excess risk analysis of LD, where the first phase is when the diffusion has not converged in any reasonable sense to a stationary distribution, and in the second phase when the diffusion has converged to a variant of Gibbs distribution. Our universality results crucially rely on the dynamics of LD. When it has converged to a stationary distribution, we obtain the optimal bounds under ε-DP. When it is run only for a very short time ∝ 1/p, we obtain the optimal bounds under (ε, δ)-DP. Here, p is the dimensionality of the model space.Our work initiates a systematic study of DP continuous time optimization. We believe this may have ramifications in the design of discrete time DP optimization algorithms analogous to that in the non-private setting, where continuous time dynamical viewpoints have helped in designing new algorithms, including the celebrated mirror-descent (see the discussion in Nemirovski and Yudin [NY83, Chapter 3]) and Polyak's momentum method [Pol64].

show abstract

“…The problem of obtaining sampling lower bounds is a notorious open problem raised in many prior works (see, e.g., Cheng et al, 2018b;Ge et al, 2020;Lee et al, 2021;Chatterji et al, 2022). So far, unconditional lower bounds have only been obtained in restricted settings such as in dimension 1; see Chewi et al (2022c) and the discussion therein, as well as the reduction to optimization in Gopi et al (2022). Our lower bounds are the first of their kind for Fisher information guarantees, and are some of the only lower bounds for sampling in general.…”

Section: Introductionmentioning

confidence: 86%

Fisher information lower bounds for sampling

Chewi¹,

Gerber²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

We prove two lower bounds for the complexity of non-log-concave sampling within the framework of , who introduced the use of Fisher information (FI) bounds as a notion of approximate first-order stationarity in sampling. Our first lower bound shows that averaged LMC is optimal for the regime of large FI by reducing the problem of finding stationary points in non-convex optimization to sampling. Our second lower bound shows that in the regime of small FI, obtaining a FI of at most ε 2 from the target distribution requires poly(1/ε) queries, which is surprising as it rules out the existence of high-accuracy algorithms (e.g., algorithms using Metropolis-Hastings filters) in this context.

show abstract

Private Convex Optimization via Exponential Mechanism

Cited by 4 publications

References 13 publications

Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler

Algorithmic Aspects of the Log-Laplace Transform and a Non-Euclidean Proximal Sampler

On the Universality of Langevin Diffusion for Private Euclidean (Convex) Optimization

Fisher information lower bounds for sampling

Contact Info

Product

Resources

About