Dispersion for Data-Driven Algorithm Design, Online Learning, and Private Optimization

Abstract

A crucial problem in modern data science is data-driven algorithm design, where the goal is to choose the best algorithm, or algorithm parameters, for a specific application domain. In practice, we often optimize over a parametric algorithm family, searching for parameters with high performance on a collection of typical problem instances. While effective in practice, these procedures generally have not come with provable guarantees. A recent line of work initiated by a seminal paper of Gupta and Roughgarden [34] analyzes application-specific algorithm selection from a theoretical perspective. We progress this research direction in several important settings. We provide upper and lower bounds on regret for algorithm selection in online settings, where problems arrive sequentially and we must choose parameters online. We also consider differentially private algorithm selection, where the goal is to find good parameters for a set of problems without divulging too much sensitive information contained therein.We analyze several important parameterized families of algorithms, including SDP-rounding schemes for problems formulated as integer quadratic programs as well as greedy techniques for several canonical subset selection problems. The cost function that measures an algorithm's performance is often a volatile piecewise Lipschitz function of its parameters, since a small change to the parameters can lead to a cascade of different decisions made by the algorithm. We present general techniques for optimizing the sum or average of piecewise Lipschitz functions when the underlying functions satisfy a sufficient and general condition called dispersion. Intuitively, a set of piecewise Lipschitz functions is dispersed if no small region contains many of the functions' discontinuities.Using dispersion, we improve over the best-known online learning regret bounds for a variety problems, prove regret bounds for problems not previously studied, and provide matching regret lower bounds. In the private optimization setting, we show how to optimize performance while preserving privacy for several important problems, providing matching upper and lower bounds on performance loss due to privacy preservation. Though algorithm selection is our primary motivation, we believe the notion of dispersion may be of independent interest. Therefore, we present our results for the more general problem of optimizing piecewise Lipschitz functions. Finally, we uncover dispersion in domains beyond algorithm selection, namely, auction design and pricing, providing online and privacy guarantees for these problems as well.Private algorithm configuration Kusner et al. [41] develop private Bayesian optimization techniques for tuning algorithm parameters. Their methods implicitly assume that the utility function is differentiable. Meanwhile, the class of functions we consider have discontinuities between pieces, and it is not enough to privately optimize on each piece, since the boundaries themselves are data-dependent.Online optimization Prior work on ...

Estimating Approximate Incentive Compatibility

“…As we describe more in Section 3.1.2, this requires us to prove that with high probability, each function sequence from several infinite families of sequences is dispersed. This facet of our analysis is notably different from prior research by Balcan et al [2018a]: in their applications, it is enough to show that with high probability, a single, finite sequence of functions is dispersed. Our proofs thus necessitate that we carefully examine the structure of the utility functions that we analyze.…”

“…Rather, we must prove that for all type vectors, the dispersion property holds. This facet of our analysis is notably different from prior work by Balcan et al [2018a]: in their applications, it is enough to show that with high probability, a single, finite sequence of functions is dispersed. In contrast, we show that under mild assumptions, with high probability, each function sequence from an infinite family is dispersed.…”

Escaping Saddle Points of Empirical Risk Privately and Scalably via DP-Trust Region Method

“…The authors of [37] studied the problem with smooth loss function and proposed using the 2 gradient-norm of a private estimator, i.e., ∇L(w priv , D) 2 , to measure the utility, which was then extended in [34,31] to the cases of non-smooth loss functions and high dimensional space. It is well known that 2 gradient-norm can estimate only the first-order stationary point (or critical point) 3 , and thus may lead to inferior generalization performance [10]. The authors of [28] are the first to show that the utility of general non-convex loss functions can also be measured in the same way as convex loss functions by the expected excess empirical risk.…”

Automatic Discovery of Privacy–Utility Pareto Fronts

“…Here, we consider a simple private logistic regression model with 2 regularization trained on the Adult dataset [28]. The model is privatized by training with mini-batched projected SGD, then applying a Gaussian perturbation at the output using the method from [49,Algorithm 2] with default parameters 5 . The only hyperparameters tuned in this example are the regularization γ and the noise standard deviation σ, while the rest are fixed 6 .…”

“…Recent work on data-driven algorithm configuration has considered the problem of tuning the hyperparameters of combinatorial optimization algorithms while maintaining DP [5]. The setting considered in [5] assumes there is an underlying distribution of problem instances, and a sample from this distribution is used to select hyperparameters that will have good computational performance on future problem instances sampled from the same distribution. In this case, the authors consider a threat model where the whole sample of problem instances used to tune the algorithm needs to be protected.…”