Distributionally Robust Optimization via Ball Oracle Acceleration

Abstract

We develop and analyze algorithms for distributionally robust optimization (DRO) of convex losses. In particular, we consider group-structured and bounded f -divergence uncertainty sets. Our approach relies on an accelerated method that queries a ball optimization oracle, i.e., a subroutine that minimizes the objective within a small ball around the query point. Our main contribution is efficient implementations of this oracle for DRO objectives. For DRO with N non-smooth loss functions, the resulting algorithms find an -accurate solution with O N −2/3 + −2 first-order oracle queries to individual loss functions. Compared to existing algorithms for this problem, we improve complexity by a factor of up to −4/3 .

Optimal and Adaptive Monteiro-Svaiter Acceleration

“…The input λ is optional: oracle implementations in prior work do not require it, but our new adaptive oracles (described in the next section) use it for improved efficiency. In Appendix B we provide a slightly more general approximation condition for MS oracles that handles non-smooth objectives and bounded domains, as well as a different, stochastic condition similar to that of [4,11].…”

“…Implementing O r-BaCoN requires only a single Hessian evaluation and a number of linear system solutions that is polylogarithmic in problem parameters. Subsequent works implementing ball oracles [13,4,11] satisfy an approximation guarantee different than the MS condition, similar to the one we describe in Appendix B.…”

“…Subtracting f (x ), multiplying by A t+1 , combining with (11) to bound f (x t+1 ) and noting that γ t+1 a t+1 = a t+1 yields…”

“…We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second-or first-order method via exact linear system solution or MinRes, respectively. On logistic regression our method outperforms previous second-order acceleration schemes, but under-performs Newton's method; simply iterating our first-order adaptive subproblem solver performs comparably to L-BFGS.∞ regression [8,12], minimizing functions with Hölder continuous higher derivatives [40], and distributionally-robust optimization [13,11].…”

“…∞ regression [8,12], minimizing functions with Hölder continuous higher derivatives [40], and distributionally-robust optimization [13,11].…”