A generic adaptive restart scheme with applications to saddle point algorithms

Abstract

We provide a simple and generic adaptive restart scheme for convex optimization that is able to achieve worst-case bounds matching (up to constant multiplicative factors) optimal restart schemes that require knowledge of problem specific constants. The scheme triggers restarts whenever there is sufficient reduction of a distance-based potential function. This potential function is always computable. We apply the scheme to obtain the first adaptive restart algorithm for saddle-point algorithms including primal-dual hybrid gradient (PDHG) and extragradient. The method improves the worst-case bounds of PDHG on bilinear games, and numerical experiments on quadratic assignment problems and matrix games demonstrate dramatic improvements for obtaining high-accuracy solutions. Additionally, for accelerated gradient descent (AGD), this scheme obtains a worst-case bound within 60% of the bound achieved by the (unknown) optimal restart period when high accuracy is desired. In practice, the scheme is competitive with the heuristic of O'Donoghue and Candes [1].Preprint. Under review.

Restarting Algorithms: Sometimes there is Free Lunch

“…While this could be acceptable in the strongly convex case, for more complex schemes to leverage, e.g., sharpness, this is unacceptable as the required parameters are hard to estimate and generally inaccessible. This however, can be remedied in the case of sharpness, at the cost of an extra O(log 2 )-factor in the rates, via scheduled restarts as done in [39] that do not require sharpness parameters as input or when an error bound (of similar convergence rate) is available as in the case of conditional gradients [27]; see also [23] for a very recent adaptive restart scheme using error bounds estimators.…”

“…On the downside, restarts often explicitly depend on parameters arising from the additional structure under consideration and obtained guarantees are off by some constant factor or even log factor. The former can be often remedied with adaptive or scheduled restarts (see e.g., [39,23]) albeit with some minor cost. This way we can obtain fully adaptive algorithms that adapt to additional structure without knowing the accompanying parameters explicitly.…”

“…This have been extensively used for lazification of otherwise expensive algorithms in [11,12] leading to several orders of speed-up in actual computations while maintaining worst-case guarantees identical to those of the original algorithms and in [28] a so-called optimal method based on lazification has been derived. Very recently, in [23] a new adaptive restart scheme has been presented that does not require any knowledge of otherwise inaccessible parameters and its efficacy for saddle point problems has been demonstrated.…”