Accelerated Methods for NonConvex Optimization

Abstract

We prove impossibility results for adaptivity in non-smooth stochastic convex optimization. Given a set of problem parameters we wish to adapt to, we define a "price of adaptivity" (PoA) that, roughly speaking, measures the multiplicative increase in suboptimality due to uncertainty in these parameters. When the initial distance to the optimum is unknown but a gradient norm bound is known, we show that the PoA is at least logarithmic for expected suboptimality, and double-logarithmic for median suboptimality. When there is uncertainty in both distance and gradient norm, we show that the PoA must be polynomial in the level of uncertainty. Our lower bounds nearly match existing upper bounds, and establish that there is no parameter-free lunch.

A hypothesis about the rate of global convergence for optimal methods (Newtons type) in smooth convex optimization

“…Идея за-ключается в том, чтобы формировать матрицу Гессе оптимизируемой функции исходя из матриц Гессе относительно небольшого числа случайно выбранных слагаемых [Ghadimi et al, 2017]. Другая идея заключается в отказе от обращения матрицы Гессе на итерации, вместо этого пред-лагается использовать информацию о собственном векторе, отвечающем наименьшему собствен-ному значению [Agarwal et al, 2017;Carmon et al, 2017]. Для приближенного вычисления тако-го вектора вполне достаточно уметь умножать матрицу Гессе на произвольный вектор:…”

Combining Stochastic Adaptive Cubic Regularization with Negative Curvature for Nonconvex Optimization

“…Carmon et al (Carmon et al, 2018) used negative curvature directions at the first phase of iterates and then switched it to accelerated stochastic gradient descent method when an iterate reaches an almost convex region.…”

An efficient adaptive accelerated inexact proximal point method for solving linearly constrained nonconvex composite problems

“…In particular, accelerated inexact proximal points (AIPP) methods, i.e. methods which use an accelerated composite gradient variant to approximately solve the generated sequence of prox subproblems, have been developed for it (see, for example, [2,5]). When Y is not a singleton, (1) can no longer be directly solved by an AIPP method due to the nonsmoothness of the max term.…”

“…x − x ≤ δ (2) in at most O(δ −3 ) gradient and proximal subproblem evaluations. Next, we develop an AIPP-S variant that computes an approximate solution involving a saddle-point formulation of (1).…”

“…However, their method does not solve the perturbed problem using an accelerated method unlike the approach taken in this paper. Each of the methods in [9,10] consider notions of approximate solutions that are different from (2) and (3), making a comparison between these methods and the one presented in this paper not straightforward. We instead defer this discussion to Section 5 where it is shown that the AIPP-S method is more efficient when a common termination criterion is used.…”