An Accelerated Hybrid Proximal Extragradient Method for Convex Optimization and Its Implications to Second-Order Methods

Abstract

This paper presents an accelerated variant of the hybrid proximal extragradient (HPE) method for convex optimization, referred to as the accelerated HPE (A-HPE) framework. Iterationcomplexity results are established for the A-HPE framework, as well as a special version of it, where a large stepsize condition is imposed. Two specific implementations of the A-HPE framework are described in the context of a structured convex optimization problem whose objective function consists of the sum of a smooth convex function and an extended real-valued non-smooth convex function. In the first implementation, a generalization of a variant of Nesterov's method is obtained for the case where the smooth component of the objective function has Lipschitz continuous gradient. In the second implementation, an accelerated Newton proximal extragradient (A-NPE) method is obtained for the case where the smooth component of the objective function has Lipschitz continuous Hessian. It is shown that the A-NPE method has a O(1/k 7/2 ) convergence rate, which improves upon the O(1/k 3 ) convergence rate bound for another accelerated Newton-type method presented by Nesterov. Finally, while Nesterov's method is based on exact solutions of subproblems with cubic regularization terms, the A-NPE method is based on inexact solutions of subproblems with quadratic regularization terms, and hence is potentially more tractable from a computational point of view.

Lower bounds for finding stationary points II: first-order methods

“…For methods using higher-order derivatives, our lower bounds [10] for finding stationary points of non-convex functions are −(p+1)/p → −1 as the order p of smoothness grows. However, similar to Appendix A.1, the results [4,16] can show that for convex functions with Lipschitz Hessian, a second-order method achieves the strictly better rate −6/7 log 1 .…”

Accelerated and Inexact Forward-Backward Algorithms

“…Paper [52] (Lemma 1) shows that criterion (2.13) is more general than both (2.3), (2.12) and, actually, it is the combination of those error criteria. We also note that (again from Lemma 1 in [52]) the error criterion proposed in [38,55] for the approximate hybrid extragradient-proximal point algorithm corresponds to a relative version of (2.13).…”

“…In [31,52] the classical proximal point algorithm is treated (f = 0 in (1.1)). Paper [38] considers inexact accelerated hybrid extragradient-proximal methods, but actually the framework is shown to include only the case of the exact accelerated forward-backward algorithm. In [22], convergence rates for an accelerated projected-subgradient method is proved.…”

Zero-convex functions, perturbation resilience, and subgradient projections for feasibility-seeking methods

“…Several other generalizations or variations of the standard notion of subdifferential have been considered in the literature, e.g., the Clarke subdifferential [41, pp. 25-27], [42], the Fréchet and Hadamard subdifferentials [105], the G-subdifferential [76], the H-subdifferential [82], Mordukhovich's Subdifferential [85,107], Plastria's lower subdifferential [96], the Quasi-subdifferential [61], the Q-subdifferential [83], the Φ-subdifferential [92], the starsubdifferential [94], the ǫ-subdifferential [84], generalizations of the subgradient inequality such as the notion of invexity [14,64] or other notions related to convexity such as approximate convexity [47,87]. For a survey on some of these concepts see [20].…”