Gradient Regularization of Newton Method with Bregman Distances

Abstract

In this paper, we propose a first second-order scheme based on arbitrary non-Euclidean norms, incorporated by Bregman distances. They are introduced directly in the Newton iterate with regularization parameter proportional to the square root of the norm of the current gradient. For the basic scheme, as applied to the composite optimization problem, we establish the global convergence rate of the order O(k −2 ) both in terms of the functional residual and in the norm of subgradients. Our main assumption on the smooth part of the objective is Lipschitz continuity of its Hessian. For uniformly convex functions of degree three, we justify global linear rate, and for strongly convex function we prove the local superlinear rate of convergence. Our approach can be seen as a relaxation of the Cubic Regularization of the Newton method [5], which preserves its convergence properties, while the auxiliary subproblem at each iteration is simpler. We equip our method with adaptive line search procedure for choosing the regularization parameter. We propose also an accelerated scheme with convergence rate O(k −3 ), where k is the iteration counter.

Optimal and Adaptive Monteiro-Svaiter Acceleration

“…Cartis et al [14], Gould et al [21] propose adaptive variants of cubic regularization for non-convex optimization. For convex optimization, Mishchenko [30] provides a simple adaptive scheme converging at rate O(t −2 ), followed by an improvement in its guarantee to O(t −3 ) [17]. For tensor methods, Jiang et al [25] proposes an adaptive regularization scheme for convex functions with Lipschitz-continuous p th derivatives which achieves the rate O(t −p−1 ).…”

“…Even for the case second-order methods with Lipschitz-continuous Hessian, an adaptive scheme with optimal rate O(t −3.5 ) remained open prior to this work. Beyond removing the bisection from the MS framework, our key algorithmic techniques include directly considering quadratically-regularized Newton step (similar to [30,17] and different from [22,25]), and using the original MS approximation condition for selecting an appropriate regularization parameter, which is new in the context of adaptive methods. These techniques allow us to adapt to both the constant and order of Hessian Hölder continuity simultaneously.…”

“…The prior works[30,17] also directly consider quadratically-regularized Newton steps, but employ approximation conditions other than (2) to select the parameter λ.…”

Yet another fast variant of Newton's method for nonconvex optimization

“…However, as noted in [33], the improvement in complexity has been obtained by trading the simple Newton step requiring only the solution of a single linear system for more complex or slower procedures, such as secular iterations, possibly using Lanczos preprocessing [6,8] (see also [12,Chapters 8 to 10]) or (conjugate-)gradient descent [29,4]. In the simpler context of convex problems, two recent papers [33,17] independently proposed another globalization technique. At an iterate x, the step s is computed as…”

SC-Reg: Training Overparameterized Neural Networks under Self-Concordant Regularization

“…Their proposed scheme achieves a global O(k −2 ) rate, where k is the iteration number. [28] follows a similar idea using the Bregman distances, and extends the idea to developing an accelerated variant of the scheme that achieves a O(k −3 ) rate.…”