Hessian averaging in stochastic Newton methods achieves superlinear convergence

Abstract

We consider minimizing a smooth and strongly convex objective function using a stochastic Newton method. At each iteration, the algorithm is given an oracle access to a stochastic estimate of the Hessian matrix. The oracle model includes popular algorithms such as Subsampled Newton and Newton Sketch, which can efficiently construct stochastic Hessian estimates for many tasks, e.g., training machine learning models. Despite using second-order information, these existing methods do not exhibit superlinear convergence, unless the stochastic noise is gradually reduced to zero during the iteration, which would lead to a computational blow-up in the per-iteration cost. We propose to address this limitation with Hessian averaging: instead of using the most recent Hessian estimate, our algorithm maintains an average of all the past estimates. This reduces the stochastic noise while avoiding the computational blow-up. We show that this scheme exhibits local Q-superlinear convergence with a non-asymptotic rate of $$(\varUpsilon \sqrt{\log (t)/t}\,)^{t}$$ ( Υ log ( t ) / t ) t , where $$\varUpsilon $$ Υ is proportional to the level of stochastic noise in the Hessian oracle. A potential drawback of this (uniform averaging) approach is that the averaged estimates contain Hessian information from the global phase of the method, i.e., before the iterates converge to a local neighborhood. This leads to a distortion that may substantially delay the superlinear convergence until long after the local neighborhood is reached. To address this drawback, we study a number of weighted averaging schemes that assign larger weights to recent Hessians, so that the superlinear convergence arises sooner, albeit with a slightly slower rate. Remarkably, we show that there exists a universal weighted averaging scheme that transitions to local convergence at an optimal stage, and still exhibits a superlinear convergence rate nearly (up to a logarithmic factor) matching that of uniform Hessian averaging.

A Variance-Reduced and Stabilized Proximal Stochastic Gradient Method with Support Identification Guarantees for Structured Optimization

“…Assumption 3.2(1) ensures that the stochastic gradient ∇ (x; ξ) is an unbiased estimator of the gradient ∇f (x) for all x ∈ R n . Assumption 3.2(2) provides a constant upper bound on the norm of an element of ∂r(x) for all x ∈ R n , which exists when r is the weighted 1 -norm or weighted group Assumption 3.2(3) by assuming that the stochastic gradient error has a sub-exponential tail, e.g., Na et al (2022), which we leave as future work. Assumption 3.2(4) is implied by the following two, perhaps more natural, assumptions: (i) There exists a constant c e > 0 such that, for all k, it holds that P{ d k − ∇f (x k ) ≤ c e |F k } = 1, i.e., the error in the stochastic gradient estimator d k is almost surely bounded; and (ii) There exists a constant c α such that, for a given α > 0 and all k ≥ 1, it holds that P{χ(x k ; α) ≤ c α |F k } = 1 (also see (3)), i.e., the optimality measure is almost surely bounded.…”

Determinantal Point Processes in Randomized Numerical Linear Algebra

“…Standard RandNLA guarantees such as the subspace embedding are sufficient (although not necessary) to ensure that Ĥt provides a good enough approximation to enable accelerated local convergence in time Õ(nd). These approaches have also been extended to distributed settings via RMT-based model averaging, with applications in ensemble methods, distributed optimization, and federated learning [110,109,144,48,92]. Further RandNLA-based Newton-type methods include: Subsampled Newton [75,159,18,17]; Hessian approximations via randomized Taylor expansion [1] and low-rank approximation [77,55]; Hessian diagonal/trace estimates via Hutchinson's method [136] and Stochastic Lanczos Quadrature, particularly for non-convex problems, e.g., PyHessian [176], AdaHessian [177]; and finally Stochastic Quasi-Newton type methods [106,137].…”