Stochastic Recursive Variance-Reduced Cubic Regularization Methods

Abstract

Stochastic Variance-Reduced Cubic regularization (SVRC) algorithms have received increasing attention due to its improved gradient/Hessian complexities (i.e., number of queries to stochastic gradient/Hessian oracles) to find local minima for nonconvex finite-sum optimization. However, it is unclear whether existing SVRC algorithms can be further improved. Moreover, the semi-stochastic Hessian estimator adopted in existing SVRC algorithms prevents the use of Hessian-vector product-based fast cubic subproblem solvers, which makes SVRC algorithms computationally intractable for high-dimensional problems. In this paper, we first present a Stochastic Recursive Variance-Reduced Cubic regularization method (SRVRC) using a recursively updated semi-stochastic gradient and Hessian estimators. It enjoys improved gradient and Hessian complexities to find an ( , √ )-approximate local minimum, and outperforms the state-of-the-art SVRC algorithms. Built upon SRVRC, we further propose a Hessian-free SRVRC algorithm, namely SRVRC free , which only needs O(n −2 ∧ −3 ) stochastic gradient and Hessian-vector product computations, where n is the number of component functions in the finite-sum objective and is the optimization precision. This outperforms the best-known result O( −3.5 ) achieved by stochastic cubic regularization algorithm proposed in Tripuraneni et al. (2018).

On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points

“…Note that this lower bound is restricted to deterministic algorithms, and thus does not apply to most existing algorithms for escaping saddle points as they are all randomized algorithms. For the stochastic setting with Lipschitz stochastic gradients, the best known query complexity is Õ(ǫ −3 ) [Fang et al, 2018, Zhou andGu, 2019], while the lower bound remains open. See Appendix A for further discussion of the literature.…”

“…, Allen-Zhu and Li [2017] obtain similar results without the requirement of a Hessian-vector product oracle. The sharpest rates in this category have been obtained by Fang et al [2018] and Zhou and Gu [2019], who show that the iteration complexity can be further reduced to Õ(ǫ −3 ). Again, however, this line of works consists of double-loop algorithms, and it remains unclear whether they will have an impact on practice.…”

“…Natasha 2 [Allen-Zhu, 2018] Õ(ǫ −3.5 ) × double-loop Stochastic Cubic [Tripuraneni et al, 2018] Õ(ǫ −3.5 ) × SPIDER [Fang et al, 2018] Õ(ǫ −3 ) × SRVRC [Zhou and Gu, 2019] Õ(ǫ −3 ) × Table 3: A summary of related work on first-order algorithms to find second-order stationary points in the stochastic setting. This table only highlights the dependencies on d and ǫ.…”

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points

“…With a recently proposed optimal variance reduced gradient techniques applied, Spider achieves the state-of-the-art Õ( −3 ) stochastic gradient computational cost (Fang et al, 2018). 8 Very recently, Zhou & Gu (2019) and Shen et al…”

Escaping Saddle Points with Compressed SGD

“…There are also a number of algorithms designed for finite sum setting where f (x) = n i=1 f i (x) [Reddi et al, 2017, Allen-Zhu and Li, 2018, Fang et al, 2018, or in case when only stochastic gradients are available [Tripuraneni et al, 2018, Jin et al, 2021, including variance reduction techniques [Allen-Zhu, 2018, Fang et al, 2018]. The sharpest rates in these settings have been obtained by Fang et al [2018], Zhou and Gu [2019] and Fang et al [2019].…”