Adaptive Step Sizes in Variance Reduction via Regularization

Abstract

The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex ones via regularization. The proposed regularization schemes are simple yet effective. Wedding the BB step size with a variance reduction method, known as SARAH, offers a free lunch compared with vanilla SARAH in convex problems. The convergence of BB step sizes in nonconvex problems is also established and its complexity is no worse than other adaptive step sizes such as AdaGrad. As a byproduct, our regularized SARAH methods for convex functions ensure that the complexity to find E, improving dependence over existing results. Numerical tests further validate the merits of proposed approaches.

A Mini-batch Stochastic Recursive Gradient Method with Barzilai-Borwein Step-Size for Machine Learning

“…Tan et al [20] introduced SGD-BB and SVRG-BB, which use the BB method to determine step size for SGD and SVRG, respectively. Li et al [21] applied the BB method to calculate the step size for SARAH, and other researchers such as, Liu et al [22] and Yang et al [23][24][25] incorporated the BB method to compute step size for the variants of SGD type algorithms. Sampling strategies play a crucial role in improving the performance of SGD during training.…”