Almost Tune-Free Variance Reduction

Abstract

The variance reduction class of algorithms including the representative ones, abbreviated as SVRG and SARAH, have well documented merits for empirical risk minimization tasks. However, they require grid search to optimally tune parameters (step size and the number of iterations per inner loop) for best performance. This work introduces 'almost tune-free' SVRG and SARAH schemes by equipping them with Barzilai-Borwein (BB) step sizes. To achieve the best performance, both i) averaging schemes; and, ii) the inner loop length are adjusted according to the BB step size. SVRG and SARAH are first reexamined through an 'estimate sequence' lens. Such analysis provides new averaging methods that tighten the convergence rates of both SVRG and SARAH theoretically, and improve their performance empirically when the step size is chosen large. Then a simple yet effective means of adjusting the number of iterations per inner loop is developed, which completes the tune-free variance reduction together with BB step sizes. Numerical tests corroborate the proposed methods.

Adaptive Step Sizes in Variance Reduction via Regularization

“…After updating x s 0 for m times, xs is chosen following a pmf vector p s . There are different options for p s , and the choice indeed affects the empirical performance of SARAH; see e.g., [Nguyen et al, 2017, Li et al, 2019b. The complexity of SARAH to ensure…”

“…Our proposed approach enjoys a lower complexity than [Ma et al, 2018] in convex problems. And combining our regularization techniques with [Li et al, 2019b], it is possible to obtain a "tune-free" version of BBstep size for (non)convex problems.…”

“…Clearly, fi is µ-strongly convex as well. The strongly convexity of fi can be helpful for choosing p s in SARAH [Li et al, 2019b]. Leveraging f , the regularized SARAH is summarized in Alg.…”

“…In addition, two parameters should be tuned in [Ma et al, 2018], while the BB-version of Alg. 2 can almost eliminate tuning if one adopts "tune-free" BB-SARAH such as [Li et al, 2019b] to minimize f .…”

“…The first adaptive step size for VR was introduced in [Tan et al, 2016], where BB step size for SVRG (BB-SVRG) was studied. BB-SVRG and BB-SARAH were further studied in [Liu et al, Yang et al, 2019, Li et al, 2019b. However, all these works require a strongly convex f .…”