ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization

Pham, Nhan H.; Nguyen, Lam M.; Phan, Dzung T.; Tran-Dinh, Quoc

doi:10.48550/arxiv.1902.05679

Cited by 12 publications

(53 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The algorithms SARAH, prox-SARAH [19], SPIDER [11], SPIDERBoost [25] and SPIDER-M [28] use this gradient estimator.…”

Section: Arxiv:190601133v1 [Mathoc] 4 Jun 2019mentioning

confidence: 99%

“…2 However, there are notable exceptions that suggest biased algorithms are worth further consideration. Recently, [11,19,25,28] proved that algorithms using the SARAH gradient estimator achieve the oracle complexity lower bound of O √ n 2 for non-convex composite optimisation. For comparison, the best complexity proved for SAGA and SVRG in this setting is O n 2/3 2 .…”

Section: Arxiv:190601133v1 [Mathoc] 4 Jun 2019mentioning

confidence: 99%

See 1 more Smart Citation

Accelerating variance-reduced stochastic gradient methods

2020

View full text Add to dashboard Cite

Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown to directly benefit from Nesterov’s acceleration techniques to match the convergence rates of accelerated gradient methods. Such approaches rely on “negative momentum”, a technique for further variance reduction that is generally specific to the SVRG gradient estimator. In this work, we show for the first time that negative momentum is unnecessary for acceleration and develop a universal acceleration framework that allows all popular variance-reduced methods to achieve accelerated convergence rates. The constants appearing in these rates, including their dependence on the number of functions n, scale with the mean-squared-error and bias of the gradient estimator. In a series of numerical experiments, we demonstrate that versions of SAGA, SVRG, SARAH, and SARGE using our framework significantly outperform non-accelerated versions and compare favourably with algorithms using negative momentum.

show abstract

“…The algorithms SARAH, prox-SARAH [19], SPIDER [11], SPIDERBoost [25] and SPIDER-M [28] use this gradient estimator.…”

Section: Arxiv:190601133v1 [Mathoc] 4 Jun 2019mentioning

confidence: 99%

Section: Arxiv:190601133v1 [Mathoc] 4 Jun 2019mentioning

confidence: 99%

Accelerating variance-reduced stochastic gradient methods

2020

View full text Add to dashboard Cite

show abstract

“…1, which necessitates SARAH's 'non-divergent' assumption. A few works have identified this issue [16,18,21], but require an n-related mini-batch size or step size 2 . The proposed L2S bypasses this n-dependence by removing the inner loop of SARAH and computing snapshot gradients following a random schedule.…”

Section: Loopless Sarah For Convex Problemsmentioning

confidence: 99%

“…Among variance reduction algorithms, the distinct feature of SARAH [15,16] and its variants [18][19][20][21] is that they rely on a biased gradient estimator v t formed by recursively using stochastic gradients. SARAH performs comparably to SVRG for strongly convex ERM, but outperforms SVRG for nonconvex losses, while unlike SAGA, it does not require to store a gradient table.…”

Section: Introductionmentioning

confidence: 99%

On the Convergence of SARAH and Beyond

Li¹,

Ma²,

Giannakis³

2019

Preprint

View full text Add to dashboard Cite

The main theme of this work is a unifying algorithm, abbreviated as L2S, that can deal with (strongly) convex and nonconvex empirical risk minimization (ERM) problems. It broadens a recently developed variance reduction method known as SARAH. L2S enjoys a linear convergence rate for strongly convex problems, which also implies the last iteration of SARAH's inner loop converges linearly. For convex problems, different from SARAH, L2S can afford step and mini-batch sizes not dependent on the data size n, and the complexity needed to guaranteeFor nonconvex problems on the other hand, the complexity is O(n + √ n/ ). Parallel to L2S there are a few side results. Leveraging an aggressive step size, D2S is proposed, which provides a more efficient alternative to L2S and SARAH-like algorithms. Specifically, D2S requires a reduced IFO complexity of O (n + κ) ln(1/ ) for strongly convex problems. Moreover, to avoid the tedious selection of the optimal step size, an automatic tuning scheme is developed, which obtains comparable empirical performance with SARAH using judiciously tuned step size.

show abstract

“…Including the variants of SARAH also employed a constant step size [30], [34]. In addition, Pham et al [33] proposed proximal SARAH (ProxSARAH) for stochastic composite nonconvex optimization and showed that ProxSARAH works with new constant and adaptive step sizes, where the constant step size is much larger than existing methods, including proximal SVRG (ProxSVRG) schemes [35] in the single sample case and adaptive step-sizes are increasing along the inner iterations rather than diminishing as in stochastic proximal gradient descent methods. However, it is complicated to compute adaptive step size for ProxSARAH.…”

Section: Introductionmentioning

confidence: 99%

Accelerating Mini-batch SARAH by Step Size Rules

Yang

Chen²,

Wang

2019

Preprint

View full text Add to dashboard Cite

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective for general nonconvex optimization, has received great attention due to its simple recursive framework for updating stochastic gradient estimates. The performance of SARAH significantly depends on the choice of step size sequence. However, SARAH and its variants often employ a best-tuned step size by mentor, which is time consuming in practice. Motivated by this gap, we proposed a variant of the Barzilai-Borwein (BB) method, referred to as the Random Barzilai-Borwein (RBB) method, to calculate step size for SARAH in the mini-batch setting, thereby leading to a new SARAH method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly in expectation for strongly convex objective functions. We analyze the complexity of MB-SARAH-RBB and show that it is better than the original method. Numerical experiments on standard data sets indicate that MB-SARAH-RBB outperforms or matches state-of-the-art algorithms.

show abstract

ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization

Cited by 12 publications

References 24 publications

Accelerating variance-reduced stochastic gradient methods

Accelerating variance-reduced stochastic gradient methods

On the Convergence of SARAH and Beyond

Accelerating Mini-batch SARAH by Step Size Rules

Contact Info

Product

Resources

About