Riemannian stochastic recursive momentum method for non-convex optimization

Han, Andi; Gao, Junbin

doi:10.48550/arxiv.2008.04555

Cited by 2 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 8. Since assumption 9 is not used, the convergence is global Andi Han et al [15] consider the problem of expectation (online) minimization over Riemannian manifold M. There assumption the stochastic gradient is unbiased, i.e., E ω gradf (x, ω) = gradF (x). And they get a convergence rate of O( 1…”

Section: Special Casementioning

confidence: 99%

“…Its cost and difficulty will be improved. Therefore, a Riemannian stochastic recursive gradient algorithm(R-SRG) independent of two distant points is proposed in [14], to avoids the calculation of contraction inverse and makes the calculation efficiency higher.In addition, from [24,25], Riemannian stochastic recursive momentum(R-SRM) algorithm is proposed in [15]. The author considers the linear combination of R-SGDand R-SVRG,and obtained the R-SRM algorithm (the linear combination coefficient and step size of the algorithm are time-vary).…”

Section: Introductionmentioning

confidence: 99%

“…So in this paper, we consider the situations with retraction mapping and vector transport. Inspired by [15], we consider the linear combination of three descent directions on Riemannian manifolds as the new descent direction(i.e.,R-SRG, R-SVRG and R-SGD) and propose two algorithms called Rimannian Stochastic Hybrid Algorithm(R-SHG) with adaptive parameters and time-varying parameters. And the parameters of linear combination are time-varying.…”

Section: Introductionmentioning

confidence: 99%

“…When special parameters are removed, our results can degenerate to [12]. Moreover, for the case of step size is fixed, we quantitative research the convergence rate of the algorithm 2) [15]considers the linear combination of R-SRG and R-SGD. Our second algorithm R-SHG with time-varying parameters can obtain a faster convergence rate than them.…”

Section: Introductionmentioning

confidence: 99%

“…Our second algorithm R-SHG with time-varying parameters can obtain a faster convergence rate than them. If we consider the problem of expectation (online) minimization over Riemannian manifold M, then we can choose special parameters such that our results can degenerate to [15] and our results better. Moreover, we also give the convergence rate under fixed step size.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Yang

2021

Preprint

View full text Add to dashboard Cite

In recent years, Riemannian stochastic variance reduction algorithms(R-SVRG) and Riemannian stochastic recursive gradient(R-SRG) have attracted considerable attention on Riemannian optimization. In this paper, we consider the linear combination of three descent directions on Riemannian manifolds as the new descent direction(i.e.,R-SRG, R-SVRG and R-SGD) with the parameters of linear combination is time-varying, and we propose two algorithms called Rimannian Stochastic Hybrid Gradient Algorithm(R-SHG) with adaptive parameters and time-varying parameters. Under normal circumstances, it is impossible to analyze the convergence of R-SRG alone. The main reason is that the conditional expectation of the descending direction is biased estimation. Therefore, the R-SHG with adaptive parameters is uses adaptive parameters to get a global convergence analysis with a decaying step size. For step size is fixed, we consider two cases of parameter inner loop fixed and inner loop time-varying, and quantitatively research the convergence speed of the algorithm. Since the global convergence of the algorithm requires higher functional differentiability, we propose R-SHG with time-varying parameters. If we choose option 2. we can get similar conclusions under weaker conditions for the case of step size is reduced and fixed.

show abstract

Section: Special Casementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Yang

2021

Preprint

View full text Add to dashboard Cite

show abstract

On Riemannian Gradient-Based Methods for Minimax Problems

Huang¹,

Gao²

2020

Preprint

View full text Add to dashboard Cite

In the paper, we study a class of useful non-convex minimax optimization problems on the Riemanian manifold and propose a class of Riemanian gradient descent ascent algorithms to solve these minimax problems. Specifically, we propose a new Riemannian gradient descent ascent (RGDA) algorithm for the deterministic minimax optimization. Moreover, we prove that the RGDA has a sample complexity of O(κ 2 −2 ) for finding an -stationary point of the nonconvex strongly-concave minimax problems, where κ denotes the condition number. At the same time, we introduce a Riemannian stochastic gradient descent ascent (RSGDA) algorithm for the stochastic minimax optimization. In the theoretical analysis, we prove that the RSGDA can achieve a sample complexity of O(κ 4 −4 ). To further reduce the sample complexity, we propose a novel momentum variance-reduced Riemannian stochastic gradient descent ascent (MVR-RSGDA) algorithm based on a new momentum variance-reduced technique of STORM. We prove that the MVR-RSGDA algorithm achieves a lower sample complexity of Õ(κ 4 −3 ) without large batches, which reaches near the best known sample complexity for its Euclidean counterparts. This is the first study of the minimax optimization over the Riemannian manifold. Extensive experimental results on the robust deep neural networks training over Stiefel manifold demonstrate the efficiency of our proposed algorithms.

show abstract

Riemannian stochastic recursive momentum method for non-convex optimization

Cited by 2 publications

References 24 publications

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

Riemannian Stochastic Hybrid Gradient Algorithm for Nonconvex Optimization

On Riemannian Gradient-Based Methods for Minimax Problems

Contact Info

Product

Resources

About