Improved Variance Reduction Methods for Riemannian Non-Convex Optimization

Han, Andi; Gao, Junbin

doi:10.1109/tpami.2021.3112139

Cited by 11 publications

(11 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…holds, which violates (23). Here, we highlight B xy is the adjoint of B yx , and therefore, the eigenvalues…”

Section: Key Propositionsmentioning

confidence: 97%

“…Let L 0 , L 1 , L 2 > 0. [78,33,23]). A function h : M − → R satisfies the PL condition on Riemannian manifold if for any p ∈ M, there exists δ > 0 such that 1 2 gradh(p) The following lemma shows the connection between smoothness of a function on manifold and its Lipschitz Riemannian gradient, which is fundamental for convergence analysis.…”

Section: Riemannian Geodesic Convex Optimizationmentioning

confidence: 99%

“…This assumption is always satisfied for a compact manifold M. The compactness appears to be necessary for retraction-based analysis for first-order algorithms [23,66,33,12]. We remark that for the case of the exponential map, the retraction curve coincides with the geodesic curve.…”

Section: Convergence Under Retractionmentioning

confidence: 99%

“…The PL condition [58] amounts to a sufficient condition to establish linear convergence for gradient-based methods to global optimality [32]. The Riemannian version of the PL condition (Definition 3) has been studied in [78,33,80,23]. In Section 3.1, we impose the Riemannian PL condition on the Hamiltonian H as it allows convergence of RHM to global optimality.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Riemannian Hamiltonian methods for min-max optimization on manifolds

Han¹,

Mishra²,

Jawanpuria³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we study the min-max optimization problems on Riemannian manifolds. We introduce a Riemannian Hamiltonian function, minimization of which serves as a proxy for solving the original min-max problems. Under the Riemannian Polyak-Łojasiewicz (PL) condition on the Hamiltonian function, its minimizer corresponds to the desired min-max saddle point. We also provide cases where this condition is satisfied. To minimize the Hamiltonian function, we propose Riemannian Hamiltonian methods (RHM) and present their convergence analysis. We extend RHM to include consensus regularization and to the stochastic setting. We illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.

show abstract

“…holds, which violates (23). Here, we highlight B xy is the adjoint of B yx , and therefore, the eigenvalues…”

Section: Key Propositionsmentioning

confidence: 97%

Section: Riemannian Geodesic Convex Optimizationmentioning

confidence: 99%

Section: Convergence Under Retractionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Riemannian Hamiltonian methods for min-max optimization on manifolds

Han¹,

Mishra²,

Jawanpuria³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, a majority of the previous studies on SGD in a Riemannian space (see e.g. ; ; Tripuraneni et al (2018); Alimisis et al (2020); Han and Gao (2020)), are purely local in nature, because of the assumption that (θ n ) n∈N stays almost surely in a (fixed and deterministic) compact and geodesically convex subset of Θ. For example, note that all the convergence results derived in depend on the diameter of the compact in which (θ n ) n∈N is assumed to stay.…”

Section: Introductionmentioning

confidence: 99%

On Riemannian Stochastic Approximation Schemes with Fixed Step-Size

Durmus¹,

Jiménez²,

Moulines³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper studies fixed step-size stochastic approximation (SA) schemes, including stochastic gradient schemes, in a Riemannian framework. It is motivated by several applications, where geodesics can be computed explicitly, and their use accelerates crude Euclidean methods. A fixed step-size scheme defines a family of timehomogeneous Markov chains, parametrized by the step-size. Here, using this formulation, non-asymptotic performance bounds are derived, under Lyapunov conditions. Then, for any step-size, the corresponding Markov chain is proved to admit a unique stationary distribution, and to be geometrically ergodic. This result gives rise to a family of stationary distributions indexed by the step-size, which is further shown to converge to a Dirac measure, concentrated at the solution of the problem at hand, as the step-size goes to 0. Finally, the asymptotic rate of this convergence is established, through an asymptotic expansion of the bias, and a central limit theorem.

show abstract

Differentially private Riemannian optimization

Han,

Mishra,

Jawanpuria

et al. 2024

Mach Learn

Self Cite

View full text Add to dashboard Cite

In this paper, we study the differentially private empirical risk minimization problem where the parameter is constrained to a Riemannian manifold. We introduce a framework for performing differentially private Riemannian optimization by adding noise to the Riemannian gradient on the tangent space. The noise follows a Gaussian distribution intrinsically defined with respect to the Riemannian metric on the tangent space. We adapt the Gaussian mechanism from the Euclidean space to the tangent space compatible to such generalized Gaussian distribution. This approach presents a novel analysis as compared to directly adding noise on the manifold. We further prove privacy guarantees of the proposed differentially private Riemannian (stochastic) gradient descent using an extension of the moments accountant technique. Overall, we provide utility guarantees under geodesic (strongly) convex, general nonconvex objectives as well as under the Riemannian Polyak-Łojasiewicz condition. Empirical results illustrate the versatility and efficacy of the proposed framework in several applications.

show abstract

Improved Variance Reduction Methods for Riemannian Non-Convex Optimization

Cited by 11 publications

References 31 publications

Riemannian Hamiltonian methods for min-max optimization on manifolds

Riemannian Hamiltonian methods for min-max optimization on manifolds

On Riemannian Stochastic Approximation Schemes with Fixed Step-Size

Differentially private Riemannian optimization

Contact Info

Product

Resources

About