Learning to Optimize on SPD Manifolds

Gao, Zhi; Wu, Yuwei; Jia, Yingmin; Harandi, Mehrtash

doi:10.1109/cvpr42600.2020.00772

Cited by 15 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When this is not possible, a budding area of research investigates more directly including the manifold structure into the amortization process. Gao et al (2020) amortize optimization problems over SPD spaces.…”

Section: Euclidean → Non-euclidean Optimizationmentioning

confidence: 99%

Tutorial on amortized optimization for learning to optimize over continuous domains

Amos¹

2022

Preprint

View full text Add to dashboard Cite

Optimization is a ubiquitous modeling tool and is often deployed in settings which repeatedly solve similar instances of the same problem. Amortized optimization methods use learning to predict the solutions to problems in these settings. This leverages the shared structure between similar problem instances. In this tutorial, we will discuss the key design choices behind amortized optimization, roughly categorizing 1) models into fully-amortized and semi-amortized approaches, and 2) learning methods into regression-based and objectivebased. We then view existing applications through these foundations to draw connections between them, including for manifold optimization, variational inference, sparse coding, meta-learning, control, reinforcement learning, convex optimization, and deep equilibrium networks. This framing enables us easily see, for example, that the amortized inference in variational autoencoders is conceptually identical to value gradients in control and reinforcement learning as they both use fully-amortized models with an objective-based loss.

show abstract

Section: Euclidean → Non-euclidean Optimizationmentioning

confidence: 99%

Tutorial on amortized optimization for learning to optimize over continuous domains

Amos¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Figure 1: We measure the time and memory consumption of the Riemannian meta-optimization method (Gao et al 2020) and our method. The method (Gao et al 2020) has heavy computational and memory burdens with the increase of the number of steps in the inner-loop. In contrast, our method reduces computational cost and memory footprint significantly.…”

Section: Legendmentioning

confidence: 99%

“…In contrast, our method reduces computational cost and memory footprint significantly. This is because the method (Gao et al 2020) differentiates through the whole inner-loop optimization to compute the meta-gradient, which involves a series of time-consumption derivatives and stores a large computation graph, while our meta-gradient computation is only related to the final two iterations.…”

Section: Legendmentioning

confidence: 99%

“…Recently, Riemannian meta-optimization methods (Gao et al 2020;Fan et al 2021) have shown promise in solving Riemannian optimization problems. In contrast to previous hand-designed Riemannian optimizers, Riemannian metaoptimization methods utilize meta-learning to automatically learn an optimizer in a data-driven fashion.…”

Section: Legendmentioning

confidence: 99%

“…Recently, Riemannian meta-optimization methods provide a promising way to address Riemannian optimization problems. Gao et al (Gao et al 2020) introduced a matrix LSTM as the optimizer that takes the gradient as input and generates stepsize and search directions for symmetric positive definite manifolds. However, training the optimizer brings the exploding gradient problem.…”

Section: Related Work Riemannian Optimizationmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Riemannian Meta-Optimization by Implicit Differentiation

Fan

Gao

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

To solve optimization problems with nonlinear constrains, the recently developed Riemannian meta-optimization methods show promise, which train neural networks as an optimizer to perform optimization on Riemannian manifolds. A key challenge is the heavy computational and memory burdens, because computing the meta-gradient with respect to the optimizer involves a series of time-consuming derivatives, and stores large computation graphs in memory. In this paper, we propose an efficient Riemannian meta-optimization method that decouples the complex computation scheme from the meta-gradient. We derive Riemannian implicit differentiation to compute the meta-gradient by establishing a link between Riemannian optimization and the implicit function theorem. As a result, the updating our optimizer is only related to the final two iterations, which in turn speeds up our method and reduces the memory footprint significantly. We theoretically study the computational load and memory footprint of our method for long optimization trajectories, and conduct an empirical study to demonstrate the benefits of the proposed method. Evaluations of three optimization problems on different Riemannian manifolds show that our method achieves state-of-the-art performance in terms of the convergence speed and the quality of optima.

show abstract