ES-MAML: Simple Hessian-Free Meta Learning

Song, Xingyou; Gao, Wenbo; Yang, Yuxiang; Choromański, Krzysztof; Pacchiano, Aldo; Tang, Yunhao

doi:10.48550/arxiv.1910.01215

Cited by 18 publications

(23 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [5], the authors noted that the second derivatives can be omitted, reducing MAML to first-order MAML (FOMAML). To address the problem of high computational cost, several other first-order approximations of MAML are proposed, including Reptile in [7], Hessian-free MAML (HF-MAML) in [8] and Evolution-Strategies MAML (ES-MAML) in [9]. In our study, FOMAML is considered in the proposed framework for reducing the computational cost.…”

Section: B Related Workmentioning

confidence: 99%

Dynamic Channel Access via Meta-Reinforcement Learning

Lu¹,

Gursoy²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we address the channel access problem in a dynamic wireless environment via meta-reinforcement learning. Spectrum is a scarce resource in wireless communications, especially with the dramatic increase in the number of devices in networks. Recently, inspired by the success of deep reinforcement learning (DRL), extensive studies have been conducted in addressing wireless resource allocation problems via DRL. However, training DRL algorithms usually requires a massive amount of data collected from the environment for each specific task and the well-trained model may fail if there is a small variation in the environment. In this work, in order to address these challenges, we propose a meta-DRL framework that incorporates the method of Model-Agnostic Meta-Learning (MAML). In the proposed framework, we train a common initialization for similar channel selection tasks. From the initialization, we show that only a few gradient descents are required for adapting to different tasks drawn from the same distribution. We demonstrate the performance improvements via simulation results.

show abstract

Section: B Related Workmentioning

confidence: 99%

Dynamic Channel Access via Meta-Reinforcement Learning

Lu¹,

Gursoy²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Meta-learning A surge of recent works have been devoted to developing theory and algorithms of MAML [2,5,13,14,17]. For example, the 'Almost No Inner Loop' (ANIL) algorithm was proposed in [5], which dissects the meta-learning into two phases: training the initialization of a meta-model, and partial fine-tuning the classification head of the meta-model.…”

Section: Related Workmentioning

confidence: 99%

Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD

Fan¹,

Ram²,

Liu³

2021

Preprint

View full text Add to dashboard Cite

We propose a new computationally-efficient first-order algorithm for Model-Agnostic Meta-Learning (MAML). The key enabling technique is to interpret MAML as a bilevel optimization (BLO) problem and leverage the sign-based SGD (signSGD) as a lower-level optimizer of BLO. We show that MAML, through the lens of signSGD-oriented BLO, naturally yields an alternating optimization scheme that just requires first-order gradients of a learned meta-model. We term the resulting MAML algorithm Sign-MAML. Compared to the conventional first-order MAML (FO-MAML) algorithm, Sign-MAML is theoretically-grounded as it does not impose any assumption on the absence of second-order derivatives during meta training. In practice, we show that Sign-MAML outperforms FO-MAML in various few-shot image classification tasks, and compared to MAML, it achieves a much more graceful tradeoff between classification accuracy and computation efficiency.Preprint. Under review.

show abstract

“…In supervised learning, many approaches to few-shot learning have been developed, ranging from using simple hand-crafted label signatures as priors [30] to more complex metric-learning [31] or meta-learning based [32] methods. In Reinforcement Learning, few-shot learning is almost always approached via meta-learning [33], [34], [35], [36], [37], [38], [39]. Earlier definitions of meta-learning include any algorithm that changes the meta-parameters of another learning algorithm, such that the performance of the latter is improved [40].…”

Section: Related Workmentioning

confidence: 99%

“…This not only makes the method more suitable for deceptive or sparse reward problems, but also leaves more freedom in the design and addition of additional meta-objectives. Second, like [33], [37], it is agnostic to the underlying models that are being trained. Finally, it is simple to implement and scale.…”

Section: Related Workmentioning

confidence: 99%

Few-shot Quality-Diversity Optimization

Salehi,

Coninx,

Doncieux

2021

Preprint

View full text Add to dashboard Cite

In the past few years, a considerable amount of research has been dedicated to the exploitation of previous learning experiences and the design of Few-shot and Meta Learning approaches, in problem domains ranging from Computer Vision to Reinforcement Learning based control. A notable exception, where to the best of our knowledge, little to no effort has been made in this direction is Quality-Diversity (QD) optimisation. QD methods have been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. However, they remain costly due to their reliance on inherently sample inefficient evolutionary processes. We show that, given examples from a task distribution, information about the paths taken by optimisation in parameter space can be leveraged to build a prior population, which when used to initialise QD methods in unseen environments, allows for few-shot adaptation. Our proposed method does not require backpropagation. It is simple to implement and scale, and furthermore, it is agnostic to the underlying models that are being trained. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimisation in these environments.

show abstract

ES-MAML: Simple Hessian-Free Meta Learning

Cited by 18 publications

References 11 publications

Dynamic Channel Access via Meta-Reinforcement Learning

Dynamic Channel Access via Meta-Reinforcement Learning

Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD

Few-shot Quality-Diversity Optimization

Contact Info

Product

Resources

About