Automatic Curriculum Learning For Deep RL: A Short Survey

Portelas, Rémy; Colas, Cédric; Weng, Li; Hofmann, Katja; Oudeyer, Pierre-Yves

doi:10.48550/arxiv.2003.04664

Cited by 31 publications

(45 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare our proposed approach with the other state-of-the-art ACL methods: (1) GoalGAN [Florensa et al, 2018], which uses a generative adversarial neural network (GAN) to propose tasks for the agent to finish; (2) ALP-GMM [Portelas et al, 2020a], which models the agent absolute learning progress with Gaussian mixture models. None of these baselines utilize multiple curricula.…”

Section: Comparing Moc With State-of-the-art Acl Methodsmentioning

confidence: 99%

“…Automatic curriculum learning aims to maximize a metric P computed over a set of target tasks T ∼ T target after some episodes t . Following the notation in [Portelas et al, 2020a], the objective is set to: max D T ∼Ttarget P t T dT , where D : H → T target is a task selection function. The input of D is H the history, and the output of D is a curriculum such as an initial state.…”

Section: Preliminariesmentioning

confidence: 99%

“…In this work, we use three curriculum modules responsible for generating pre-defined curricula [Portelas et al, 2020a]: initial state generator, sub-goal state generator, and reward shaping generator.Our approach can be easily extended to include other forms of curricula (e.g., selecting environments from a discrete set [Matiisen et al, 2019]) by adding another curriculum generator to the shared hyper-net. These Base-RNNs simultaneously output the actual curricula for the DRL agent in a synergistic manner.…”

Section: Manually Designed Curriculamentioning

confidence: 99%

“…Curriculum learning. Automatic curriculum learning (ACL) for deep reinforcement learning (DRL) [Narvekar et al, 2017, Portelas et al, 2020a, Svetlik et al, 2017, Narvekar and Stone, 2018, Campero et al, 2020 has recently emerged as a promising tool to learn how to adapt an agent's learning tasks based on its capacity during training. ACL [Graves et al, 2017] can be applied to DRL in a variety of ways, including adapting initial states [Florensa et al, 2017, Salimans and Chen, 2018, Ivanovic et al, 2019, shaping reward functions [Bellemare et al, 2016, Pathak et al, 2017, Shyam et al, 2019, or generating goals [Lair et al, 2019, Sukhbaatar et al, 2017, Florensa et al, 2018, Long et al, 2020.…”

Section: Related Workmentioning

confidence: 99%

“…Over time, curriculum learning has become more widely used in machine learning to control the stream of examples provided to training algorithms [Elman, 1993, Bengio et al, 2009, to adapt model capacity [Krueger and Dayan, 2009], and to organize exploration [Schmidhuber, 1991]. Automatic curriculum learning (ACL) for deep reinforcement learning (DRL) [Portelas et al, 2020a] has recently emerged as a promising tool to learn how to adapt an agent's learning tasks to its capabilities during training. ACL can be applied to DRL in various ways, including adapting initial states [Florensa et al, 2017, Ivanovic et al, 2019, shaping reward functions [Bellemare et al, 2016, Shyam et al, 2019, and generating goals [Lair et al, 2019, Sukhbaatar et al, 2017.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learning Multi-Objective Curricula for Deep Reinforcement Learning

Kang¹,

Liu²,

Gupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL). They are designed to control how a DRL agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabilities. For example, ACL can be used for subgoal generation, reward shaping, environment generation, or initial state generation. However, prior work only considers curriculum learning following one of the aforementioned predefined paradigms. It is unclear which of these paradigms are complementary, and how the combination of them can be learned from interactions with the environment. Therefore, in this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net to parameterize all those curriculum modules. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum, which may otherwise be difficult to design manually. We evaluate our method on a series of robotic manipulation tasks and demonstrate its superiority over other state-of-the-art ACL methods in terms of sample efficiency and final performance.

show abstract

Section: Comparing Moc With State-of-the-art Acl Methodsmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Manually Designed Curriculamentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning Multi-Objective Curricula for Deep Reinforcement Learning

Kang¹,

Liu²,

Gupta³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Adaptive algorithms for shaping behavior

Tong,

Iyer,

Murthy

et al. 2023

Preprint

View full text Add to dashboard Cite

Dogs and laboratory mice are commonly trained to perform complex tasks by guiding them through a curriculum of simpler tasks (‘shaping’). What are the principles behind effective shaping strategies? Here, we propose a machine learning framework for shaping animal behavior, where an autonomous teacher agent decides its student’s task based on the student’s transcript of successes and failures on previously assigned tasks. Using autonomous teachers that plan a curriculum in a common sequence learning task, we show that near-optimal shaping algorithms adaptively alternate between simpler and harder tasks to carefully balance reinforcement and extinction. Based on this intuition, we derive an adaptive shaping heuristic with minimal parameters, which we show is near-optimal on the sequence learning task and robustly trains deep reinforcement learning agents on navigation tasks that involve sparse, delayed rewards. Extensions to continuous curricula are explored. Our work provides a starting point towards a general computational framework for shaping animal behavior.

show abstract

Embodied Visual Navigation With Automatic Curriculum Learning in Real Environments

Morad

Mecca

Poudel

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

We present NavACL, a method of automatic curriculum learning tailored to the navigation task. NavACL is simple to train and efficiently selects relevant tasks using geometric features. In our experiments, deep reinforcement learning agents trained using NavACL in collision-free environments significantly outperform state-of-the-art agents trained with uniform sampling -the current standard. Furthermore, our agents are able to navigate through unknown cluttered indoor environments to semantically-specified targets using only RGB images. Collision avoidance policies and frozen feature networks support transfer to unseen real-world environments, without any modification or retraining requirements. We evaluate our policies in simulation, and in the real world on a ground robot and a quadrotor drone. Videos of real-world results are available in the supplementary material. 1

show abstract

Automatic Curriculum Learning For Deep RL: A Short Survey

Cited by 31 publications

References 0 publications

Learning Multi-Objective Curricula for Deep Reinforcement Learning

Learning Multi-Objective Curricula for Deep Reinforcement Learning

Adaptive algorithms for shaping behavior

Embodied Visual Navigation With Automatic Curriculum Learning in Real Environments

Contact Info

Product

Resources

About