Learning Multi-Objective Curricula for Deep Reinforcement Learning

Abstract

Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL). They are designed to control how a DRL agent collects data, which is inspired by how humans gradually adapt their learning processes to their capabilities. For example, ACL can be used for subgoal generation, reward shaping, environment generation, or initial state generation. However, prior work only considers curriculum learning following one of the aforementioned predefined paradigms. It is unclear which of these paradigms are complementary, and how the combination of them can be learned from interactions with the environment. Therefore, in this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula that are generated by a set of parametric curriculum modules. Each curriculum module is instantiated as a neural network and is responsible for generating a particular curriculum. In order to coordinate those potentially conflicting modules in unified parameter space, we propose a multi-task hyper-net learning framework that uses a single hyper-net to parameterize all those curriculum modules. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum, which may otherwise be difficult to design manually. We evaluate our method on a series of robotic manipulation tasks and demonstrate its superiority over other state-of-the-art ACL methods in terms of sample efficiency and final performance.

Censored deep reinforcement patrolling with information criterion for monitoring large water resources using Autonomous Surface Vehicles

“…Using the Q-Censoring strategy, learning is simplified and allows for a higher reward in time with little effort. The Non-Censoring Noisy Network version can be seen as a curricular learning like in [39], where it is assimilated first through the boundaries of the map, and later, the entropy reduction policy is optimized. The censored strategy alleviates this condition, especially in longer episodes, where the possibility of collision is greater.…”

Censored Deep Reinforcement Patrolling with Information Criterion for Monitoring Large Water Resources Using Autonomous Surface Vehicles

“…ular learning like in [35], where it is assimilated first through the boundaries of the map, and later, the entropy reduction policy is optimized. The censored strategy alleviates this condition, especially in longer episodes, where the possibility of collision is greater.…”

Gradient-based Bi-level Optimization for Deep Learning: A Survey

“…The work [Lv et al 2017] proposes training tricks such as random scaling to improve the generalization. Besides the above, some work [Knyazev et al 2021][Kang et al 2021 proposes to directly predict parameters, which can be seen as a specially learned optimizer without any gradient update.…”