Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning

Abstract

Transfer learning is a method where an agent reuses knowledge learned in a source task to improve learning on a target task. Recent work has shown that transfer learning can be extended to the idea of curriculum learning, where the agent incrementally accumulates knowledge over a sequence of tasks (i.e. a curriculum). In most existing work, such curricula have been constructed manually. Furthermore, they are fixed ahead of time, and do not adapt to the progress or abilities of the agent. In this paper, we formulate the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and propose a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum. We use our approach to automatically sequence tasks for 3 agents with varying sensing and action capabilities in an experimental domain, and show that our method produces curricula customized for each agent that improve performance relative to learning from scratch or using a different agent's curriculum.

Curriculum Learning for Cumulative Return Maximization

“…For GridWorld, Experiment 3 ( Figure 3) has parameters n := |T | = 12 and L = 4, while in Experiment 4 n = 7, and L = 7. For both domains, the intermediate tasks have been generated manually using methods from Narvekar et al [2017], by varying the size of the environment, and adding and removing elements (pits and fires in GridWorld, and columns and movable blocks in BlockDude). We intentionally created both tasks that provide positive and negative transfer towards the final task, in order to test the ability of the sequencing algorithm to choose the most appropriate ones.…”

“…The automatic generation of curricula [Da Silva and Costa, 2019] has been divided into two sub-problems: task generation [Narvekar et al, 2016;Da Silva and Costa, 2018], that is the problem of creating a set of tasks such that transferring from them is most likely beneficial for the final task; and task sequencing [Svetlik et al, 2017;Narvekar et al, 2017;Da Silva and Costa, 2018;Foglino et al, 2019], whereby previously generated tasks are optimally selected and ordered. Current methods for task sequencing attempt to determine the optimal order of tasks either with [Narvekar et al, 2017;Baranes and Oudeyer, 2013] or without [Svetlik et al, 2017;Da Silva and Costa, 2018] executing the tasks. All task sequencing methods mentioned above are heuristic algorithms tailored to the minimization of time-to-threshold.…”

Curriculum Learning in Reinforcement Learning

“…Transfer learning is leveraged to transfer information between each pair of tasks in this sequence. Our work builds upon the model proposed by Narvekar et al [16], which formulates curriculum generation as an interaction between two agents acting in two different MDPs. One is a learning agent that is trying to solve a specific target task MDP M t , as is the standard case in reinforcement learning.…”

“…a curriculum) for an agent to train on, such that after training on that sequence, learning speed or performance on a target task is improved. Automatically designing a curriculum is an open problem that has only recently begun to be examined [5,8,9,16,19,23]. One recent approach [16] proposed formulating the selection of tasks using a (meta-level) curriculum Markov Decision Process (MDP).…”

“…Automatically designing a curriculum is an open problem that has only recently begun to be examined [5,8,9,16,19,23]. One recent approach [16] proposed formulating the selection of tasks using a (meta-level) curriculum Markov Decision Process (MDP). A policy over this MDP, called a curriculum policy, maps from the current knowledge of an RL agent to the task it should learn next.…”