An Optimization Framework for Task Sequencing in Curriculum Learning

Abstract

Curriculum learning in reinforcement learning is used to shape exploration by presenting the agent with increasingly complex tasks. The idea of curriculum learning has been largely applied in both animal training and pedagogy. In reinforcement learning, all previous task sequencing methods have shaped exploration with the objective of reducing the time to reach a given performance level. We propose novel uses of curriculum learning, which arise from choosing different objective functions. Furthermore, we define a general optimization framework for task sequencing and evaluate the performance of popular metaheuristic search methods on several tasks. We show that curriculum learning can be successfully used to: improve the initial performance, take fewer suboptimal actions during exploration, and discover better policies.

A Gray-Box Approach for Curriculum Learning

“…For this example we set N m = 400. See [8] for futher details about these examples. Algorithm C 0 requires 1 curriculum evaluation, i.e.…”

“…environments) different from the final task m L , see e.g. [8] and references therein. The curriculum c = (m 0 , .…”

“…[6,7]. See [8] and the references therein for possible numerical procedures to tackle problem (3). As we show in section 6, the methods proposed in [8] constitute only a preliminary step in order to solve efficiently the curriculum learning problem.…”

“…c = ∅, but the agent is trained directly to solve the final task m L with starting point θ L = 0 K . -GREEDY Par: Greedy algorithm which constructs the curriculum incrementally by considering at each iteration the n tasks which mostly improve the final performance [8]. This is used as benchmark.…”

“…Curriculum learning is gaining popularity in (deep) reinforcement learning, see e.g. [8] and references therein. It can provide better exploration policies through transfer and generalization from less complex tasks.…”