Curriculum Design for Teaching via Demonstrations: Theory and Applications

Abstract

We consider the problem of teaching via demonstrations in sequential decisionmaking settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to the state of the art, our strategy doesn't require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to teach a learner using domain knowledge in the form of task-specific difficulty scores when the teacher's optimal policy is unknown. Experiments on a car driving simulator environment and shortest path problems in a grid-world environment demonstrate the effectiveness of our proposed curriculum strategy.Preprint. Under review.

A Data-efficiency Training Framework for Deep Reinforcement Learning

“…Reward shaping needs in-depth insight to the environment and task to construct proper extra reward functions. Curriculum learning [33][34][35][36] is a methodology to optimize the order in which experience is accumulated by the agent, in order to accelerate the training process and increase the performance. Hierarchical reinforcement learning (HRL) [37][38][39] has recently shown its advantage in sampleefficient learning on the difficult long-horizon tasks.…”