Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

Abstract

Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decisionmaking task. We formalize the problem of finding maximally informative demonstrations for IRL as a machine teaching problem where the goal is to find the minimum number of demonstrations needed to specify the reward equivalence class of the demonstrator. We extend previous work on algorithmic teaching for sequential decision-making tasks by showing a reduction to the set cover problem which enables an efficient approximation algorithm for determining the set of maximallyinformative demonstrations. We apply our proposed machine teaching algorithm to two novel applications: providing a lower bound on the number of queries needed to learn a policy using active IRL and developing a novel IRL algorithm that can learn more efficiently from informative demonstrations than a standard IRL approach.

Machine Teaching for Human Inverse Reinforcement Learning

“…then following π after, and Q*(s, a) refers to the optimal Q-value in a state and a possible action (Watkins and Dayan, 1992). Brown and Niekum (2019) proved that the BEC(D|π) of a set of demonstrations D of a policy π can be formulated similarly as the intersection of the following half-spaces…”

“…Cakmak and Lopes (2012) instead focused on IRL learners and selected demonstrations that maximally reduced uncertainty over all viable reward parameters, posed as a volume removal problem. Brown and Niekum (2019) improved this method (particularly for high dimensions) by solving an equivalent set cover problem instead with their Set Cover Optimal Teaching (SCOT) algorithm. However, SCOT is not explicitly designed for human learners and this paper builds on SCOT to address that gap.…”

“…Though IRL offers a principled measure of a demonstration’s informativeness, human learning is multi-faceted and is also influenced by other factors, such as the simplicity of explanations ( Lombrozo, 2016 ). Thus, unlike prior work on machine teaching that optimizes solely for IRL ( Brown and Niekum, 2019 ), this paper incorporates insights on how humans effectively learn to further accommodate human learners.…”

From demonstrations to task-space specifications. Using causal analysis to extract rule parameterization from demonstrations

“…One way to learn such latent structure can be in the form of a reward function obtained from Inverse Reinforcement Learning as described in Ng et al [28], Zhifei and Meng Joo [38], Brown and Niekum [4]. However, it is not always clear that the underlying true reward, in the sense of being the unique reward an expert may have used, is re-constructable or even if it can be sufficiently approximated.…”

Analyzing Sensor-Based Individual and Population Behavior Patterns via Inverse Reinforcement Learning

“…This person may be an expert in the problem domain or, as in our work, an individual whose behavior we want to better understand. Because researchers realize the value of IRL for training artificial agents in solving difficult problems, researchers have introduced several IRL variations, including Maximum Entropy [ 38 , 39 ], Relative Entropy [ 40 , 41 ], and Bayesian IRL [ 42 , 43 ]. Existing techniques can be broadly categorized into model-based approaches [ 39 ] and model-free approaches [ 41 , 44 ].…”