Machine Teaching for Human Inverse Reinforcement Learning

Abstract

As robots continue to acquire useful skills, their ability to teach their expertise will provide humans the two-fold benefit of learning from robots and collaborating fluently with them. For example, robot tutors could teach handwriting to individual students and delivery robots could convey their navigation conventions to better coordinate with nearby human workers. Because humans naturally communicate their behaviors through selective demonstrations, and comprehend others’ through reasoning that resembles inverse reinforcement learning (IRL), we propose a method of teaching humans based on demonstrations that are informative for IRL. But unlike prior work that optimizes solely for IRL, this paper incorporates various human teaching strategies (e.g. scaffolding, simplicity, pattern discovery, and testing) to better accommodate human learners. We assess our method with user studies and find that our measure of test difficulty corresponds well with human performance and confidence, and also find that favoring simplicity and pattern discovery increases human performance on difficult tests. However, we did not find a strong effect for our method of scaffolding, revealing shortcomings that indicate clear directions for future work.

Reasoning about Counterfactuals to Improve Human Inverse Reinforcement Learning

“…IRL) but also their beliefs and subsequently what counterfactuals they would consider. We thus extend our previous work [13] to evaluate a demonstration's informativeness based on counterfactuals generated via potential reward functions on the human's mind as opposed to counterfactuals generated via one-action deviations, and scaffold by showing demonstrations of increasing informativeness.…”

“…Brown and Niekum [12] proposed the Set Cover Optimal Teaching (SCOT) algorithm for selecting demonstrations that provide the tightest constraints on a target reward function for a pure IRL learner. However, human learning is more multifaceted and our prior work [13] tailored SCOT for humans by incorporating human learning techniques and concepts such as scaffolding. Our initial method of scaffolding via IRL did not yield significant learning gains, which we aim to improve in this work by incorporating counterfactuals that are based on the human's beliefs regarding the robot's reward function.…”

“…We test learners accordingly by asking them to predict the robot's optimal behavior in unseen instances of a domain. In our prior work [13], we showed that informativeness of a demonstration during teaching could simply be inverted to measure the expected difficulty of correctly predicting it during testing. In this work, we propose to update the difficulty measure by explicitly conditioning on the learner's beliefs of the robot's reward function.…”

“…Generalizing to higher dimensions, we can Testing: The area of a demonstration's BEC intuitively correlates with its informativeness during teaching as smaller areas indicate less uncertainty regarding w * . Our prior work [13] showed that a demonstration's BEC area may also be inverted to measure the difficulty of correctly predicting the demonstration as a test if the human has not seen it before (e.g. so that a smaller BEC area indicates a difficult test).…”

Learning from Explanations With Maximum Likelihood Inverse Reinforcement Learning

“…Explanations help establish a connection between what has been observed and its causes, and serve as a principled basis for generalization [14]. Consequently, explanations scaffold causal learning and have a crucial role in inference [44]. Following this idea, our work also generate explanations in the form of sentence-trajectories and uses maximum likelihood inverse reinforcement learning to find a weighting of the state features that (locally) maximizes the probability of these trajectories.…”