Interactive Teaching Algorithms for Inverse Reinforcement Learning

Kamalaruban, Parameswaran; Devidze, Rati; Cevher, Volkan; Singla, Adish

doi:10.24963/ijcai.2019/374

Cited by 31 publications

(30 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We consider a synthetic driving task and a environment based on previous work [28,25,32]. Environment design, episodes, and objective.…”

Section: Environment For Synthetic Car Driving Taskmentioning

confidence: 99%

“…Moreover, the classifiers are trained to predict the labels of all samples in the training set as in full automation. The extensive body of work on human-machine collaboration has predominantly considered settings in which the machine and the human interact with each other [5,16,17,18,19,25,30,34,35,39,42,48,50,51,52,54]. In this context, our work is more closely connected to a line of work that studies switching behavior and switching costs in the context of human-computer interaction [7,20,22,24,26], which we see as complementary.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning Under Algorithmic Triage

Straitouri,

Singla,

Meresht

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Methods to learn under algorithmic triage have predominantly focused on supervised learning settings where each decision, or prediction, is independent of each other. Under algorithmic triage, a supervised learning model predicts a fraction of the instances and humans predict the remaining ones. In this work, we take a first step towards developing reinforcement learning models that are optimized to operate under algorithmic triage. To this end, we look at the problem through the framework of options and develop a two-stage actor-critic method to learn reinforcement learning models under triage. The first stage performs offline, off-policy training using human data gathered in an environment where the human has operated on their own. The second stage performs on-policy training to account for the impact that switching may have on the human policy, which may be difficult to anticipate from the above human data. Extensive simulation experiments in a synthetic car driving task show that the machine models and the triage policies trained using our two-stage method effectively complement human policies and outperform those provided by several competitive baselines.

show abstract

“…We consider a synthetic driving task and a environment based on previous work [28,25,32]. Environment design, episodes, and objective.…”

Section: Environment For Synthetic Car Driving Taskmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning Under Algorithmic Triage

Straitouri,

Singla,

Meresht

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some works (Walsh and Goschin, 2012 ; Haug et al, 2018 ; Melo et al, 2018 ) investigate the impact that the mismatch between the learner and the teacher's model of the learner may have in the teaching dimension—a situation particularly relevant in group settings. The aforementioned works focus on supervised learning settings, although some more recent works have explored inverse reinforcement learning (IRL) settings (Kamalaruban et al, 2019 ).…”

Section: Introductionmentioning

confidence: 99%

Teaching Multiple Inverse Reinforcement Learners

Melo

Lopes

2021

Front. Artif. Intell.

View full text Add to dashboard Cite

In this paper, we propose the first machine teaching algorithm for multiple inverse reinforcement learners. As our initial contribution, we formalize the problem of optimally teaching a sequential task to a heterogeneous class of learners. We then contribute a theoretical analysis of such problem, identifying conditions under which it is possible to conduct such teaching using the same demonstration for all learners. Our analysis shows that, contrary to other teaching problems, teaching a sequential task to a heterogeneous class of learners with a single demonstration may not be possible, as the differences between individual agents increase. We then contribute two algorithms that address the main difficulties identified by our theoretical analysis. The first algorithm, which we dub SplitTeach, starts by teaching the class as a whole until all students have learned all that they can learn as a group; it then teaches each student individually, ensuring that all students are able to perfectly acquire the target task. The second approach, which we dub JointTeach, selects a single demonstration to be provided to the whole class so that all students learn the target task as well as a single demonstration allows. While SplitTeach ensures optimal teaching at the cost of a bigger teaching effort, JointTeach ensures minimal effort, although the learners are not guaranteed to perfectly recover the target task. We conclude by illustrating our methods in several simulation domains. The simulation results agree with our theoretical findings, showcasing that indeed class teaching is not possible in the presence of heterogeneous students. At the same time, they also illustrate the main properties of our proposed algorithms: in all domains, SplitTeach guarantees perfect teaching and, in terms of teaching effort, is always at least as good as individualized teaching (often better); on the other hand, JointTeach attains minimal teaching effort in all domains, even if sometimes it compromises the teaching performance.

show abstract

“…Their curriculum strategies do not incorporate any feedback from the learner, hence unable to adapt the teaching to the learner's progress. In the interactive setting [28], the teacher can leverage the learner's progress to adaptively choose the next demonstrations to accelerate the learning process. We focus on designing a personalized curriculum of demonstrations in the interactive teaching setting.…”

Section: Introductionmentioning

confidence: 99%

“…The teacher's performance is then measured by the number of demonstrations required to achieve this objective. Based on existing work [28,38], we assume that ∃ θ * ∈ Θ such that π E = π θ * , we refer to θ * as the target teaching parameter, and that a smoothness condition holds in the policy parameter space:…”

mentioning

confidence: 99%

Curriculum Design for Teaching via Demonstrations: Theory and Applications

Yengera¹,

Devidze²,

Kamalaruban³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem of teaching via demonstrations in sequential decisionmaking settings. In particular, we study how to design a personalized curriculum over demonstrations to speed up the learner's convergence. We provide a unified curriculum strategy for two popular learner models: Maximum Causal Entropy Inverse Reinforcement Learning (MaxEnt-IRL) and Cross-Entropy Behavioral Cloning (CrossEnt-BC). Our unified strategy induces a ranking over demonstrations based on a notion of difficulty scores computed w.r.t. the teacher's optimal policy and the learner's current policy. Compared to the state of the art, our strategy doesn't require access to the learner's internal dynamics and still enjoys similar convergence guarantees under mild technical conditions. Furthermore, we adapt our curriculum strategy to teach a learner using domain knowledge in the form of task-specific difficulty scores when the teacher's optimal policy is unknown. Experiments on a car driving simulator environment and shortest path problems in a grid-world environment demonstrate the effectiveness of our proposed curriculum strategy.Preprint. Under review.

show abstract

Interactive Teaching Algorithms for Inverse Reinforcement Learning

Cited by 31 publications

References 1 publication

Reinforcement Learning Under Algorithmic Triage

Reinforcement Learning Under Algorithmic Triage

Teaching Multiple Inverse Reinforcement Learners

Curriculum Design for Teaching via Demonstrations: Theory and Applications

Contact Info

Product

Resources

About