Deep Reinforcement Learning from Policy-Dependent Human Feedback

Arumugam, Dilip; Lee, Jun Ki; Saskin, Sophie; Littman, Michael L.

doi:10.48550/arxiv.1902.04257

Cited by 23 publications

(37 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Learning from human feedback. Several works have successfully utilized feedback from real humans to train agents where it is assumed that the feedback is available at all times (Pilarski et al, 2011;MacGlashan et al, 2017;Arumugam et al, 2019). Due to this high feedback frequency, these approaches are difficult to scale to more complex learning problems that require substantial agent experience.…”

Section: Related Workmentioning

confidence: 99%

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Lee¹,

Smith²,

Abbeel³

2021

Preprint

View full text Add to dashboard Cite

Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample-and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.

show abstract

Section: Related Workmentioning

confidence: 99%

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Lee¹,

Smith²,

Abbeel³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Low-level manipulation Surgical assistance [49,68] Vehicle manipulation [80] Robotic arm [61] VR teleoperation [79] High-level tasks 2D gameplay [59] 3D gameplay [4] Navigation [28] Sports analysis [78] evaluated on 3D games like GTAV or Minecraft for evaluation. This taxonomy could be meaningful since it clearly reflects the target domain of the proposed algorithm, as the variance on their evaluation methods could be smaller, this may help to design a unified evaluation metric for IL.…”

Section: Classes Examples and Publicationsmentioning

confidence: 99%

Imitation Learning: Progress, Taxonomies and Opportunities

Zheng¹,

Verma²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Imitation learning aims to extract knowledge from human experts' demonstrations or artificially created agents in order to replicate their behaviours. Its success has been demonstrated in areas such as video games, autonomous driving, robotic simulations and object manipulation. However, this replicating process could be problematic, such as the performance is highly dependent on the demonstration quality, and most trained agents are limited to perform well in task-specific environments. In this survey, we provide a systematic review on imitation learning. We first introduce the background knowledge from development history and preliminaries, followed by presenting different taxonomies within Imitation Learning and key milestones of the field. We then detail challenges in learning strategies and present research opportunities with learning policy from suboptimal demonstration, voice instructions and other associated optimization schemes.

show abstract

“…This insight motivated the design of the Convergent Actor-Critic by Humans (COACH) algorithm, which treats human feedback as an advantage signal. Subsequent work has also successfully applied COACH to training deep learning agents [108].…”

Section: Applying Human Inverse Models To the Design Of Autonomous Sy...mentioning

confidence: 99%

Cognitive science as a source of forward and inverse models of human decisions for robotics and control

Ho¹,

Griffiths²

2021

Preprint

View full text Add to dashboard Cite

Those designing autonomous systems that interact with humans will invariably face questions about how humans think and make decisions. Fortunately, computational cognitive science offers insight into human decision-making using tools that will be familiar to those with backgrounds in optimization and control (e.g., probability theory, statistical machine learning, and reinforcement learning). Here, we review some of this work, focusing on how cognitive science can provide forward models of human decisionmaking and inverse models of how humans think about others' decision-making. We highlight relevant recent developments, including approaches that synthesize blackbox and theory-driven modeling, accounts that recast heuristics and biases as forms of bounded optimality, and models that characterize human theory of mind and communication in decision-theoretic terms. In doing so, we aim to provide readers with a glimpse of the range of frameworks, methodologies, and actionable insights that lie at the intersection of cognitive science and control research.

show abstract

Deep Reinforcement Learning from Policy-Dependent Human Feedback

Cited by 23 publications

References 12 publications

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Imitation Learning: Progress, Taxonomies and Opportunities

Cognitive science as a source of forward and inverse models of human decisions for robotics and control

Contact Info

Product

Resources

About