The EMPATHIC Framework for Task Learning from Implicit Human Feedback

Cui, Yuchen; Zhang, Qiping; Allievi, Alessandro; Stone, Peter; Niekum, Scott; Knox, W. Bradley

doi:10.48550/arxiv.2009.13649

Cited by 6 publications

(5 citation statements)

References 44 publications

(66 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For fluency, however, new roles of interaction are proposed and discussed, which would also require new kinds of interface. An agent could integrate implicit empathic feedback from a human in the form of gestures, vocalizations, and facial expressions as shown by Cui et al (2020), allowing a more intuitive interaction and richer feedback from the human to the learning agent. To effectively leverage such feedback, appropriate user interfaces should be developed, and the underlying model should be able to process multimodal data such as speech or image.…”

Section: Focus: Interaction Designmentioning

confidence: 99%

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Retzlaff,

Das,

Wayllace

et al. 2024

jair

View full text Add to dashboard Cite

Artificial intelligence (AI) and especially reinforcement learning (RL) have the potential to enable agents to learn and perform tasks autonomously with superhuman performance. However, we consider RL as fundamentally a Human-in-the-Loop (HITL) paradigm, even when an agent eventually performs its task autonomously. In cases where the reward function is challenging or impossible to define, HITL approaches are considered particularly advantageous. The application of Reinforcement Learning from Human Feedback (RLHF) in systems such as ChatGPT demonstrates the effectiveness of optimizing for user experience and integrating their feedback into the training loop. In HITL RL, human input is integrated during the agent’s learning process, allowing iterative updates and fine-tuning based on human feedback, thus enhancing the agent’s performance. Since the human is an essential part of this process, we argue that human-centric approaches are the key to successful RL, a fact that has not been adequately considered in the existing literature. This paper aims to inform readers about current explainability methods in HITL RL. It also shows how the application of explainable AI (xAI) and specific improvements to existing explainability approaches can enable a better human-agent interaction in HITL RL for all types of users, whether for lay people, domain experts, or machine learning specialists. Accounting for the workflow in HITL RL and based on software and machine learning methodologies, this article identifies four phases for human involvement for creating HITL RL systems: (1) Agent Development, (2) Agent Learning, (3) Agent Evaluation, and (4) Agent Deployment. We highlight human involvement, explanation requirements, new challenges, and goals for each phase. We furthermore identify low-risk, high-return opportunities for explainability research in HITL RL and present long-term research goals to advance the field. Finally, we propose a vision of human-robot collaboration that allows both parties to reach their full potential and cooperate effectively.

show abstract

Section: Focus: Interaction Designmentioning

confidence: 99%

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Retzlaff,

Das,

Wayllace

et al. 2024

jair

View full text Add to dashboard Cite

show abstract

“…In [25] human feedback is used to remove bias of extracted skills from offline datasets sets and produce more human-aligned skills. Human feedback can also take more subtle forms, such as implicitly from facial features to learn reward rankings [26]. Human-robot collaborative manipulation policies can also be learned from datasets of human-human collaboration, such as learning handover tasks from conversations to obtain diverse strategies of human-robot collaboration [27], or to improve backchanneling behaviours [28] for social robots from behaviours such as nodding.…”

Section: Related Workmentioning

confidence: 99%

Aligning Human Preferences with Baseline Objectives in Reinforcement Learning

Marta

Holk

Pek

et al. 2023

2023 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Practical implementations of deep reinforcement learning (deep RL) have been challenging due to an amplitude of factors, such as designing reward functions that cover every possible interaction. To address the heavy burden of robot reward engineering, we aim to leverage subjective human preferences gathered in the context of human-robot interaction, while taking advantage of a baseline reward function when available. By considering baseline objectives to be designed beforehand, we are able to narrow down the policy space, solely requesting human attention when their input matters the most. To allow for control over the optimization of different objectives, our approach contemplates a multi-objective setting. We achieve human-compliant policies by sequentially training an optimal policy from a baseline specification and collecting queries on pairs of trajectories. These policies are obtained by training a reward estimator to generate Pareto optimal policies that include human preferred behaviours. Our approach ensures sample efficiency and we conducted a user study to collect real human preferences, which we utilized to obtain a policy on a social navigation environment.

show abstract

“…The human subjects then provide input to some algorithm that has no knowledge of the performance metric, and this algorithm or learned model is evaluated on how well its output performs with respect to the hidden metric. For another example, see Cui et al [27].…”

Section: D4 the Study Design Patternmentioning

confidence: 99%

Models of human preference for learning reward functions

Knox¹,

Hatgis-Kessell²,

Booth³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments. These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling preferences instead as arising from a different statistic: each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences. We also prove that the previous partial return model lacks this identifiability property without preference noise that reveals rewards' relative proportions, and we empirically show that our proposed regret preference model outperforms it with finite training data in otherwise the same setting. Additionally, our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research.

show abstract

The EMPATHIC Framework for Task Learning from Implicit Human Feedback

Cited by 6 publications

References 44 publications

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Aligning Human Preferences with Baseline Objectives in Reinforcement Learning

Models of human preference for learning reward functions

Contact Info

Product

Resources

About