How Humans Teach Agents

Auton Agent Multi-Agent Syst

et al. 2020

Interactive reinforcement learning provides a way for agents to learn to solve tasks from evaluative feedback provided by a human user. Previous research showed that humans give copious feedback early in training but very sparsely thereafter. In this article, we investigate the potential of agent learning from trainers' facial expressions via interpreting them as evaluative feedback. To do so, we implemented TAMER which is a popular interactive reinforcement learning method in a reinforcement-learning benchmark problem -Infinite Mario, and conducted the first large-scale study of TAMER involving 561 participants. With designed CNN-RNN model, our analysis shows that telling trainers to use facial expressions and competition can improve the accuracies for estimating positive and negative feedback using facial expressions. In addition, our results with a simulation experiment show that learning solely from predicted feedback based on facial expressions is possible and using strong/effective prediction models or a regression method, facial responses would significantly improve the performance of agents. Furthermore, our experiment supports previous studies demonstrating the importance of bi-directional feedback and competitive elements in the training interface.

Section: Reinforcement Learning From Human Rewardmentioning

confidence: 99%

Facial feedback for reinforcement learning: a case study and offline analysis using the TAMER framework

Dibeklioğlu

Auton Agent Multi-Agent Syst

et al. 2020

“…Their results show that the gaze behavior reduces the redundancy of guidance from the human trainer but does not further improve the agent's learning. In addition, Knox et al [23] also examine how human trainers respond to changes in their perception of the agent and to certain changes in the agent's behavior and find that the agent can induce the human trainer to give more feedback but with lower performance when the quality of the agent's behavior is deliberately reduced whenever the rate of human feedback decreases.…”

Section: Learning From Human Rewardmentioning

confidence: 99%

Using informative behavior to increase engagement while learning from human reward

Auton Agent Multi-Agent Syst

Knox

et al. 2015

Self Cite

In this work, we address a relatively unexplored aspect of designing agents that learn from human reward. We investigate how an agent's non-task behavior can affect a human trainer's training and agent learning. We use the TAMER framework, which facilitates the training of agents by human-generated reward signals, i.e., judgements of the quality of the agent's actions, as the foundation for our investigation. Then, starting from the premise that the interaction between the agent and the trainer should be bi-directional, we propose two new training interfaces to increase a human trainer's active involvement in the training process and thereby improve the agent's task performance. One provides information on the agent's uncertainty which is a metric calculated as data coverage, the other on its performance. Our results from a 51-subject user study show that these interfaces can induce the trainers to train longer and give more feedback. The agent's performance, however, increases only in response to the addition of performance-oriented information, not by sharing uncertainty levels. These results suggest that the organizational maxim about human behavior, "you get what you measure"-i.e., sharing metrics with people causes them to focus on optimizing those metrics while de-emphasizing other objectives-also applies to the training of agents. Using principle component analysis, we show how trainers in the two conditions train agents differently. In addition, by simulating the influence of the agent's uncertainty-informative behavior on a human's training behavior, we show that trainers could be distracted by the agent sharing its uncertainty levels about its actions, giving poor feedback for the sake of reducing the agent's uncertainty without improving the agent's performance.

“…Using TAMER as a foundation, Knox et al [11] examine how human trainers respond to changes in their perception of the agent and to certain changes in the agent's behavior, while Li et al [7] investigate how informative feedback from the agent affects trainers' behaviors. Knox et al find that the agent can induce the human trainer to give more feedback but with lower performance when the quality of the agent's behavior is deliberately reduced whenever the rate of human feedback decreases.…”

Section: A Learning From Human Rewardsmentioning

confidence: 99%

Learning from human reward benefits from socio-competitive feedback

Hung

4th International Conference on Development and Learning and on Epigenetic Robotics

et al. 2014

Self Cite

Abstract-Learning from rewards generated by a human trainer observing an agent in action has proven to be a powerful method for non-experts in autonomous agents to teach such agents to perform challenging tasks. Since the efficacy of this approach depends critically on the reward the trainer provides, we consider how the interaction between the trainer and the agent should be designed so as to increase the efficiency of the training process. This paper investigates the influence of the agent's socio-competitive feedback on the human trainer's training behavior and the agent's learning. The results of our user study with 85 subjects suggest that the agent's sociocompetitive feedback substantially increases the engagement of the participants in the game task and improves the agents' performance, even though the participants do not directly play the game but instead train the agent to do so. Moreover, making this feedback active further induces more subjects to train the agents longer but does not further improve agent performance. Our analysis suggests that this may be because some trainers train a more complex behavior in the agent that is appropriate for a different performance metric that is sometimes associated with the target task.