Using informative behavior to increase engagement while learning from human reward

Abstract

In this work, we address a relatively unexplored aspect of designing agents that learn from human reward. We investigate how an agent's non-task behavior can affect a human trainer's training and agent learning. We use the TAMER framework, which facilitates the training of agents by human-generated reward signals, i.e., judgements of the quality of the agent's actions, as the foundation for our investigation. Then, starting from the premise that the interaction between the agent and the trainer should be bi-directional, we propose two new training interfaces to increase a human trainer's active involvement in the training process and thereby improve the agent's task performance. One provides information on the agent's uncertainty which is a metric calculated as data coverage, the other on its performance. Our results from a 51-subject user study show that these interfaces can induce the trainers to train longer and give more feedback. The agent's performance, however, increases only in response to the addition of performance-oriented information, not by sharing uncertainty levels. These results suggest that the organizational maxim about human behavior, "you get what you measure"-i.e., sharing metrics with people causes them to focus on optimizing those metrics while de-emphasizing other objectives-also applies to the training of agents. Using principle component analysis, we show how trainers in the two conditions train agents differently. In addition, by simulating the influence of the agent's uncertainty-informative behavior on a human's training behavior, we show that trainers could be distracted by the agent sharing its uncertainty levels about its actions, giving poor feedback for the sake of reducing the agent's uncertainty without improving the agent's performance.

Learning from human reward benefits from socio-competitive feedback

“…In our experiment, 100 subjects consented to install the app within the first three days of this study. By contrast, our earlier experiment [7] obtained only 51 subjects using a more aggressive recruitment effort that included manually sending emails to potential subjects, putting up flyers and posters, and sending reminder emails.…”

“…Using TAMER as a foundation, Knox et al [11] examine how human trainers respond to changes in their perception of the agent and to certain changes in the agent's behavior, while Li et al [7] investigate how informative feedback from the agent affects trainers' behaviors. Knox et al find that the agent can induce the human trainer to give more feedback but with lower performance when the quality of the agent's behavior is deliberately reduced whenever the rate of human feedback decreases.…”

“…The control and performance conditions are replicated from our earlier work [7]. The passive and active social conditions are the novel conditions we propose in this paper.…”

“…In particular, we showed that if an agent keeps the trainer informed about the agent's past and current performance, the trainer will provide more feedback and the agent will ultimately perform better [7]. Hence, this result shows that the interaction between the agent and the trainer should ideally be bi-directional: not only should the trainer give the agent the feedback it needs for learning, the agent should explicitly give the trainer feedback on how well that learning is going.…”

“…While both the interface in [7] and the social extension mentioned above are bi-directional, the agent's role is passive: it merely displays feedback for the trainer, which the trainer can choose to look at or ignore. To address this limitation, we developed an extension to the Facebook app that uses Facebook notifications, i.e., messages sent to Facebook users while they are not using the app, that update the trainers on their performance relative to other trainers.…”