Facial feedback for reinforcement learning: a case study and offline analysis using the TAMER framework

Abstract

Interactive reinforcement learning provides a way for agents to learn to solve tasks from evaluative feedback provided by a human user. Previous research showed that humans give copious feedback early in training but very sparsely thereafter. In this article, we investigate the potential of agent learning from trainers' facial expressions via interpreting them as evaluative feedback. To do so, we implemented TAMER which is a popular interactive reinforcement learning method in a reinforcement-learning benchmark problem -Infinite Mario, and conducted the first large-scale study of TAMER involving 561 participants. With designed CNN-RNN model, our analysis shows that telling trainers to use facial expressions and competition can improve the accuracies for estimating positive and negative feedback using facial expressions. In addition, our results with a simulation experiment show that learning solely from predicted feedback based on facial expressions is possible and using strong/effective prediction models or a regression method, facial responses would significantly improve the performance of agents. Furthermore, our experiment supports previous studies demonstrating the importance of bi-directional feedback and competitive elements in the training interface.

A Survey of Collaborative Reinforcement Learning: Interactive Methods and Design Patterns

“…Li et al trained a mapping model that can map implicit emotions to different explicit feedback data. Facial expressions were marked with different types of feedback in advance, such as 1 for "happy", 0 or -1 for "sadness" [40]. Based on this work, Gadanho introduced a facial feedback reinforcement learning method, which is based on an emotion recognition system.…”

A Review on Interactive Reinforcement Learning From Human Social Feedback

“…Li et al trained a prediction model mapping the facial feedback to explicit keypress feedback with collected data. Their simulated experiment showed that with enough recognition accuracy, agents can learn a comparative performance from solely facial feedback compared to learning from explicit keypress feedback [50].…”

“…In addition, these natural interactive feedback can even be combined with hardware delivered feedback to train agents. For example, Li et al mapped the facial expressions to explicit keystroke feedback and proposed to allow an agent to learn from both the predicted facial feedback and keystroke feedback [50].…”

The EMPATHIC Framework for Task Learning from Implicit Human Feedback

“…Our work relates closely to the growing literature of interactive reinforcement learning (RL), or humancentered RL [2,21,22,23,24,25,26,27,28,29] , in which agents learn from interactions with humans in addition to, or instead of, predefined environmental rewards. In the EMPATHIC framework, we use the term implicit human feedback to refer to any multi-modal evaluative signals humans naturally emit during social interactions, including facial expressions, tone of voice, head gestures, hand gestures and other body-language and vocalization modalities not aimed at explicit communication.…”