Applying affective feedback to reinforcement learning in ZOEI, a comic humanoid robot

Reinforcement Learning Approaches in Social Robotics

“…We identify three types of evaluation methodologies: (1) an evaluation from the algorithm point of view, (2) evaluation and assessment of user experience-related subjective measures, and (3) evaluation of both learning algorithm-related factors and user experience-related factors. Several studies only reported the self-rated questionnaire results [45] or user opinions [55]. There are also studies which do not include any evaluation, and only a short discussion regarding the learned policy [53,57,100,101] The cumulative collected reward over time is the most commonly used evaluation method.…”

“…Q-learning, along with its different variations, is the most commonly used RL method in social robotics. The studies using Q-learning are [ 3 , 13 , 34 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 ]. These comprise studies using standard Q-learning [ 3 , 54 , 55 , 58 , 60 , 62 ], studies modify Q-learning for dealing with delayed reward [ 52 ], tuning the parameters for Q-learning such as [ 13 , 34 , 52 ], dealing with decreasing human feedback over time [ 52 ], comparing their proposed algorithm with Q-learning [ 33 , 49 , 61 , 63 , 64 ], variation of Q-learning called Object Q-learning [ 64 , 65 , 66 ], combining Q-learning with fuzzy inference [ 67 ], SARSA [ 68 , 69 ], TD( ) [ 70 ], MAXQ [ 33 , 71 , 72 ], R-learning [ 32 ], and Deep Q-learning [ 35 , 36 , 73 , 74 ].…”

“…The authors reported that the trained system performed better and it was found to be faster at detecting user engagement than the hand-coded one, while the latter was more stable. In [ 55 , 57 , 59 ], the authors investigated the entertainment capabilities of social robots using RL. Ritschel et al [ 57 ] presented a social-cues-driven Q-learning approach for adapting the Reeti robot to keep the user engaged during the interaction.…”

Autonomous animation of humanoid robots

“…These metrics can be incorporated in creating the audience ratings for motions. Feedback from the audience ranges from using visual cues such as the audience holding colored markers such as paddles [Knight et al, 2011] to indicate their preference or audio feedback such as the applause or cheers from the audience or surveys at the end of the interaction [Addo and Ahamed, 2014]. These feedback can be converted into a noisy numerical value to model the distribution of the observed audience preference.…”

“…The approach of Knight et al assumes that the audience preference on features do not vary over time, whereas we model the boredom of the audience and account for different weights assigned to the motions in different sequences. Addo and Ahamed also use a robot to tell jokes, but use reinforcement learning [Addo and Ahamed, 2014]. However, to learn a good policy, they have to explore all jokes in all the states, whereas we do not have to query all sequences to pick the best sequence.…”