A User Study on Robot Skill Learning Without a Cost Function: Optimization of Dynamic Movement Primitives via Naive User Feedback

Abstract

Enabling users to teach their robots new tasks at home is a major challenge for research in personal robotics. This work presents a user study in which participants were asked to teach the robot Pepper a game of skill. The robot was equipped with a state-of-the-art skill learning method, based on dynamic movement primitives (DMPs). The only feedback participants could give was a discrete rating after each of Pepper's movement executions ("very good," "good," "average," "not so good," "not good at all"). We compare the learning performance of the robot when applying user-provided feedback with a version of the learning where an objectively determined cost via hand-coded cost function and external tracking system is applied. Our findings suggest that (a) an intuitive graphical user interface for providing discrete feedback can be used for robot learning of complex movement skills when using DMP-based optimization, making the tedious definition of a cost function obsolete; and (b) un-experienced users with no knowledge about the learning algorithm naturally tend to apply a working rating strategy, leading to similar learning performance as when using the objectively determined cost. We discuss insights about difficulties when learning from user provided feedback, and make suggestions how learning continuous movement skills from non-expert humans could be improved.

Interactive Robot Task Learning: Human Teaching Proficiency With Different Feedback Approaches

“…Therefore, we present a system that does not need a predefined cost function or feature representation, and can learn successful movement skills from non-expert users in a couple of minutes. In contrast to work done by Vollmer and Hemion [3], we have looked into the drawbacks of using absolute-scale feedback from users, which is influenced by a drift in evaluation and the requirement of anchoring to a reference point, by utilizing preference-based feedback from users.…”

“…The goal of the game is to catch the ball with the cup through skillful movement. Kober and Peters [19] have demonstrated that the cup-and-ball movement can be learned by a robot arm using DMP-based optimization, and Vollmer and Hemion [3] have demonstrated that Pepper is capable of mastering the game with human absolute scale ratings as reward. In this study, the cup-and-ball toy was built such that the size of the cup and ball resulted in a level of difficulty suitable for our purposes.…”

“…Therefore, this work investigates the applicability of an optimization system from a user-centered perspective and investigates what kind of user feedback for the optimization is intuitively usable without much effort. We base our work on recent work by Vollmer and Hemion [3], who have shown that naive users can teach robots complex continuous movement skills via a simple user interface. We here also concentrate on robot learning for complex movement skills with a human teacher and compare the types of feedback a teacher could give as a performance measure: feedback as star ratings on an absolute scale for single roll-outs (as in [3]) versus preference-based feedback for pairwise comparisons.…”

“…We base our work on recent work by Vollmer and Hemion [3], who have shown that naive users can teach robots complex continuous movement skills via a simple user interface. We here also concentrate on robot learning for complex movement skills with a human teacher and compare the types of feedback a teacher could give as a performance measure: feedback as star ratings on an absolute scale for single roll-outs (as in [3]) versus preference-based feedback for pairwise comparisons. In the following we will refer to the two conditions as 'absolute scale' and 'preference-based', respectively.…”

“…6]. This is supported by the number of users who were not able to successfully teach the robot, because their strategy was not compatible with the properties of the underlying learning algorithm in [3]. On the other hand, outside of interactive task learning, people have been shown to be very proficient at giving preference-based feedback and at comparing things [7].…”