Dynamic Reward Shaping: Training a Robot by Voice

Tenorio-González, Ana Cecilia; Morales, Eduardo F.; Villaseñor-Pineda, Luis

doi:10.1007/978-3-642-16952-6_49

Cited by 63 publications

(63 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this learning scenario, feedback can be restricted to express various intensities of approval and disapproval; such feedback is mapped to numeric "reward" that the agent uses to revise its behavior [2], [3], [8], [1], [9]. Compared to learning from demonstration, learning from human reward requires only a simple task-independent interface and may require less expertise and place less cognitive load on the trainer [10].…”

Section: A Learning From Human Rewardsmentioning

confidence: 99%

Learning from human reward benefits from socio-competitive feedback

Hung

Whiteson

et al. 2014

4th International Conference on Development and Learning and on Epigenetic Robotics

View full text Add to dashboard Cite

Abstract-Learning from rewards generated by a human trainer observing an agent in action has proven to be a powerful method for non-experts in autonomous agents to teach such agents to perform challenging tasks. Since the efficacy of this approach depends critically on the reward the trainer provides, we consider how the interaction between the trainer and the agent should be designed so as to increase the efficiency of the training process. This paper investigates the influence of the agent's socio-competitive feedback on the human trainer's training behavior and the agent's learning. The results of our user study with 85 subjects suggest that the agent's sociocompetitive feedback substantially increases the engagement of the participants in the game task and improves the agents' performance, even though the participants do not directly play the game but instead train the agent to do so. Moreover, making this feedback active further induces more subjects to train the agents longer but does not further improve agent performance. Our analysis suggests that this may be because some trainers train a more complex behavior in the agent that is appropriate for a different performance metric that is sometimes associated with the target task.

show abstract

Section: A Learning From Human Rewardsmentioning

confidence: 99%

Learning from human reward benefits from socio-competitive feedback

Hung

Whiteson

et al. 2014

4th International Conference on Development and Learning and on Epigenetic Robotics

View full text Add to dashboard Cite

show abstract

“…Reward and punishment are frequently received in a social context, from another social agent. In recent years, this form of communication and its machine-learning analog-reinforcement learning-have been adapted to permit teaching of artificial agents by their human users [4,14,6,13,11,10]. In this form of teaching-which we call interactive shaping-a user observes an agent's behavior while generating human reward instances through varying interfaces (e.g., keyboard, mouse, or verbal feedback); each instance is received by the learning agent as a time-stamped numeric value and used to inform future behavioral choices.…”

Section: Introductionmentioning

confidence: 99%

“…Investigating the six previous projects that we know to have involved learning from positively and negatively valued humangenerated reward [4,14,13,11,9,10] (including by email with corresponding authors), we identified a curious trend: all such projects have been much more myopic-i.e., using high discount rates-than is usual in RL. We hypothesized that a cause of this pattern is the general positivity of human reward.…”

Section: Introductionmentioning

confidence: 99%

Learning non-myopically from human-generated reward

Knox

Stone

2013

Proceedings of the 2013 International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Recent research has demonstrated that human-generated reward signals can be effectively used to train agents to perform a range of reinforcement learning tasks. Such tasks are either episodic-i.e., conducted in unconnected episodes of activity that often end in either goal or failure statesor continuing-i.e., indefinitely ongoing. Another point of difference is whether the learning agent highly discounts the value of future reward-a myopic agent-or conversely values future reward appreciably. In recent work, we found that previous approaches to learning from human reward all used myopic valuation [7]. This study additionally provided evidence for the desirability of myopic valuation in task domains that are both goal-based and episodic.In this paper, we conduct three user studies that examine critical assumptions of our previous research: task episodicity, optimal behavior with respect to a Markov Decision Process, and lack of a failure state in the goal-based task. In the first experiment, we show that converting a simple episodic task to non-episodic (i.e., continuing) task resolves some theoretical issues present in episodic tasks with generally positive reward and-relatedly-enables highly successful learning with non-myopic valuation in multiple user studies. The primary learning algorithm in this paper, which we call "VI-TAMER", is the first algorithm to successfully learn nonmyopically from human-generated reward; we also empirically show that such non-myopic valuation facilitates higherlevel understanding of the task. Anticipating the complexity of real-world problems, we perform two subsequent user studies-one with a failure state added-that compare (1) learning when states are updated asynchronously with local bias-i.e., states quickly reachable from the agent's current state are updated more often than other states-to (2) learning with the fully synchronous sweeps across each state in the VI-TAMER algorithm. With these locally biased updates, we find that the general positivity of human reward creates problems even for continuing tasks, revealing a distinct research challenge for future work.

show abstract

“…Accordingly, other algorithms for learning from human reward [4,21,20,16,18,13] do not directly account for delay, do not model human reward explicitly, and are not fully myopic (i.e., they employ discount factors greater than 0).…”

Section: Background On Tamermentioning

confidence: 99%

“…Though a few past projects have considered this problem of learning from human reward [4,21,20,16,18,13,9], only two of these implemented their solution for a robotic agent. In one such project [13], the agent learned partially in simulation and from hardcoded reward, demonstrations, and human reward.…”

Section: Introductionmentioning

confidence: 99%

Training a Robot via Human Feedback: A Case Study

Knox

Stone

Breazeal

2013

Lecture Notes in Computer Science

106

View full text Add to dashboard Cite

Abstract. We present a case study of applying a framework for learning from numeric human feedback-TAMER-to a physically embodied robot. In doing so, we also provide the first demonstration of the ability to train multiple behaviors by such feedback without algorithmic modifications and of a robot learning from free-form human-generated feedback without any further guidance or evaluative feedback. We describe transparency challenges specific to a physically embodied robot learning from human feedback and adjustments that address these challenges.

show abstract

Dynamic Reward Shaping: Training a Robot by Voice

Cited by 63 publications

References 8 publications

Learning from human reward benefits from socio-competitive feedback

Learning from human reward benefits from socio-competitive feedback

Learning non-myopically from human-generated reward

Training a Robot via Human Feedback: A Case Study

Contact Info

Product

Resources

About