Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences

Abstract

Reward functions are a common way to specify the objective of a robot. As designing reward functions can be extremely challenging, a more promising approach is to directly learn reward functions from human teachers. Importantly, data from human teachers can be collected either passively or actively in a variety of forms: passive data sources include demonstrations (e.g., kinesthetic guidance), whereas preferences (e.g., comparative rankings) are actively elicited. Prior research has independently applied reward learning to these different data sources. However, there exist many domains where multiple sources are complementary and expressive. Motivated by this general problem, we present a framework to integrate multiple sources of information, which are either passively or actively collected from human users. In particular, we present an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then actively probes the user with preference queries to zero-in on their true reward. This algorithm not only enables us combine multiple data sources, but it also informs the robot when it should leverage each type of information. Further, our approach accounts for the human’s ability to provide data: yielding user-friendly preference queries which are also theoretically optimal. Our extensive simulated experiments and user studies on a Fetch mobile manipulator demonstrate the superiority and the usability of our integrated framework.

Invariance in Policy Optimisation and Partial Identifiability in Reward Learning

“…Prior work has explored learning from expert behaviour and preferences (Ibarz et al, 2018;Palan et al, 2019;Bıyık et al, 2022;Koppol et al, 2020), or other multi-modal data sources (Tung et al, 2018;Jeon et al, 2020). One motivation is that different data sources may provide complementary reward information (Koppol et al, 2020), decreasing ambiguity.…”

Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks

“…Hence it is natural to integrate preference and action demonstration via a joint IRL framework (Palan et al, 2019;Bıyık et al, 2020), with a nice insight that these two sources of information are complementary under the IRL framework: "demonstrations provide a high-level initialization of the human's overall reward functions, while preferences explore specific, fine-grained aspects of it" (Bıyık et al, 2020). Therefore they use demonstrations to initialize a reward distribution, and refine the reward function with preference queries (Palan et al, 2019;Bıyık et al, 2020). Ibarz et al (2018) takes a different approach to combine demonstration and preference information, by using human demonstrations to pre-train the agent.…”

“…Under this framework, it is possible to develop a unified learning paradigm that accepts multiple types of human guidance. We start to notice efforts towards this goal (Abel et al, 2017;Waytowich et al, 2018;Goecks et al, 2019;Woodward et al, 2020;Najar et al, 2020;Bıyık et al, 2020).…”

Models of human preference for learning reward functions

“…Influential recent research has focused on reward learning from preferences over pairs of fixed-length trajectory segments. Nearly all of this recent work assumes that human preferences arise probabilistically from only the sum of rewards over a segment, i.e., the segment's partial return [9][10][11][12][13][14][15][16]. That is, these works assume that people tend to prefer trajectory segments that yield greater rewards during the segment.…”