Active Preference-Based Learning of Reward Functions

Sadigh, Dorsa; Dragan, Anca D.; Sastry, S. Shankar; Seshia, Sanjit A.

doi:10.15607/rss.2017.xiii.053

Cited by 220 publications

(320 citation statements)

References 15 publications

Supporting

Mentioning

303

Contrasting

Order By: Relevance

“…In contrast to related POMDP formulations, the exploration-exploitation trade-off is not addressed yet and is encoded only by a linear combination of objectives in the reward function. By contrast, the weights of the reward function can also be found by having a human driver choose a preferred trajectory iteratively from a set of two candidate trajectories (109). This allows the vehicle to learn the reward function without a set of expert trajectories and predefined labels.…”

Section: Wwwannualreviewsorg • Decision-making For Autonomousmentioning

confidence: 99%

Planning and Decision-Making for Autonomous Vehicles

Schwarting

Alonso–Mora

Rus

2018

Annu. Rev. Control Robot. Auton. Syst.

712

378

View full text Add to dashboard Cite

In this review, we provide an overview of emerging trends and challenges in the field of intelligent and autonomous, or self-driving, vehicles. Recent advances in the field of perception, planning, and decision-making for autonomous vehicles have led to great improvements in functional capabilities, with several prototypes already driving on our roads and streets. Yet challenges remain regarding guaranteed performance and safety under all driving circumstances. For instance, planning methods that provide safe and systemcompliant performance in complex, cluttered environments while modeling the uncertain interaction with other traffic participants are required. Furthermore, new paradigms, such as interactive planning and end-to-end learning, open up questions regarding safety and reliability that need to be addressed. In this survey, we emphasize recent approaches for integrated perception and planning and for behavior-aware planning, many of which rely on machine learning. This raises the question of verification and safety, which we also touch upon. Finally, we discuss the state of the art and remaining challenges for managing fleets of autonomous vehicles. 8.1

show abstract

Section: Wwwannualreviewsorg • Decision-making For Autonomousmentioning

confidence: 99%

Planning and Decision-Making for Autonomous Vehicles

Schwarting

Alonso–Mora

Rus

2018

Annu. Rev. Control Robot. Auton. Syst.

712

378

View full text Add to dashboard Cite

show abstract

“…These demonstrations are used to generate queries, to which the human responds by selecting her preferred trajectory (bottom). learning (IRL) [1,35,46], where we learn a reward function directly from expert demonstrations of the task, and preference-based learning [17,39], where we learn a reward function by repeatedly asking a human to pick between two trajectories. While these methods have found some success, they still struggle in practice, especially in robotics.…”

Section: Introductionmentioning

confidence: 99%

Learning Reward Functions by Integrating Human Demonstrations and Preferences

Palan¹,

Shevchuk²,

Landolfi³

et al. 2019

Robotics: Science and Systems XV

Self Cite

View full text Add to dashboard Cite

Our goal is to accurately and efficiently learn reward functions for autonomous robots. Current approaches to this problem include inverse reinforcement learning (IRL), which uses expert demonstrations, and preference-based learning, which iteratively queries the user for her preferences between trajectories. In robotics however, IRL often struggles because it is difficult to get high-quality demonstrations; conversely, preference-based learning is very inefficient since it attempts to learn a continuous, high-dimensional function from binary feedback. We propose a new framework for reward learning, DemPref, that uses both demonstrations and preference queries to learn a reward function. Specifically, we (1) use the demonstrations to learn a coarse prior over the space of reward functions, to reduce the effective size of the space from which queries are generated; and (2) use the demonstrations to ground the (active) query generation process, to improve the quality of the generated queries. Our method alleviates the efficiency issues faced by standard preference-based learning methods and does not exclusively depend on (possibly low-quality) demonstrations. In numerical experiments, we find that DemPref is significantly more efficient than a standard active preference-based learning method. In a user study, we compare our method to a standard IRL method; we find that users rated the robot trained with DemPref as being more successful at learning their desired behavior, and preferred to use the DemPref system (over IRL) to train the robot.

show abstract

“…Note that we do not consider these a contribution of our work: we choose the simplest approximations that facilitate tractability. There are many methods for approximate inference of θ studied in the literature that could be used for the joint (θ, β) spaces as well, from Metropolis Hastings [16], [39], to acquiring an MLE only via importance sampling of the partition function [6] or via a Laplace approximation [40].…”

Section: B Approximationmentioning

confidence: 99%