Simulating Emergent Properties of Human Driving Behavior Using Multi-Agent Reward Augmented Imitation Learning

Abstract

Recent developments in multi-agent imitation learning have shown promising results for modeling the behavior of human drivers. However, it is challenging to capture emergent traffic behaviors that are observed in real-world datasets. Such behaviors arise due to the many local interactions between agents that are not commonly accounted for in imitation learning. This paper proposes Reward Augmented Imitation Learning (RAIL), which integrates reward augmentation into the multi-agent imitation learning framework and allows the designer to specify prior knowledge in a principled fashion. We prove that convergence guarantees for the imitation learning process are preserved under the application of reward augmentation. This method is validated in a driving scenario, where an entire traffic scene is controlled by driving policies learned using our proposed algorithm. Further, we demonstrate improved performance in comparison to traditional imitation learning algorithms both in terms of the local actions of a single agent and the behavior of emergent properties in complex, multi-agent settings.

DeepSIL: A Software-in-the-Loop Framework for Evaluating Motion Planning Schemes Using Multiple Trajectory Prediction Networks

“…This, however, is required for scenariobased verification methods. On the other hand, inverse reinforcement learning and imitation learning methods, such as GAIL [4] and variations thereof [5], [6], have been designed to actually control an automated vehicle and, thus, are capable of both, to comply with a scenario script book and to realistically mimic other road users. However, so far, these methods have mostly been tested on the NGSIM dataset [17], which comes with a limited variation of scenarios.…”

“…However, so far, these methods have mostly been tested on the NGSIM dataset [17], which comes with a limited variation of scenarios. Furthermore, several shortcomings, like performance issues or difficulties to work for multiple agents in parallel have been reported [6]. The main goal of DeepSIL is to provide a realistically simulated environment to evaluate a motion planning software under test (MPUT).…”

“…1 All authors are with the Institute of Measurement, Control and Microtechnology, Ulm University, D-89081 Ulm, Germany {jan.strohbeck, johannes-christian.mueller, adrian.holzbock, michael.buchholz}@uni-ulm.de realistically modeling the interaction between multiple traffic participants. Recently, deep learning methods like generative adversarial imitation learning (GAIL) [4] and variants thereof [5], [6], have been proposed to describe the behavior of other road participants more realistically. However, while we acknowledge the recent achievements, these approaches are still in their infancies and often suffer from poor generalization to multi-agent scenarios and performance issues [6].…”

“…Recently, deep learning methods like generative adversarial imitation learning (GAIL) [4] and variants thereof [5], [6], have been proposed to describe the behavior of other road participants more realistically. However, while we acknowledge the recent achievements, these approaches are still in their infancies and often suffer from poor generalization to multi-agent scenarios and performance issues [6]. Furthermore, these approaches have mostly been applied to the NGSIM dataset [7] only, which covers a limited spectrum of possible traffic scenarios.…”

Reactive and Safe Road User Simulations using Neural Barrier Certificates

“…Such non-reactive or handcrafted agents are unrealistic, and therefore create a big gap between simulators and realworld scenarios. On the other side, imitation learning (IL) techniques [5], [6], [7], [8], [9] try to "mimic" humans from the existing traffic datasets, by learning actions or reward functions instead of directly enforcing some basic safety properties of the road users (e.g. avoiding collisions, staying within lanes and below speed limits).…”