SimNet: Learning Reactive Self-driving Simulations from Real-world Observations

Abstract

In this work we present a simple end-to-end trainable machine learning system capable of realistically simulating driving experiences. This can be used for verification of self-driving system performance without relying on expensive and time-consuming road testing. In particular, we frame the simulation problem as a Markov Process, leveraging deep neural networks to model both state distribution and transition function. These are trainable directly from the existing raw observations without the need of any handcrafting in the form of plant or kinematic models. All that is needed is a dataset of historical traffic episodes. Our formulation allows the system to construct never seen scenes that unfold realistically reacting to the self-driving car's behaviour. We train our system directly from 1,000 hours of driving logs and measure both realism, reactivity of the simulation as the two key properties of the simulation. At the same time we apply the method to evaluate performance of a recently proposed state-of-the-art ML planning system [1] trained from human driving logs. We discover this planning system is prone to previously unreported causal confusion issues that are difficult to test by non-reactive simulation. To the best of our knowledge, this is the first work that directly merges highly realistic data-driven simulations with a closed loop evaluation for self-driving vehicles. We make the data, code, and pre-trained models publicly available to further stimulate simulation development.

TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Driving

“…To avoid generating collision trajectories, a common-sense loss is designed. Similarly, SimNet [14] uses a multi-modal trajectory predictor as the agent reactive behavior model to generate trajectories for background vehicles. RouteGAN [27] can generate diverse interaction behaviors by controlling the agents separately with desired styles and given final goals.…”

“…The common approaches are trajectory prediction or motion forecasting [9], [10] and reinforcement learning (RL) related methods [11], [12]. For the trajectory prediction model, the stochastic multi-modality behaviors can be captured based on the real-world data [13], [14]. However, the feasibility of generated trajectories cannot be guaranteed such as avoiding the unrealistic collision and satisfying vehicle kinematic constraints, as the agent is considered as a particle model.…”

Autonomy 2.0: Why is self-driving always 5 years away?

“…Simulation outcomes must be very realistic since any difference between simulation and reality would result in inaccurate performance estimates, but it does not need to be photo-realistic [29] and Figure 4: A data-driven reactive simulator [28] enables the synthesis of new, realistic driving scenarios based on a recorded log. This allows the training and evaluation self-driving stack offline, without the need for extensive road testing (sec.…”

“…We reason that that in order to achieve a high level of realism, the simulation itself must be learned directly from the real world. Recently, [28] showed how realistic and reactive simulations can be constructed from previously collected real-world logs using birds-eye-view representations. As shown in Fig.…”

SafetyNet: Safe planning for real-world self-driving vehicles using machine-learned policies

“…In terms of the ML planner, the presented approach is relatively simple, based on imitation learning. It can be improved by drawing from recent advancements in model-based Reinforcement Learning (RL) [36], offline RL [37], or closed-loop training in a data-driven simulation [38].…”