Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers

Abstract

Abstract-The safety of Automated Vehicles (AVs) must be assured before their release and deployment. The current approach to evaluation relies primarily on (i) testing AVs on public roads or (ii) track testing with scenarios defined in a test matrix. These two methods have completely opposing drawbacks: the former, while offering realistic scenarios, takes too much time to execute; the latter, though it can be completed in a short amount of time, has no clear correlation to safety benefits in the real world. To avoid the aforementioned problems, we propose Accelerated Evaluation, focusing on the car-following scenario. The stochastic human-controlled vehicle (HV) motions are modeled based on 1.3 million miles of naturalistic driving data collected by the University of Michigan Safety Pilot Model Deployment Program. The statistics of the HV behaviors are then modified to generate more intense interactions between HVs and AVs to accelerate the evaluation procedure. The Importance Sampling theory was used to ensure that the safety benefits of AVs are accurately assessed under accelerated tests. Crash, injury and conflict rates for a simulated AV are simulated to demonstrate the proposed approach. Results show that test duration is reduced by a factor of 300 to 100,000 compared with the non-accelerated (naturalistic) evaluation. In other words, the proposed techniques have great potential for accelerating the AV evaluation process.

Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

“…It was found that the evaluation time of a car following scenario can be reduced by up to a factor of 100,000, and for a lane change scenario by up to 20,000 [6], [13]. In [2], importance sampling was used to evaluate an adaptive cruise control system.…”

“…The general disadvantage of variance reduction techniques is their requirement of prior knowledge in order to shape the probability distribution used for sampling. Often this data is taken from accident databases or naturalistic driving trails [6]. This runs the risk however to exclude scenarios that emerge due to the new automation technology, which is often not considered in accident databases or naturalistic driving trails.…”

Evaluation Uncertainty in Data-Driven Self-Driving Testing

“…We use ξ i to denote a realization of the random vector ξ. In Accelerated Evaluation approaches [1], [5], [7], the traffic environment is considered an uncertainty for the AV system under study and is represented by ξ.…”

“…By properly We gratefully acknowledge the support from the National Science Foundation under grants CAREER CMMI-1653339/1834710, IIS-1849280, and IIS-1849304. 1 integrating the Monte Carlo approach into the modeling scheme, researchers have been able to estimate the risk from driving situations based on control and trajectory prediction [3], [4] and corner cases in various driving scenarios [1], [5].…”

Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology

“…Input: SM and offline generated library Output: Customized library 1 Step 1: Observe f with initial testing scenarios. (Sec 4.A) 2 Step 2: while the stop criteria (e.g., budget or estimation precision) is not satisfied do 3 Step 2.1: Obtain the estimationf (Sec 4.B); 4 Step 2.2: Update SM and library (Sec 4.C); 5 Step 2.3: Decide next iteration of testing scenarios (Sec 4.D); 6 Step 2.4: Observe f by testing the CAV with new scenarios; 7 end…”