Applying Heuristics to Generate Test Cases for Automated Driving Safety Evaluation

Abstract

Comprehensive safety evaluation methodologies for automated driving systems that account for the large complexity real traffic are currently being developed. This work adopts a scenario-based safety evaluation approach and aims at investigating an advanced methodology to generate test cases by applying heuristics to naturalistic driving data. The targeted requirements of the generated test cases are severity, exposure, and realism. The methodology starts with the extraction of scenarios from the data and their split in two subsets—containing the relatively more critical scenarios and, respectively, the normal driving scenarios. Each subset is analysed separately, in regard to the parameter value distributions and occurrence of dependencies. Subsequently, a heuristic search-based approach is applied to generate test cases. The resulting test cases clearly discriminate between safety critical and normal driving scenarios, with the latter covering a wider spectrum than the former. The verification of the generated test cases proves that the proposed methodology properly accounts for both severity and exposure in the test case generation process. Overall, the current study contributes to fill a gap concerning the specific applicable methodologies capable of accounting for both severity and exposure and calls for further research to prove its applicability in more complex environments and scenarios.

Scenario Parameter Generation Method and Scenario Representativeness Metric for Scenario-Based Assessment of Automated Vehicles

“…Zofka et al [18] presented how recorded sensor data can be exploited to create scenarios that might lead to critical situations by modifying parameters of the recorded parameterized scenarios. Stepien et al [19] generate scenarios by sampling scenario parameter values from generalized extreme value distributions, where the distribution parameters are fitted using scenario parameter values extracted from safety-critical scenarios observed in naturalistic driving data. In [10,[20][21][22][23][24], also parameterized scenarios were generated and, in addition, importance sampling techniques were presented that automatically generate scenarios in which the system-under-test shows (safety-)critical behavior.…”

“…• The scenarios are oversimplified. For example, a vehicle's speed profile follows a predetermined functional form [10,19,21].…”

“…• Assumptions regarding the scenario parameter distributions are made that potentially compromise the quality of the scenarios. For example, the parameters are assumed to originate from a Gaussian [29] or generalized extreme value [19] distribution, and/or it is assumed that (some of) the parameters are uncorrelated [24]. • Because no pdf of the scenario parameters is known [18], no evaluation can be made of the performance of the system once deployed on the road, because it is unknown how realistic and likely the scenarios are.…”

Optimal method of extreme scenarios for intelligent driving vehicle testing based on time window

“…One of them is replicating frequently occurring situations encountered during real-life driving experiences [1] . The other kind of methods is generating rare but highly critical hazardous scenarios specifically [2,3,4] . Furthermore, in addition to creating fragmented scenarios, some experts also consider dynamic traffic flow when reproducing these test environments [5] .…”

Cut-Out Scenario Generation With Reasonability Foreseeable Parameter Range From Real Highway Dataset for Autonomous Vehicle Assessment

“…ADS capabilities to perform its intended function competently are often compared to competent and careful drivers [22,23]. Such drivers drive their vehicles safely carrying out perception, decision-making, and action implementation tasks with a high level of performance [24].…”