Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles

Mullins, Galen E.; Stankiewicz, Paul G.; Hawthorne, Ryan; Gupta, Satyandra K.

doi:10.1016/j.jss.2017.10.031

Cited by 103 publications

(67 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we present a novel approach to finding safety-critical scenarios by identifying the performance boundary of an AV. The performance boundary separates the scenario space into regions according to the outcome of the scenario [7]. The outcome of a scenario can be quantitatively judged by different criticality metrics, such as the frequently used Time-to-Collision (TTC) [8].…”

Section: Introductionmentioning

confidence: 99%

Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

Batsch

Daneshkhah

Cheah

et al. 2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

View full text Add to dashboard Cite

Safety is an essential aspect in the facilitation of automated vehicle deployment. Current testing practices are not enough, and going beyond them leads to infeasible testing requirements, such as needing to drive billions of kilometres on public roads. Automated vehicles are exposed to an indefinite number of scenarios. Handling of the most challenging scenarios should be tested, which leads to the question of how such corner cases can be determined. We propose an approach to identify the performance boundary, where these corner cases are located, using Gaussian Process Classification. We also demonstrate the classification on an exemplary traffic jam approach scenario, showing that it is feasible and would lead to more efficient testing practices.

show abstract

Section: Introductionmentioning

confidence: 99%

Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

Batsch

Daneshkhah

Cheah

et al. 2019

2019 IEEE Intelligent Transportation Systems Conference (ITSC)

View full text Add to dashboard Cite

show abstract

“…Cluster analysis aims at grouping objects into clusters so that the similarity between two objects is maximal if they belong to the same cluster and minimal otherwise (Kaufman and Rousseeuw 1990). Popular clustering algorithms include Hierarchical Agglomerative clustering (Kaufman and Rousseeuw 1990), K-means++ (Arthur and Vassilvitskii 2007), Kmedoids (Kaufman and Rousseeuw 1987), and Gaussian Mixture Models (McLachlan and Basford 1988).…”

Section: Classification Vs Regressionmentioning

confidence: 99%

“…7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;Wolschke et al 2017;Fremont et al 2019), am...…”

Section: Domains (Rq 13)mentioning

confidence: 99%

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

show abstract

“…It includes a scenarios verification module, a method of model transformation (defined as mapping rules) and criteria for browsing the reachability tree of Petri-Nets to generate scenarios. Then, Mullins & al., 34 developed a testing method of autonomous vehicles, which deals with the issues of the dimensionality of the configuration space and the computational expense of high-fidelity simulations. The method is focused on finding performance boundaries of the system to generate challenging scenarios.…”

Section: Simulation Architecture For Safety Validationmentioning

confidence: 99%

Safety Demonstration of Autonomous Vehicles: A Review and Future Research Questions

Koné¹,

Bonjour²,

Levrat³

et al. 2019

Complex Systems Design &Amp; Management

View full text Add to dashboard Cite

The safety demonstration and validation of Autonomous vehicles (AVs) remains a challenging activity. In this paper, we firstly review what those challenges are and how they affect the safety validation of the AV. Then, we particularly focus on the simulation-based validation process, which seems to be inevitable among the recommended safety validation approaches. We show what is actually done and required in terms of scenarios generation, their assessment taking into account uncertainty and the simulation architecture to test and validate them. Finally, we end our review by summarizing key research questions that need to be addressed to help with this safety validation issue. 1 Introduction. An automated vehicle (AV) is a vehicle, which is able, according to the conditions of its operating environment and the level of automation, to move with or without human intervention. The Society of Automotive Engineers 1 (SAE) identifies six levels of automation: No Automation (Level 0), Driver Assistance (Level 1), Partial Automation (Level 2), Conditional Automation (Level 3), High Automation (Level 4), and Full Automation (Level 5). For its operation, an automated vehicle collects information about its environment, processes them, plans its trajectory and decides on actions to be performed. To implement this, manufacturers use specific technologies such as sensors and localization systems, communication systems and intelligent control systems. These embedded technologies are sometimes new, difficult to specify and have functional performance limitations regarding environmental conditions. This affects standard safety validation procedures, which face new challenges and are

show abstract

Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles

Cited by 103 publications

References 12 publications

Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification

Testing machine learning based systems: a systematic mapping

Safety Demonstration of Autonomous Vehicles: A Review and Future Research Questions

Contact Info

Product

Resources

About