Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles

Mullins, Galen E.; Stankiewicz, Paul G.; Gupta, Satyandra K.

doi:10.1109/icra.2017.7989173

Cited by 77 publications

(31 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An approach to infer the vehicle intelligence level from a finite number of tests is presented in [28]; however, this work does not consider fully automatic test case generation. While S-TaLiRo offers many optimization engines to create interesting situations, other approaches focus on machine learning techniques: Interesting scenarios are grouped by unsupervised learning to find situations where small deviations of the environment lead to great performance variations in [29]. Combination and mutation of recorded data is used in [30] to create new test cases.…”

Section: Introductionmentioning

confidence: 99%

Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles

Althoff

Lutz

2018

2018 IEEE Intelligent Vehicles Symposium (IV)

105

View full text Add to dashboard Cite

It is apparent that one cannot rely solely on physical test drives for ensuring the correct functionality of autonomous vehicles. Since physical test drives are costly and time consuming, it is advantageous to accompany them with computer simulations. However, since most traffic scenarios are not challenging, even simulations are often too time consuming. To address this issue, we present an approach that creates automatically critical driving situations, i.e., situations with a small solution space for avoiding a collision. Our approach combines reachability analysis for determining the size of the solution space with optimization techniques to shrink it. The solution space is reduced by shifting the initial states of traffic participants, demanding an immediate and correct action of the vehicle under test. We demonstrate our approach by automatically increasing the criticality of several initially uncritical situations recorded from real traffic.

show abstract

Section: Introductionmentioning

confidence: 99%

Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles

Althoff

Lutz

2018

2018 IEEE Intelligent Vehicles Symposium (IV)

105

View full text Add to dashboard Cite

show abstract

“…He uses adaptive search-algorithms to iteratively generate new concrete scenarios based on previous results. Related work has already been published by the author in [134], [135]. Gangopadhyay et al [136] use a Bayesian optimization, [137] use a random forest model and Abbas et al [138] use simulated annealing in their test harness capable of testing perception algorithms.…”

Section: Simulation-based Falsificationmentioning

confidence: 99%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

show abstract

“…One straightforward approach for increasing the efficiency of virtual testing is the extraction and classification of relevant scenarios from large databases of recorded traffic data as demonstrated in, e.g., [5], [6]. In [7], scenarios are grouped by unsupervised learning to find situations where small deviations of the environment lead to changes in behavioral modes, e.g., when the vehicle under test is forced to take a different path. Despite the fact that collecting data at this scale is challenging, one is restricted to observed situations, which are typically not critical most of the time.…”

Section: A Related Workmentioning

confidence: 99%

Generating Critical Test Scenarios for Automated Vehicles with Evolutionary Algorithms

Klischat

Althoff

2019

2019 IEEE Intelligent Vehicles Symposium (IV)

116

View full text Add to dashboard Cite

Virtual testing of automated vehicles using simulations is essential during their development. When it comes to the testing of motion planning algorithms, one is mainly interested in challenging, critical scenarios for which it is hard to find a feasible solution. However, these situations are rare under usual traffic conditions, demanding an automatic generation of critical test scenarios. We present an approach that automatically generates critical scenarios based on a minimization of the solution space of the vehicle under test. By formulating a scenario parametrization and automatic determination of relevant parameter intervals, we are able to optimize the criticality of complex scenarios. We use evolutionary algorithms to tackle the resulting highly nonlinear optimization problem. Compared to our previous approach, we are now able to handle complex situations, in particular those involving intersections. Finally, we demonstrate our approach by generating critical scenarios from initially uncritical scenarios.All authors are with the Technische Universität München,

show abstract

Automated generation of diverse and challenging scenarios for test and evaluation of autonomous vehicles

Cited by 77 publications

References 14 publications

Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles

Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Generating Critical Test Scenarios for Automated Vehicles with Evolutionary Algorithms

Contact Info

Product

Resources

About