Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology

Feng, Shuo; Feng, Yiheng; Yu, Chunhui; Zhang, Yi; Liu, Henry

doi:10.1109/tits.2020.2972211

Cited by 137 publications

(96 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where q(x) is called the importance function. By introducing importance functions, the testing priority of critical scenarios will be improved, so does the evaluation efficiency [14][15][16][17] . However, all existing IS-based methods suffer from the "curse of dimensionality" 19 , and thus cannot be applied directly for the complex driving environment.…”

Section: Theoremmentioning

confidence: 99%

“…Towards solving the inefficiency issue, scenario-based approaches have been proposed. Based on the importance sampling (IS) theory, critical scenarios can be purposely designed for accelerating the efficiency of AV evaluation [12][13][14][15][16][17] . However, existing scenario generation methods can only be applied for scenarios that involve simple maneuvers of a very limited number of vehicles with very short duration, for instance, a cut-in maneuver from a background vehicle for a few seconds.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

et al. 2021

Self Cite

View full text Add to dashboard Cite

Driving intelligence tests are critical to the development and deployment of autonomous vehicles. The prevailing approach tests autonomous vehicles in life-like simulations of the naturalistic driving environment. However, due to the high dimensionality of the environment and the rareness of safety-critical events, hundreds of millions of miles would be required to demonstrate the safety performance of autonomous vehicles, which is severely inefficient. We discover that sparse but adversarial adjustments to the naturalistic driving environment, resulting in the naturalistic and adversarial driving environment, can significantly reduce the required test miles without loss of evaluation unbiasedness. By training the background vehicles to learn when to execute what adversarial maneuver, the proposed environment becomes an intelligent environment for driving intelligence testing. We demonstrate the effectiveness of the proposed environment in a highway-driving simulation. Comparing with the naturalistic driving environment, the proposed environment can accelerate the evaluation process by multiple orders of magnitude.

show abstract

Section: Theoremmentioning

confidence: 99%

mentioning

confidence: 99%

Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The focus is on a standardized interface for reading in different data sources and processing them into a machine-readable format. A further framework for creating a database, called the Testing Scenario Library, is described in detail in [41], [42]. They also use the definitions for the different scenario types from the PEGASUS project.…”

Section: ) Scenario Databasementioning

confidence: 99%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

show abstract

“…These methods are generally designed to be calibrated and improved over time using the feedback from conducted tests. Lastly, reinforcement learning has been gaining more interest lately in the testing of autonomous systems [16], [17]. In reinforcement learning-based approaches, testing methods are designed to automatically learn the test case selection and prioritize the improvement of test quality and coverage.…”

Section: A Related Workmentioning

confidence: 99%

Autonomous Vehicles Scenario Testing Framework and Model of Computation: On Generation and Coverage

2021

View full text Add to dashboard Cite

Autonomous vehicle (AV) technology started to shift the perception of the transportation systems. However, for AVs to operate at their optimum capabilities, they need to go through a comprehensive testing and verification process. While a large amount of research and funding has been provided for solving this problem, there is still a lack of a systematic method to develop standardized tests that can be used to judge if the decision-making capability functions are within acceptable parameters. To that end, the tests need to cover all possible situations that an AV may run into. This paper focuses on defining the notion of coverage mathematically when using pseudo-randomly generated simulations for testing. The approach defines new equivalence relations between scenes, which are the systems' various states, to achieve this goal. Considering the substantial need for computation, even with the obtained coverage, we also introduce the mathematical definition of a sub-scene and additional strategies, such as expanding the equivalence classes of scenes and combining actors in scenes, to reduce the amount of testing required to certify AVs.

show abstract

Testing Scenario Library Generation for Connected and Automated Vehicles, Part I: Methodology

Cited by 137 publications

References 40 publications

Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Autonomous Vehicles Scenario Testing Framework and Model of Computation: On Generation and Coverage

Contact Info

Product

Resources

About