To ensure safety and functional correctness of automated and autonomous driving systems, virtual scenariobased testing is used. Experts derive traffic scenario types and generate instances of these types with the support of test generation tools. Since driving systems operate in a real-world environment, it is always possible to find a new scenario type as well as new instances of scenario types that are different from all other scenario types and instances. Thus, the testing process to find faulty behavior may continue forever. There is a practical need for test ending criteria for both of the following problems: Did we test all scenario types? Did we sufficiently test each type with specific instances? We address the first question and present a suitable test ending criterion and methodology. Whether the system is tested in each scenario type is reduced to the question whether all test scenarios are known. We analyze driving data to provide a statistical guarantee that all scenario types are covered. We model this as a Coupon Collector's Problem. We present experimental results for the application of this model to different driving tasks of automated and autonomous driving systems.
Functional specifications and real drive data are typically used to derive parameterized scenarios for scenario-based testing of driving systems. The domains of the parameters span a huge space of possible test cases, from which "good" ones have to be selected. Heuristic search, guided by fitness functions, has been proposed as a suitable technique in the past. However, the methodological challenge of creating suitable fitness functions has not been addressed yet. We provide templates to formulate fitness functions for testing automated and autonomous driving systems. Those templates ensure correct positioning of scenario objects in space, yield a suitable ordering of maneuvers in time, and enable the search for scenarios in which the system leaves its safe operating envelope. We show how to compose them into fitness functions for heuristic search. Collision and close-to-collision scenarios from real drive data serve as a use case to show the applicability of the presented templates.
Many approaches for testing automated and autonomous driving systems in dynamic traffic scenarios rely on the reuse of test cases, e.g., recording test scenarios during real test drives or creating "test catalogs." Both are widely used in industry and in literature. By counterexample, we show that the quality of test cases is system-dependent and that faulty system behavior may stay unrevealed during testing if test cases are naïvely re-used. We argue that, in general, system-specific "good" test cases need to be generated. Thus, recorded scenarios in general cannot simply be used for testing, and regression testing strategies needs to be rethought for automated and autonomous driving systems. The counterexample involves a system built according to state-of-the-art literature, which is tested in a traffic scenario using a highfidelity physical simulation tool. Test scenarios are generated using standard techniques from the literature and state-of-theart methodologies. By comparing the quality of test cases, we argue against a naïve re-use of test cases. I. INTRODUCTIONStriving for highly automated and autonomous driving systems results in more and more complex and capable systems. The complexity of these systems as well as the complexity and sheer number of possible scenarios makes safety and functional correctness a crucial challenge [17]. Since testing by real test drives alone becomes practically infeasible [15], [33], the focus shifts to virtual test drives. For virtual testing of vehicle safety, scenario-based closed-loop testing in the form of X-in-the-loop settings is used [30]. Such scenarios usually contain dynamic traffic situations to test the behavior of automated and autonomous driving systems. An exemplary test scenario for testing a highway pilot is depicted in Fig. 1.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.