Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments

Abstract

Testing with simulation environments help to identify critical failing scenarios emerging autonomous systems such as self-driving cars (SDCs) and are safer than in-field operational tests. However, these tests are very expensive and are too many to be run frequently within limited time constraints. In this paper, we investigate test case prioritization techniques to increase the ability to detect SDC regression faults with virtual tests earlier.Our approach, called SDC-Prioritizer, prioritizes virtual tests for SDCs according to static features of the roads used within the driving scenarios. These features are collected without running the tests and do not require past execution results. SDC-Prioritizer utilizes meta-heuristics to prioritize the test cases using diversity metrics (black-box heuristics) computed on these static features. Our empirical study conducted in the SDC domain shows that SDC-Prioritizer doubles the number of safety-critical failures that virtual tests can detect at the same level of execution time compared to baselines: random and greedy-based test case orderings. Furthermore, this meta-heuristic search performs statistically better than both baselines in terms of detecting safety-critical failures. SDC-Prioritizer effectively prioritize test cases for SDCs with a large improvement in fault detection while its overhead (up to 0.34% of the test execution cost) is negligible.CCS Concepts: • Software and its engineering → Search-based software engineering; Software testing and debugging.

Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles

“…Euclidean diversity (𝑑𝑖𝑣 𝐸𝑢 ) is the measure of the distance between two test instances in an Euclidean space. This measure of diversity has widely been used as an adequacy metric in the testing of AVs [13,14,39,49,50]. We calculate 𝑑𝑖𝑣 𝐸𝑢 following the approach adopted in [50], which is summarised below.…”

“…For this, using the methodology explained in the above sections, we create an instance space and present its important components in Figure 2. The test suite for the generation of this instance space is obtained from a previous study on test-case prioritisation for AVs [13]. Detailed information about the features of these test scenarios is provided in Section 4.2, where we discuss the dataset and other relevant aspects of our experiments.…”

“…The current study employs test suites from two previous studies on search-based test selection and prioritisation for AVs [13,49].…”

“…Another widely used adequacy measure for traditional and AIbased systems is test suite diversity computed on test inputs or outputs [5,13,29,50]. The diversity metrics are designed based on the intuition that similar test cases exercise similar parts of the source code or training examples, thus revealing the same faults.…”

Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation

“…Recently, DNN testing has become a very active research area [3,4,30,44], with quite a few techniques proposed to ensure the quality of DNN from different angles. As a very early testing technique, DeepXplore, proposed by Pei et al, first defined the concept of neuron coverage and utilized it as a measure to generate test data to test DNN models.…”