BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems

Abstract

Testing Autonomous Driving Systems (ADSs) is a critical task for ensuring the reliability and safety of autonomous vehicles. Existing methods mainly focus on searching for safety violations while the diversity of the generated test cases is ignored, which may generate many redundant test cases and failures. Such redundant failures can reduce testing performance and increase failure analysis costs.In this paper, we present a novel behavior-guided fuzzing technique (BehAVExplor) to explore the different behaviors of the ego vehicle (i.e., the vehicle controlled by the ADS under test) and detect diverse violations. Specifically, we design an efficient unsupervised model, called BehaviorMiner, to characterize the behavior of the ego vehicle. BehaviorMiner extracts the temporal features from the given scenarios and performs a clustering-based abstraction to group behaviors with similar features into abstract states. A new test case will be added to the seed corpus if it triggers new behaviors (e.g., cover new abstract states). Due to the potential conflict between the behavior diversity and the general violation feedback, we further propose an energy mechanism to guide the seed selection and the mutation. The energy of a seed quantifies how good it is. We evaluated BehAVExplor on Apollo, an industrial-level ADS, and LGSVL simulation environment. Empirical evaluation results show that BehAVExplor can effectively find more diverse violations than the state-of-the-art. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

TM-fuzzer: fuzzing autonomous driving systems through traffic management

“…Currently, the more popular method is fuzz testing based on multi-objective optimization (Huai et al, 2023;Cheng et al, 2023;Kim et al, 2022;Li et al, 2020;Tian et al, 2022). The general process of such fuzz testing is as follows:…”

“…Building upon AV-Fuzzer, Hu et al (2021) developed ASF to enhance fuzz testing diversity through trajectory coverage. Cheng et al (2023) defined trajectory diversity in BehAVExplor to encourage more varied scenario discovery. Tang et al (2023) aimed to include road structure in critical scenario searching for testing ADS with EvoScenario, further expanding the search space.…”

“…In this context, researchers have proposed many violated scenario search methods (Han and Zhou, 2020;Haq et al, 2022;Zhou et al, 2023a,b), such as guided searching methods (Li et al, 2020;Cheng et al, 2023;Tang et al, 2023), data-based methods Li et al (2022); Hou et al (2023); Priisalu et al (2022), agent-based methods (Huai et al, 2023;Kim et al, 2022;Feng et al, 2023). These methods involve identifying security-critical scenarios within an infinite scenario space that are likely to result in violations, through various risk assessment indicators (e.g., vehicle distance) (Li et al, 2020), and conducting ADS simulation tests.…”