AV-FUZZER: Finding Safety Violations in Autonomous Driving Systems

scenoRITA: Generating Less-Redundant, Safety-Critical and Motion Sickness-Inducing Scenarios for Autonomous Vehicles

“…Figure 2(b) demonstrates scenoRITA's application of a crossover operator on two individuals (i.e., obstacles) compared to how related work recombine their individuals (Figure 2(c)). Previous approaches [37,51,67], represent obstacles as genes, resulting in obstacles being partially mutable during recombination and mutation operators. For example, the crossover operator in AV-Fuzzer [67] does not alter properties of obstacles, instead it simply swaps two randomly selected obstacles in two scenarios, Scenario B and Scenario D, with a certain probability.…”

“…scenoRITA combines both (i) AV software domain knowledge and (ii) search-based testing [49,69]. These two elements have been combined by previous techniques to test AVs by automatically generating safety-critical scenarios [30,31,35,37,51,67]. However, unlike these approaches, scenoRITA's gene representation enables obstacles to be fully mutable, i.e., an obstacle's individual properties such as its start and end location, type (e.g., vehicle, pedestrian, and bike), heading, speed, size, and mobility (e.g., static or dynamic) can be altered.…”

“…However, unlike these approaches, scenoRITA's gene representation enables obstacles to be fully mutable, i.e., an obstacle's individual properties such as its start and end location, type (e.g., vehicle, pedestrian, and bike), heading, speed, size, and mobility (e.g., static or dynamic) can be altered. While previous techniques do not specify their gene representations or do so in such a way that allows obstacles to be only partially mutable, obstacles' attributes are altered only during mutation and with a small probability, while during crossover, obstacles are transferred across scenarios without altering their states or properties [37,50,51,67]. Thus, these techniques ignore the challenge of ensuring creation of valid obstacle trajectories, reducing their effectiveness at generating driving scenarios with unique violations.…”

“…To the best of our knowledge, scenoRITA is the first search-based testing technique for AV software that uses multiple test oracles at the same time and considers both comfort and safety violations as part of those oracles. • We evaluate the efficiency and effectiveness of scenoRITA by comparing it with random search and a state-of-the art searchbased approach-which adopts the gene representation in prior work [37,51,67] that allows only partial mutation of obstacles (i.e., obstacles are transferred across scenarios without altering their states or properties during crossover). Our extensive evaluation-which consists of executing a total of 31,413 virtual tests on Baidu Apollo using 3 high-definition maps of cities/street blocks located in California: Sunnyvale (3,061 lanes); San Mateo (1,305 lanes); and Borregas Ave (60 lanes)-shows that scenoRITA generates driving scenarios that expose the ego car to critical and realistic situations.…”

Seven Technical Issues That May Ruin Your Virtual Tests for ADAS

“…They make it possible to consider a wide range of operational conditions, in order to identify the safety-relevant corner cases. Related work in the area has proposed test generation approaches based on combinatorial testing [7], metaheuristic search [8,9], machine learning [10] or a hybridization of metaheuristic search and machine learning [11].…”