Semantically Adversarial Driving Scenario Generation with Explicit Knowledge Integration

Abstract

Deep Generative Models (DGMs) are known for their superior capability in generating realistic data. Extending purely data-driven approaches, recent specialized DGMs may satisfy additional controllable requirements such as embedding a traffic sign in a driving scene, by manipulating patterns implicitly in the neuron or feature level. In this paper, we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve semantically controllable scene generation. We categorize our knowledge into two types to be consistent with the composition of natural scenes, where the first type represents the property of objects and the second type represents the relationship among objects. We then propose a tree-structured generative model to learn complex scene representation, whose nodes and edges are naturally corresponding to the two types of knowledge respectively. Knowledge can be explicitly integrated to enable semantically controllable scene generation by imposing semantic rules on properties of nodes and edges in the tree structure. We construct a synthetic example to illustrate the controllability and explainability of our method in a clean setting. We further extend the synthetic example to realistic autonomous vehicle driving environments and conduct extensive experiments to show that our method efficiently identifies adversarial traffic scenes against different state-of-the-art 3D point cloud segmentation models satisfying the traffic rules specified as the explicit knowledge.Preprint. Under review.

A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

“…Directly generating high-fidelity sensing data is quite difficult, and there are an increasing number of works focused on this area [85]. An alternating method is leveraging differential renderer [86], [87] and LiDAR simulator [43], [76] to generate high-dimensional data with ray casting algorithms. If we want to evaluate a motion planning or control system, we can turn to lower dimensions.…”

“…They use RL methods to generate diverse scenarios and show improvement of the AV trained in their scenarios. [76] represent the explicit knowledge (e.g., the vehicles should not have overlap, the orientation of vehicles should follow the direction of the lane) as first-orderlogic [131], which can be embedded into a tree structure. Then they search in the latent space of a VAE model and apply the knowledge to the tree encoder to constraint the searching process.…”

CausalAF: Causal Autoregressive Flow for Goal-Directed Safety-Critical Scenes Generation

“…(Alcorn et al, 2019;Xiao et al, 2019;Jain et al, 2019) manipulate the pose of objects in traffic scenes. (Tu et al, 2020;Abdelfattah et al, 2021) adds objects on the top of existing vehicles to make them disappear, (Sun et al, 2020) creates a ghost vehicle by adding an ignorable number of points, and (Ding et al, 2021b) generates the layout of the traffic scene with a tree structure integrated with human knowledge. Another line of research generates the risky scenes while also considering the likelihood of occurring of the scenes in the real world, which requires a probabilistic model of the environment.…”