Semantically Adversarial Driving Scenario Generation with Explicit Knowledge Integration

Wenhao Ding, Haohong Lin, Bo Li, Ji Eun Kim, Ding Zhao

Abstract: Generating adversarial scenarios, which have the potential to fail autonomous driving systems, provides an effective way to improve the robustness. Extending purely data-driven generative models, recent specialized models satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns \textit{implicitly} in the neuron level. In this paper, we introduce a method to incorporate domain knowledge \textit{explicitly} in the generation process to achieve the \textit{Semantically Adversarial Generation (SAG)}. To be consistent with the composition of driving scenes, we first categorize the knowledge into two types, the property of objects and the relationship among objects. We then propose a tree-structured variational auto-encoder (T-VAE) to learn hierarchical scene representation. By imposing semantic rules on the properties of nodes and edges in the tree structure, explicit knowledge integration enables controllable generation. We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend to realistic environments for autonomous vehicles: our method efficiently identifies adversarial driving scenes against different state-of-the-art 3D point cloud segmentation models and satisfies the traffic rules specified as the explicit knowledge.