Causal Scene BERT: Improving object detection by searching for challenging groups

Cinjon Resnick, Or Litany, Amlan Kar, Karsten Kreis, James Lucas, Kyunghyun Cho, Sanja Fidler

Keywords: object detection, computer vision, masked language modeling, autonomous vehicles, causality

Abstract: Autonomous vehicles (AV) rely on learning-based perception modules parametrized with neural networks for tasks like object detection. These modules frequently have low expected error overall but high error on atypical groups of data due to biases inherent in the training process. Multiple heuristics are employed to identify "failures" in AVs, a typical example being driver interventions. After identification, a human team combs through the associated data to group perception failures that share common causes. More data from these groups is then collected and annotated before retraining the model to fix the issue. In other words, error groups are found and addressed in hindsight as they appear. Our main contribution is a pseudo-automatic method to discover such groups in foresight by performing causal interventions on simulated driving scenes. To keep our interventions on the data manifold, we use masked language models. We verify that the prioritized groups found via intervention are challenging for the object detector and show that retraining with data collected from these groups helps inordinately compared to adding more IID data. We also release software to run interventions in simulated scenes, which we hope will benefit the causality community.

One-sentence Summary: The status quo in object detection finds error groups like difficult vehicle types or weather patterns in hindsight. Let’s find them in foresight using masked language models to take causal interventions on scene graphs.