UniFlow: Zero-Shot LiDAR Scene Flow via Cross-Domain Generalization

Siyi Li, Qingwen Zhang, Ishan Khatri, Kyle Vedder, Deva Ramanan, Neehar Peri

Keywords: LiDAR Scene Flow, Zero-Shot, Autonomous Veicles

TL;DR: We present UniFlow, LiDAR scene flow method that combines existing datasets and demonstrates state-of-the-art performance on both in-domain and out-of-domain generalization.

Abstract: Scene flow estimation is an important primitive for 3D motion understanding and dynamic scene reconstruction. Recent LiDAR-based methods have made significant progress in achieving centimeter-level accuracy on popular autonomous vehicle (AV) datasets. Notably, such methods typically only train and evaluate on the same dataset because each dataset has its own unique sensor setup. Motivated by recent work in zero-shot image-based scene flow, we argue that multi-dataset training is essential for scaling up LiDAR-based methods. However, prior work in LiDAR-based semantic segmentation and 3D object detection demonstrate that naively training on multiple datasets yields worse performance than single-dataset models. We re-examine this conventional wisdom in the context of LiDAR-based scene flow. Contrary to popular belief, we find that state-of-the-art scene flow methods greatly benefit from cross-dataset training. We posit that low-level tasks such as motion estimation may be less sensitive to sensor configuration than high-level tasks such as detection. Informed by our analysis, we propose UniFlow, a feedforward model that unifies and trains on multiple large-scale LiDAR scene flow datasets with diverse point density and velocity distributions. Our frustratingly simple solution establishes a new state-of-the-art on Waymo and nuScenes, improving over prior work by 16.4% and 34.5% respectively. Moreover, UniFlow achieves state-of-the-art zero-shot accuracy on TruckScenes, outperforming prior dataset-specific models by 38.4%!

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5165