Vision-only 3D Tracking for Self-Driving Cars

Carlos Diaz-Ruiz, Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, Mark E. Campbell

Abstract: A vision-only tracking framework is developed and experimentally evaluated for self-driving cars. Vision-only object detection is achieved by transforming stereo depth maps into point clouds, followed by LiDAR-based detectors. Each detection yields location, orientation, and object size. A tracking algorithm is then used to combine the detections with a physics based model to create robust vehicle tracks and IDs. We empirically evaluate our approach to the ones relying on LiDAR using the KITTI Tracking dataset. We found that vision-only trackers yield comparable performance in short ranges, but are still outperformed by the LiDAR-based one at far distances. Specifically, vision-only detection and tracking can generate good estimates achieving close performance to LiDAR based detection at close range. The approach is generalizable to other trackers, particularly those which use multiple sensors.