RGB-DE: Event camera calibration for fast 6-dof object tracking

Etienne Dubeau, Mathieu Garon, Benoit Debaque, Raoul de Charette, Jean-Francois Lalonde

Abstract:

Augmented reality devices require multiple sensors to perform vari-
ous tasks such as localization and tracking. Currently, popular cam-
eras are mostly frame-based (e.g. RGB and Depth) which impose a
high data bandwidth and power usage. With the necessity for low
power and more responsive augmented reality systems, using solely
frame-based sensors imposes limits to the various algorithms that
needs high frequency data from the environement. As such, event-
based sensors have become increasingly popular due to their low
power, bandwidth and latency, as well as their very high frequency
data acquisition capabilities. In this paper, we propose, for the first
time, to use an event-based camera to increase the speed of 3D object
tracking in 6 degrees of freedom. This application requires handling
very high object speed to convey compelling AR experiences. To
this end, we propose a new system which combines a recent RGB-D
sensor (Kinect Azure) with an event camera (DAVIS346). We de-
velop a deep learning approach, which combines an existing RGB-D
network along with a novel event-based network in a cascade fashion,
and demonstrate that our approach significantly improves the robust-
ness of a state-of-the-art frame-based 6-DOF object tracker using our
RGB-D-E pipeline. Our code and our RGB-D-E evaluation dataset
are available at https://github.com/lvsn/rgbde_tracking.