Virtual Reality via Object Pose Estimation and Active Learning: Realizing Telepresence Robots with Aerial Manipulation Capabilities

Jongseok Lee, Ribin Balachandran, Konstantin Kondak, Andre Coelho, Marco De Stefano, Matthias Humt, Jianxiang Feng, Tamim Asfour, Rudolph Triebel

Keywords: Aerial Manipulation, Virtual Reality, Telepresence Robots, Pose Estimation, Active Learning

TL;DR: Latest development of the DLR suspended aerial manipulation research

Abstract:

This article presents a novel telepresence system for advancing aerial manipulation in dynamic and
unstructured environments. The proposed system not only features a haptic device, but also a virtual
reality (VR) interface that provides real-time 3D displays of the robot’s workspace as well as a
haptic guidance to its remotely located operator. To realize this, multiple sensors namely a LiDAR,
cameras and IMUs are utilized. For processing of the acquired sensory data, pose estimation pipelines
are devised for industrial objects of both known and unknown geometries. We further propose an
active learning pipeline in order to increase the sample efficiency of a pipeline component that relies
on Deep Neural Networks (DNNs) based object detection. All these algorithms jointly address
various challenges encountered during the execution of perception tasks in industrial scenarios.
In the experiments, exhaustive ablation studies are provided to validate the proposed pipelines.
Methodologically, these results commonly suggest how an awareness of the algorithms’ own failures
and uncertainty (”introspection”) can be used tackle the encountered problems. Moreover, outdoor
experiments are conducted to evaluate the effectiveness of the overall system in enhancing aerial
manipulation capabilities. In particular, with flight campaigns over days and nights, from spring to
winter, and with different users and locations, we demonstrate over 70 robust executions of pick-and-
place, force application and peg-in-hole tasks with the DLR cable-Suspended Aerial Manipulator
(SAM). As a result, we show the viability of the proposed system in future industrial applications