AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation

Nitin Saini, Elia Bonetto, Eric Price, Aamir Ahmad, Michael J. Black

Abstract: In this letter, we present a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerial vehicles (UAVs) with on-board RGB cameras and computation. Existing methods are limited by calibrated cameras and off-line processing. Thus, we present the first method (AirPose) to estimate human pose and shape using images captured by multiple extrinsically <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uncalibrated</b> flying cameras. AirPose itself calibrates the cameras <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">relative to the person</i> instead of relying on any pre-calibration. It uses distributed neural networks running on each UAV that communicate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">viewpoint-independent</i> information with each other about the person (i.e., their 3D shape and articulated pose). The person’s shape and pose are parameterized using the SMPL-X body model, resulting in a compact representation, that minimizes communication between the UAVs. The network is trained using synthetic images of realistic virtual environments, and fine-tuned on a small set of real images. We also introduce an optimization-based post-processing method (AirPose <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{+}$</tex-math></inline-formula> ) for offline applications that require higher MoCap quality. We make our method’s code and data available for research at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/robot-perception-group/AirPose</uri> . A video describing the approach and results is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/xLYe1TNHsfs</uri> .