Betsu-Betsu: Separable 3D Reconstruction of Two Interacting Objects from Multiple Views

Suhas Gopal, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt

Keywords: SDF, multi-view geometry

Abstract:

Separable 3D reconstruction of multiple objects from multi-view RGB images—resulting in two different 3D shapes for the two objects with a clear separation between them—remains a sparely researched problem. It is challenging due to severe mutual occlusions and ambiguities
along the objects’ interaction boundaries. This paper investigates the setting and introduces a new neuro-implicit method that can reconstruct the geometry and appearance of two objects undergoing close interactions while disjoining both in 3D, avoiding surface inter-penetrations and enabling novel-view synthesis of the observed scene. In our approach, the objects in the scene are first encoded using a shared multi-resolution hash grid. Next, its features are decoded into two neural SDFs for the respective objects. The framework is end-to-end trainable and supervised using a novel alpha-blending regularization that ensures that the two geometries are well separated even under extreme occlusions. Our reconstruction method is markerless and can be applied to rigid as well as articulated objects. We introduce a new dataset consisting of close interactions between a human and an object and also evaluate on two scenes of humans performing martial arts. The experiments confirm the effectiveness of our framework and substantial improvements using 3D and novel view synthesis metrics compared to several existing approaches applicable in our setting.

Submission Number: 14