Volumetric Disentanglement for 3D Scene Manipulation

Sagie Benaim, Frederik Rahbæk Warburg, Peter Ebert Christensen, Serge Belongie

Keywords: 3D Object Editing, Neural Radiance Fields, Disentanglement

TL;DR: We propose a framework for disentangling a 3D scene into a foreground and background volumetric representations and show a variety of downstream applications involving 3D manipulation.

Abstract:

Recently, advances in differential volumetric rendering enabled significant breakthroughs in the photo-realistic and fine-detailed reconstruction of complex 3D scenes, which is key for many virtual reality applications. However, in the context of augmented reality, one may also wish to effect semantic manipulations or augmentations of objects within a scene. To this end, we propose a volumetric framework for (i) disentangling or separating, the volumetric representation of a given foreground object from the background, and (ii) semantically manipulating the foreground object, as well as the background.
Our framework takes as input a set of 2D masks specifying the desired foreground object for training views, together with the associated 2D views and poses, and produces a foreground-background disentanglement that respects the surrounding illumination, reflections, and partial occlusions, which can be applied to both training and novel views. Unlike previous work, our method does not rely on 3D information in the form of 3D object bounding boxes or a scene voxel grid. It correctly captures reflective foreground objects, objects occluded by the background, and objects with noisy and inaccurate masks.
Our method enables the separate control of pixel color and depth as well as 3D similarity transformations of both the foreground and background objects. We subsequently demonstrate our framework's applicability on several downstream manipulation tasks, going beyond the placement and movement of foreground objects. These tasks include object camouflage, non-negative 3D object inpainting, 3D object translation, 3D object inpainting, and 3D text-based object manipulation.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Applications (eg, speech processing, computer vision, NLP)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/volumetric-disentanglement-for-3d-scene/code)