3D PixBrush: Image-Guided Local Texture Synthesis

Dale Decatur, Itai Lang, Kfir Aberman, Rana Hanocka

Keywords: Generative 3D Modeling, Image-based 3D Texturing, Image-based Texture Generation, Meshes

TL;DR: We introduce a method for performing local, image-guided texturing of meshes.

Abstract:

We present 3D PixBrush, a method for performing image-driven edits of local regions on 3D meshes. 3D PixBrush predicts a localization mask and a synthesized texture that are guided the object in the reference image. Our predicted localizations are both globally coherent and locally precise. Globally - our method contextualizes the object in the reference image and automatically positions it onto
the input mesh. Locally - our method produces masks that conform to the geometry of the reference image. Notably, our method does not require any user input (in the form of scribbles or bounding boxes) to achieve accurate localizations. Instead, our method predicts a localization mask on the 3D mesh from scratch. To achieve this, we propose a modification to the score distillation sampling technique
which incorporates both the predicted localization and the reference image, referred to as localization-modulated image guidance. We demonstrate the effectiveness of our proposed technique on a wide variety of meshes and images.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 1054