Painting with 3D Gaussian Splat Brushes

Or Perel, Karran Pandey, Anita Hu, Maria Shugrina, Clement Fuji Tsang, Karan Singh

Abstract:

We explore interactive painting on 3D Gaussian splat scenes and other
surfaces using 3D Gaussian splat brushes, each containing a chunk of realistic
texture-geometry that make capture representations so appealing. The suite
of brush capabilities we propose enables 3D artists to capture and then
remix real world imagery and geometry with direct interactive control. In
particular, we propose a set of algorithms for 1) selecting subsets of Gaussians
as a brush pattern interactively, 2) applying the brush interactively to the
same or other 3DGS scenes or other 3D surfaces using stamp-based painting,
3) using an inpainting Diffusion Model to adjust stamp seams for seamless
and realistic appearance. We also present an ensemble of artistic brush
parameters, resulting in a wide range of appearance options for the same
brush. Our contribution is a judicious combination of algorithms, design
features and creative affordances, that together enable the first prototype
implementation of interactive brush-based painting with 3D Gaussian splats.
We evaluate our system by showing compelling results on a diverse set of 3D scenes; and a user study with VFX/animation professionals, to validate system features, workflow, and potential for creative impact. Code and data
for this paper can be accessed from splatpainting.github.io.