Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos

Alfredo Rivero, ShahRukh Athar, Zhixin Shu, Dimitris Samaras

Keywords: Neural Rendering; Neural Portraits;

Abstract: We present Rig3DGS, a novel technique for creating reanimatable 3D portraits from short monocular smartphone videos. Rig3DGS learns to reconstruct a set of controllable 3D gaussians from a monocular video of a dynamic subject captured with varying head poses and facial expressions in an in-the-wild scene. In contrast to synchronized multi-view studio captures, this in-the-wild, single camera setup brings fresh challenges to learning high quality 3D gaussians. We address these challenges by learning to deform 3D gaussians from a fixed canonical space to the deformed space that is consistent with the target facial expression and head-pose. Our key contribution is a carefully designed deformation model that is guided by a 3D face morphable model. This deformation not only enables control over facial expression and head-poses but also allows our method to generates high quality photorealistic renders of the whole scene. Once trained, Rig3DGS is able to generate photorealistic renders of a subject and their scene for novel facial expression, head-poses, and viewing directions. Through extensive experiments we demonstrate that Rig3DGS significantly outperforms prior art while being orders of magnitude faster.

Submission Number: 259