Reconstructing Animatable Categories from Videos

Gengshan Yang, Chaoyang Wang, N. Dinesh Reddy, Deva Ramanan

Abstract: Building animatable 3D models is challenging due to the need for 3D scans, laborious registration, and rigging. Recently, differentiable rendering provides a pathway to obtain high-quality 3D models from monocular videos, but these are limited to rigid categories or single instances. We present RAC, a method to build category-level 3D models from monocular videos, disentangling variations over instances and motion over time. Three key ideas are introduced to solve this problem: (1) specializing a category-level skeleton to instances, (2) a method for latent space regularization that encourages shared structure across a category while maintaining instance details, and (3) using 3D background models to disentangle objects from the background. We build 3D models for humans, cats and dogs given monocular videos. Project page: https://gengshan-y.github.io/rac-www/.