Gaussian Shell Maps for Efficient 3D Human Generation

Rameen Abdal, Yifan Wang, Zifan Shi, Yinghao Xu, Ryan Po, Zhengfei Kuang, Qifeng Chen, Dit-Yan Yeung, Gordon Wetzstein

Abstract: Efficient generation of 3D digital humans is important in several industries, including virtual reality, social media, and cinematic production. 3D generative adversarial net-works (GANs) have demonstrated state-of-the-art (SOTA) quality and diversity for generated assets. Current 3D GAN architectures, however, typically rely on volume representations, which are slow to render, thereby hampering the GAN training and requiring multi- view-inconsistent 2D upsam-plers. Here, we introduce Gaussian Shell Maps (GSMs) as a framework that connects SOTA generator network archi-tectures with emerging 3D Gaussian rendering primitives using an articulable multi shell-based scaffold. In this set-ting, a CNN generates a 3D texture stack with features that are mapped to the shells. The latter represent inflated and deflated versions of a template surface of a digital human in a canonical body pose. Instead of rasterizing the shells directly, we sample 3D Gaussians on the shells whose at-tributes are encoded in the texture features. These Gaus-sians are efficiently and differentiably rendered. The ability to articulate the shells is important during GAN training and, at inference time, to deform a body into arbitrary user-defined poses. Our efficient rendering scheme bypasses the need for view-inconsistent upsamplers and achieves high-quality multi-view consistent renderings at a native resolution of 512 ×512 pixels. We demonstrate that GSMs suc-cessfully generate 3D humans when trained on single-view datasets, including SHHQ and DeepFashion. Project Page: rameenabdal.github.io/GaussianShellMaps