Exp-GAN: 3D-Aware Facial Image Generation with Expression Control

Yeonkyeong Lee, Taeho Choi, Hyunsung Go, Hyunjoon Lee, Sunghyun Cho, Junho Kim

Abstract: This paper introduces Exp-GAN, a 3D-aware facial image generator with explicit control of facial expressions. Unlike previous 3Daware GANs, Exp-GAN supports fine-grained control over facial shapes and expressions disentangled from poses. To this ends, we propose a novel hybrid approach that adopts a 3D morphable model (3DMM) with neural textures for the facial region and a neural radiance field (NeRF) for non-facial regions with multi-view consistency. The 3DMM allows finegrained control over facial expressions, whereas the NeRF contains volumetric features for the non-facial regions. The two features, generated separately, are combined seamlessly with our depth-based integration method that integrates the two complementary features through volume rendering. We also propose a training scheme that encourages generated images to reflect control over shapes and expressions faithfully. Experimental results show that the proposed approach successfully synthesizes realistic view-consistent face images with fine-grained controls. Code is available at https://github.com/kakaobrain/expgan