Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding

Ziyang Yuan, Yiming Zhu, Yu Li, Hongyu Liu, Chun Yuan

Abstract:

3D GAN inversion aims to achieve high reconstruction
fidelity and reasonable 3D geometry simultaneously from a
single image input. However, existing 3D GAN inversion
methods rely on time-consuming optimization for each individual case. In this work, we introduce a novel encoderbased inversion framework based on EG3D, one of the most
widely-used 3D GAN models. We leverage the inherent
properties of EG3D’s latent space to design a discriminator and a background depth regularization. This enables us
to train a geometry-aware encoder capable of converting
the input image into corresponding latent code. Additionally, we explore the feature space of EG3D and develop an
adaptive refinement stage that improves the representation
ability of features in EG3D to enhance the recovery of finegrained textural details. Finally, we propose an occlusionaware fusion operation to prevent distortion in unobserved
regions. Our method achieves impressive results comparable to optimization-based methods while operating up to
500 times faster. Our framework is well-suited for applications such as semantic editing.