PIVQGAN: Posture and Identity Disentangled Image-to-Image Translation via Vector Quantization

Bingchen Liu, Yizhe Zhu, Xiao Yang, Ahmed Elgammal

Keywords: deep learning, generative model, image synthesis, generative adversarial network, self-supervised learning, image-to-image translation

Abstract: One popular objective for the image-to-image translation task is to independently control the coarse-level object arrangements (posture) and the fine-grained level styling (identity) of the generated image from two exemplar sources. To approach this objective, we propose PIVQGAN with two novel techniques in the framework of StyleGAN2. First, we propose a Vector-Quantized Spatial Normalization (VQSN) module for the generator for better pose-identity disentanglement. The VQSN module automatically learns to encode the shaping and composition information from the commonly shared objects inside the training-set images. Second, we design a joint-training scheme with self-supervision methods for the GAN-Inversion encoder and the generator. Specifically, we let the encoder and generator reconstruct images from two differently augmented variants of the original ones, one defining the pose and the other for identity. The VQSN module facilitates a more delicate separation of posture and identity, while the training scheme ensures the VQSN module learns the pose-related representations. Comprehensive experiments conducted on various datasets show better synthesis image quality and disentangling scores of our model. Moreover, we present model applications beyond posture-identity disentangling, thanks to the latent-space reducing feature of the leveraged VQSN module.

One-sentence Summary: Image to image translation model with unsupervised training scheme on unpaired and unlabeled data, with a weak but useful self-learned semantic segmentation capability.