JoIN: Joint GANs Inversion for Intrinsic Image Decomposition
Abstract: Intrinsic Image Decomposition (IID) is a challenging inverse problem that seeks to decompose a natural image into its underlying intrinsic components such as albedo and shading. While traditional IID methods rely on hand-crafted or learned priors on these components, they often suffer from poor disentanglement, component cross-contamination, and Sim-to-Real gap due to reliance on synthetic training data. In this work, we propose to solve the IID problem using a bank of Generative Adversarial Networks (GANs) as priors where each GAN is independently trained only on a single intrinsic component, providing stronger and more disentangled priors. At the core of our approach is the idea that the latent space of a GAN is a well-suited optimization domain to solve inverse problems. Given an input image, we propose to jointly invert the latent codes of a set of GANs and combine their outputs to reproduce the input. Contrary to all existing GAN inversion methods that are limited to inverting only a single GAN, our proposed approach, JoIN, is able to jointly invert multiple GANs using only a single image as supervision while still maintaining distribution priors of each intrinsic component. We show that our approach is modular, allowing various forward imaging models, and that it can successfully decompose both synthetic and real images. Further, by leveraging generator fine-tuning technique, our method is able to achieve excellent generalization on real images even though it uses only synthetic data to train the GAN priors. We demonstrate the success of our approach through exhaustive qualitative and quantitative evaluations and ablation studies on different datasets.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Jiajun_Wu1
Submission Number: 3575
Loading