Multi-view Consistent Image Generation through Self-calibrated Latent Refinement

Jaeyeon Kim, Ka Chun SHUM, Binh-Son Hua, Duc Thanh Nguyen, Yaxin Shi, Ivor Tsang, Sai-Kit Yeung

Keywords: 3D aware image synthesis, 3D generation, GAN, Diffusion.

Abstract: In this paper, we introduce a novel 3D-aware image generation framework that ensures high-quality and view-consistent image generation. Our core idea is to leverage the semantic latent space of a pre-trained 2D GAN for 3D view-consistent image generation, eliminating need for large-scale dataset use and prior knowledge of camera poses. To achieve this, we propose a latent refiner with multi-view and geometry-preserving capabilities, enabled by self-calibrated depth and pose estimation. Thanks to the advances of diffusion models, our refiner allows for view-consistent latent manipulation in GANs and can be trained using a self-supervised fashion. Our method optimizes the latent codes of a pre-trained 2D GAN across a wide range of pose angles. We demonstrate the effectiveness of our method through evaluations and comparisons with existing baselines on benchmark datasets. Experimental results show the superiority of our method over existing works in both the quality and view consistency of generated images.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9440