Contrastive guidance and feedback: A Suitable way to improve 3D Consistency of Multi-view Diffusion Model

Ruofeng Yang, Yaqing Zhang, Le Wan, Shuai Li

Keywords: Contrastive guidance, Multi-view diffusion model

Abstract:

Recently, diffusion models have shown potential in 3D generation tasks, and the novel view synthesis (NVS) task, a bridge between 2D and 3D generation, has received great attention. The goal of the NVS task is to generate multi-view images from reference images, and the core challenge is to maintain the 3D consistency between different view images.
Recent works construct large 3D consistency multi-view image datasets and utilize the supervised fine-tuning (SFT) method to improve the 3D consistency. However, the SFT method suffers from the distribution shift, data inefficient problems, and lacks theoretical insight. To solve these problems, we discuss how to provide a suitable direction to the multi-view models and achieve better performance. More specifically, we first analyze the training-free guidance-based method and prove that contrastive guidance, which contains ground-truth and generated samples, can provide the right direction to improve 3D consistency. Based on the theoretical insight, we further design a contrastive 3D consistency metric and use it as the feedback in the following phase. To avoid the distribution shift problem, we use direct preference optimization (DPO) to fine-tune the multi-view diffusion models. Through qualitative and quantitative experiments, we demonstrate that after the fine-tuning phase with the above method, the 3D consistency of the multi-view images is significantly improved and achieves better performance compared to the SFT method.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5886