Text-to-3D Generation with Bidirectional Diffusion using both 3D and 2D priors
Keywords: 3D generation, foundation model
TL;DR: Text-to-3D Generation with Bidirectional Diffusion using both 3D and 2D priors
Abstract: Most research in 3D object generation focuses on boosting 2D foundational models into the 3D space, either by minimizing 2D SDS loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these methods often lead to geometric anomalies and multi-view inconsistency. Recently, researchers have attempted to improve the genuineness of 3D objects by training directly on 3D datasets, albeit at the cost of low-quality texture generation due to the limited 2D texture variation in 3D datasets. To harness the advantages of both approaches, in this paper, we propose Bidirectional Diffusion (BiDiff), a unified framework that incorporates both a 3D and a 2D diffusion process, to preserve both 3D fidelity and 2D texture richness, respetively. Recognizing that a simple combination of two diffusion processes can yield inconsistent generation results, we further bridge them with innovative bidirectional guidance. At last, we also demonstrate that this is the first 3D generative model that can separately control texture and geometry generation. Experimental results have shown that our model achieves high-quality, diverse, and scalable 3D generation. The project website is https://bidiff.github.io
Supplementary Material: pdf
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3240
Loading