Taming Mode Collapse in Score Distillation for Text-to-3D Generation

Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, YILEI LI, qiang liu, Zhangyang Wang, Vikas Chandra

Keywords: Text-to-3D Generation, Mode Collapse

TL;DR: We examine Janus artifact through the lens of mode collapse and employ entropy regluarization as an effective solution.

Abstract:

Despite the remarkable performance of score distillation in text-to-3D generation,
such techniques notoriously suffer from view inconsistency issues, also known
as “Janus” artifact, where the generated objects fake each view with multiple
front faces. Although empirically effective methods have approached this problem
via time re-scheduling or prompt engineering, a statistical view to explain and
tackle this problem remains elusive. In this paper, we reveal that the existing
score distillation-based text-to-3D generation frameworks degenerate to maximal
likelihood seeking on each view independently and thus suffer from the mode
collapse problem, manifesting as the Janus artifact in practice. To tame mode
collapse, we improve score distillation by re-establishing the entropy term in the
correponding variational objective and derive a new update rule for 3D score
distillation, dubbed Entropic Score Distillation (ESD). The entropy is applied to
the distribution of rendered images. Maximizing the entropy encourages diversity
among different views in generated 3D assets, thereby alleviating the Janus problem.
We conduct experiments with our proposed ESD, and validate that ESD can be an
effective treatment for Janus artifacts for score distillation.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3211