MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

Xudong XU, Yitong Wang, Zhaoyang Lyu, Xingang Pan, Bo Dai

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Text-to-3D generation, stable diffusion, material generation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Based on powerful text-to-image diffusion models, text-to-3D generation has made significant progress in generating compelling geometry and appearance. However, existing methods still struggle to recover high-fidelity object materials, either only considering Lambertian reflectance, or failing to disentangle BRDF materials from the environment lights. In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR (MATLABER) that leverages a novel latent BRDF auto-encoder for material generation.
The BRDF auto-encoder is trained with large-scale real-world BRDF collections, serving as a useful prior to constrain the generated material in a natural distribution. To further disentangle material from environment lights, we adopt a semantic-aware material regularization that motivates object parts with the same semantics to share similar materials.
Through exhaustive experiments, our approach demonstrates the superiority over existing ones in generating realistic and coherent object materials.
Moreover, high-quality materials naturally enable multiple downstream tasks such as relighting and material editing.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4993