LISA: UNLEASHING 2D DIFFUSION FOR 3D GENERATION VIA LIGHTWEIGHT IMAGE SPLATS ADAPTATION
Keywords: 3D asset creation, lightweight adapter, Gaussian Splats
Abstract: Despite its potential, 3D generation lags behind 2D generation in quality and utility, primarily due to the vast gap in the scale and diversity of training data—high-quality 2D data is abundant, while high-quality 3D assets remain limited by orders of magnitude. Existing methods use 2D generative priors for 3D asset creation via distillation or generate-and-reconstruct schemes, both of which suffer from quality loss during optimization. In this paper, we propose a novel scheme to exploit 2D diffusion prior for 3d generation by integrating a lightweight adapter into the decoder of a frozen 2D diffusion model, allowing it to generate RGB images, Gaussian splats, and physics-based rendering material maps simultaneously. Once trained, the proposed Lightweight Image Splats Adaptation (LISA) directly produces relightable Gaussian splats in feed-forward manner, which can be converted into high-quality, relightable 3D meshes through an inverse rendering framework. Quantitative and qualitative results demonstrate that our method outperforms state-of-the-art approaches with a significantly lower computational budget for both training and sampling. More results can be found at https://LISA-3dgen.github.io.
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13468
Loading