Looks Great, Functions Better: Physics Compliance Text-to-3D Shape Generation

Qingshan Xu, Jiao Liu, Melvin Wong, Caishun Chen, Yew-Soon Ong

Keywords: 3D shape generation, Functional 3D model, Physics perception, Differentiable physics layer, Solid mechanics

TL;DR: Looks great, functions better

Abstract: Text-to-3D shape generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring function for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications. In this work, we propose Fun3D, a physics driven functional text-to-3D shape generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics. To this end, we leverage 3D diffusion models to provide 3D shape priors and design a data-driven differentiable physics layer to optimize 3D shape priors with solid mechanics. This allows us to optimize geometry efficiently and learn physics information about 3D shapes at the same time. Experimental results demonstrate that our method can consider both geometric plausibility and functional requirement, further bridging 3D virtual modeling and physical worlds.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9022