Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints
Keywords: text-driven 3D room generation, 3D scene editing, diffusion model, indoor scene layout, panorama generation
Abstract: Text-driven 3D indoor scene generation could be useful for gaming, film industry,
and AR/VR applications. However, existing methods cannot faithfully capture the
room layout, nor do they allow flexible editing of individual objects in the room.
To address these problems, we present Ctrl-Room, which is able to generate con-
vincing 3D rooms with designer-style layouts and high-fidelity textures from just
a text prompt. Moreover, Ctrl-Room enables versatile interactive editing opera-
tions such as resizing or moving individual furniture items. Our key insight is
to separate the modeling of layouts and appearance. Our proposed method con-
sists of two stages, a ‘Layout Generation Stage’ and an ‘Appearance Generation
Stage’. The ‘Layout Generation Stage’ trains a text-conditional diffusion model to
learn the layout distribution with our holistic scene code parameterization. Next,
the ‘Appearance Generation Stage’ employs a fine-tuned ControlNet to produce a
vivid panoramic image of the room guided by the 3D scene layout and text prompt.
In this way, we achieve a high-quality 3D room with convincing layouts and lively
textures. Benefiting from the scene code parameterization, we can easily edit the
generated room model through our mask-guided editing module, without expen-
sive editing-specific training. Extensive experiments on the Structured3D dataset
demonstrate that our method outperforms existing methods in producing more rea-
sonable, view-consistent, and editable 3D rooms from natural language prompts.
Supplementary Material: pdf
Primary Area: generative models
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3612
Loading