Clothing-disentangled 3D character generation from a single image

Junting Dong, Mingze Sun, Qi Fang, Yan-Pei Cao, Jingbo Wang, Bo Dai

Keywords: 3D human reconstruction; 3D human generation

Abstract: This paper tackles the challenge of generating clothing-disentangled 3D characters from a single image. Existing approaches typically employ multi-layer 3D representations to model the body and each garment and then iteratively optimize these representations to fit the observations, which is time-consuming and not scalable. To address this, we propose the first feed-forward method enabling efficient and robust clothing disentanglement. Our approach first generates the multi-view images for each component of the clothed character and then employs a generalizable multi-view reconstruction method to create the 3D models of each component. For high-quality disentanglement, we propose a two-stage disentanglement approach that first disentangles each component in the 2D image space and then generates the multi-view images for each part. During the 2D component disentanglement stage, we introduce a novel multi-part diffusion model that allows information exchange among different components. Additionally, for component combination, we incorporate a novel combination attention mechanism into the multi-view diffusion model, enabling the integration of information from multiple parts to create the final combined character. For training, we have contributed a large clothing-disentangled character dataset consisting of more than 10k anime characters. Extensive experiments demonstrate that our proposed approach not only facilitates efficient and high-quality disentangled 3D character generation with distinct clothing layers but also supports various cloth editing applications.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5805