PFAvatar: Avatar Reconstruction from Multiple In-the-wild Images

Dianbing Xi, Yuan Liu, Zhijian Liu, Jingsen Zhu, Yuchi Huo, Ruiyuan Zhang, Jiayuan Lu, Rui Wang

Keywords: Avatar Reconstruction, Text-to-Image Diffusion Model, Image-based Modeling, Text-guided 3D Generation, Digital Human;

TL;DR: reconstructs a textured 3D Avatar from few-shot in-the-wild images

Abstract: In this work, we present \textit{PFAvatar}, a new approach to avatar reconstruction and editing from multiple in-the-wild images with varying poses, unknown camera conditions, cropped views, and occlusions. Traditional methods often rely on full-body images captured with controlled avatar pose, camera settings, lighting, and background, while struggling to reconstruct under in-the-wild settings.To address this issue, we fuse the varying pose priors of avatars in in-the-wild images, thereby enabling precise control over avatar generation.Specifically, we first inject avatar features (pose, appearance) from input images using a Vision-Language Model (VLM) and ControlNet. Subsequently, we employ a pose-conditioned 3D-Consistent Score Distillation Sampling (3D-SDS), which enables reconstructing a high-quality 3D avatar. Additionally, we propose a Condition Prior Preservation Loss (CPPL) to mitigate the issues of language and control drift caused by fine-tuning VLM and ControlNet with few-shot data. Through comprehensive experiments and evaluation, we demonstrate the effectiveness of our method for reconstructing avatars from in-the-wild images, supporting further applications like avatar editing.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2172