FaithfulFaces: Pose-Faithful Facial Identity Preservation for Text-to-Video Generation

Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng, bing ma, Kai Yu, Wenyue Li, Tianxiang Zheng, Qinglin Lu, Zhen Cui

Keywords: Pose-Faithful Facial Identity Preservation, Identity-Preserving Text-to-Video Generation

Abstract: Identity-preserving text-to-video generation (IPT2V) empowers users to produce diverse and imaginative videos with consistent human facial identity. Although existing open-source and commercial methods have demonstrated impressive performance in typical scenarios, they still face significant limitations when confronted with challenging cases, such as large facial pose variations or facial occlusions. These challenges frequently result in identity distortion in the generated videos. In this paper, we propose FaithfulFaces, a pose-faithful facial identity preservation learning framework to improve IPT2V in complex dynamic scenes. Specifically, FaithfulFaces first proposes a pose-shared identity aligner that refines and aligns facial poses across distinct views via a pose-shared dictionary and a pose variation–identity invariance constraint. Then, the well-learned aligner can capture the global facial pose representation from the input single-view face image with explicit Euler angle embeddings, which could provide a pose-faithful facial prior for foundational generative models to better preserve identity in the generated videos. In particular, we develop a high-quality video dataset pipeline featuring substantial facial pose variations specifically for our FaithfulFaces to facilitate robust training. Compared to other IPT2V methods, FaithfulFaces achieves state-of-the-art performance across multiple metrics, generating high-quality videos with clear facial structures and consistent identity preservation, even as facial pose changes and occlusions occur. The code and dataset pipeline will be released.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 3869