CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation

ICLR 2026 Conference Submission19565 Authors

Keywords: Speech-driven generation, Head animation, Text-driven generation

Abstract:

Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips.
While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users’ ability to freely control speaking styles.
Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio.
To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion.
Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions.
Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 19565