Enhancing Realism in 3D Facial Animation Using Conformer-Based Generation and Automated Post-Processing

Yi Zhao, Chunyu Qiang, Hao Li, Yulan Hu, Wangjin Zhou, Sheng Li

Abstract: Recent progress has propelled the development of realistic talking-face videos for avatars. Yet, animating 3D cartoon avatars remains intricate due to the imprecise nature of facial-driven data. This often manifests as inconsistent mouth configurations and rigid facial expressions, curbing the animation’s realism. Addressing these issues, we introduce a conformer-based framework that derives expression coefficients directly from phonemes, thereby elevating prediction precision and minimizing manual oversight. Furthermore, by harnessing a pre-trained emotion blending module coupled with the keyframe of the target emotional character, we employ a zero-shot adaptation technique. This serves to amplify emotional expressions and bolster the authenticity of lip dynamics. Our methodology adeptly registers nuanced expression shifts in avatars, leading to remarkably lifelike animations, as substantiated by our experimental findings.