Semantic-aware Responsive Listener Head Synthesis

Wei Zhao, Peng Xiao, Rongju Zhang, Yijun Wang, Jianxin Lin

Abstract: Audience providing proper reaction during a conversation can bring positive impact to speaker, which is significant to digital human and social agent areas. Given information sent by speaker, responsive listener head synthesis task aims to generate corresponding listener behaviours such as nodding, thinking and smiling. A common method is to build listener responsive pattern by analyzing acoustic and facial feature of speaker. However, it is hard to understand what speaker means, purely based on acoustic and facial feature since numerous message is buried in language. Traditional method may lead to similar results ignoring the diversity of input. Therefore, in this paper we presents a new Semantic-aware Responsive Listener Head Synthesis (SaRLHS) approach by considering semantic information lied in language patterns in addition to acoustic and facial feature. Besides, we implement a post-face enhancement process to increase the visual effects. Moreover, we won the People's Selection Awards and the second place on Grand Challenges of ACM 2022 conference.