TransGesture: Autoregressive Gesture Generation with RNN-Transducer

Naoshi Kaneko, Yuna Mitsubayashi, Geng Mu

Track: Challenge paper

Team Name: TransGesture

Keywords: gesture generation, speech audio, neural networks, deep learning

Abstract: This paper presents a gesture generation model based on an RNN-transducer, submitted to the GENEA Challenge 2022. The proposed model consists of three neural networks: Encoder, Prediction Network, and Joint Network, which can be jointly trained in an end-to-end manner. We also introduce new loss functions, namely statistical losses, as the additional term to the standard MSE loss to put motion statistics of generated gestures close to the ground truths'. Finally, we show the subjective evaluation results and discuss the results and takeaways from the challenge.