DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis

Zixuan Wang, Jiayi Li, Xiaoyu Qin, Shikun Sun, Songtao Zhou, Jia Jia, Jiebo Luo

Abstract: Synthesizing camera movement from music and dance is highly challenging due to the contradictions and complexities of dance cinematography. Unlike human movement, which is always continuous, dance camera movements involve continuous sequences of varying lengths and sudden drastic changes to simulate the switching of multiple cameras. However, in previous works, every camera frame is equally treated which results in jittering and unavoidable smoothing post-processing. To solve this problem, in this paper, we propose to formalize animator dance cinematography knowledge by formulating this problem as a three-stage process: keyframe detection, keyframe synthesis, and tween curve prediction. Following this formulation, we design a novel end-to-end dance camera synthesis framework \textbf{DanceCamAnimator}, which first imitates human animation procedure and shows powerful keyframe-based controllability with variable length. Extensive experiments on the DCM dataset demonstrate that our method surpasses previous works quantitatively and qualitatively. We will make our code publicly available to promote future research.

Primary Subject Area: [Experience] Art and Culture

Secondary Subject Area: [Experience] Multimedia Applications, [Generation] Generative Multimedia

Relevance To Conference: 3D Dance camera generation is a multi-modal processing problem because it needs to consider information from multiple modalities such as music and dance movements. After processing, the model produces 3D camera motion data, which can ultimately be applied to 2D videos or 3D scenes to generate results across multiple modalities.

Submission Number: 2054