How Animals Dance (When You're Not Looking)

Xiaojuan Wang, Aleksander Holynski, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz

Keywords: animal dance video generation; choreography pattern; long video generation

Abstract: We present a framework for generating music-synchronized, choreography aware animal dance videos. Our framework introduces choreography patterns---structured sequences of motion beats that define the long-range structure of a dance---as a novel high-level control signal for dance video generation. These patterns can be automatically estimated from human dance videos. Starting from a few keyframes representing distinct animal poses, generated via text-to-image prompting or GPT-4o, we formulate dance synthesis as a graph optimization problem that seeks the optimal keyframe structure to satisfy a specified choreography pattern of beats. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 seconds dance videos across a wide range of animals and music tracks.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 6370