Audio-Synchronized Visual Animation

Lin Zhang, Shentong Mo, Yijing Zhang, Pedro Morgado

Abstract: Current visual generation methods can produce high-quality videos guided by text prompts. However, effectively controlling object dynamics remains a challenge. This work explores audio as a cue to generate temporally synchronized image animations. We introduce Audio-Synchronized Visual Animation (ASVA), a task that aims to animate a static image of an object with motions temporally guided by audio clips. To this end, we present AVSync15, a dataset curated from VGGSound with videos featuring synchronized audio-visual events across 15 categories. We also present a diffusion model, AVSyncD, capable of generating audio-guided animations. Extensive evaluations validate AVSync15 as a reliable benchmark for synchronized generation and demonstrate our model’s superior performance. We further explore AVSyncD ’s potential in a variety of audio-synchronized generation tasks, from generating full videos without a base image to controlling object motions with various sounds. We hope our established benchmark can open new avenues for controllable visual generation.

External IDs:doi:10.1007/978-3-031-72940-9_1