VidMP3: Video Editing by Representing Motion with Pose and Position Priors

Sandeep Mishra, Oindrila Saha, Alan Bovik

Keywords: Diffusion models, Video Editing, Test-time adaptation, Motion Priors

TL;DR: We present VidMP3 which learns a generalized motion representation using pose and position priors for challenging video editing tasks.

Abstract: Motion-preserved video editing is crucial for creators, particularly in scenarios that demand flexibility in both the structure and semantics of swapped objects. Despite its potential, this area remains underexplored. Existing diffusion-based editing methods excel in structure-preserving tasks, using dense guidance signals to ensure content integrity. While some recent methods attempt to address structure-variable editing, they often suffer from issues such as temporal inconsistency, subject identity drift, and the need for human intervention. To address these challenges, we introduce VidMP3, a novel approach that leverages pose and position priors to learn a generalized motion representation from source videos. Our method enables the generation of new videos that maintain the original motion while allowing for structural and semantic flexibility. Both qualitative and quantitative evaluations demonstrate the superiority of our approach over existing methods.

Submission Number: 6