PuppetMaster: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

ICLR 2025 Conference Submission1631 Authors

Keywords: Video generation, motion, diffusion models

TL;DR: We present an interactive video generator to model part-level object dynamics.

Abstract: We present PuppetMaster, a video generator that understands part-level object dynamics. Given an image of an object and a number of drags defining the desired trajectory of selected points of the object, PuppetMaster synthesizes a video where the object moves according to the specified drags in a physically plausible manner. PuppetMaster is obtained by fine-tuning an off-the-shelf video diffusion model, extended with a new component that encodes the input drags. PuppetMaster also introduces all-to-first attention, a replacement for the common spatial attention module, which removes artifacts that arise from fine-tuning a video generator out-of-domain and significantly improves the quality of the synthesized videos. PuppetMaster is learned from Objaverse-Animation-HQ, a new dataset of curated part-level motion clips obtained by rendering synthetic 3D animations. We propose strategies to automatically filter out sub-optimal animations and augment the synthetic renderings with meaningful drags. By using this data, PuppetMaster learns to generate part-level motions, unlike other motion-conditioned video generators that mostly move the object as a whole. PuppetMaster generalizes well to real images, outperforming existing methods in real-world benchmarks in a zero-shot manner. We refer the reader to the supplementary material for video visualizations.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1631