Human Motion Diffusion as a Generative Prior

Yoni Shafir, Guy Tevet, Roy Kapon, Amit Haim Bermano

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: diffusion, prior, composition, generation, motion, generative

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Recent work has demonstrated the significant potential of denoising diffusion models
for generating human motion, including text-to-motion capabilities.
However, these methods are restricted by the paucity of annotated motion data,
a focus on single-person motions, and a lack of detailed control.
In this paper, we introduce three forms of composition based on diffusion priors:
sequential, parallel, and model composition.
Using sequential composition, we tackle the challenge of long sequence
generation. We introduce DoubleTake, an inference-time method with which
we generate long animations consisting of sequences of prompted intervals
and their transitions, using a prior trained only for short clips.
Using parallel composition, we show promising steps toward two-person generation.
Beginning with two fixed priors as well as a few two-person training examples, we learn a slim
communication block, ComMDM, to coordinate interaction between the two resulting motions.
Lastly, using model composition, we first train individual priors
to complete motions that realize a prescribed motion for a given joint.
We then introduce DiffusionBlending, an interpolation mechanism to effectively blend several
such models to enable flexible and efficient fine-grained joint and trajectory-level control and editing.
We evaluate the composition methods using an off-the-shelf motion diffusion model,
and further compare the results to dedicated models trained for these specific tasks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: generative models

Submission Number: 2385