MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusion

Roy Kapon, Guy Tevet, Amit Haim Bermano

Keywords: diffusion, prior, wild, generation, motion, generative, SDS, dream, dreamfusion

Abstract:

We introduce Multi-view Ancestral Sampling (MAS), a method for generating consistent multi-view 2D samples of a motion sequence, enabling the creation of its corresponding 3D counterpart.
While abundant 2D samples are readily available, such as those found in videos, 3D data collection is involved and expensive, often requiring specialized motion-capture systems. MAS leverages diffusion models trained solely on 2D data to produce coherent and realistic 3D motions.
This is achieved by simultaneously applying multiple ancestral samplings to denoise multiple 2D sequences representing the same motion from different angles.
Our consistency block ensures 3D consistency at each diffusion step by combining the individual generations into a unified 3D sequence, and projecting it back to the original views for the next iteration.
We evaluate MAS using 2D pose data from intricate and unique motions, including professional basketball maneuvers, rhythmic gymnastic performances featuring ball apparatus interactions, and horse obstacle course races. In each of these domains, MAS generates diverse, high-quality, and unprecedented 3D sequences that would otherwise require expensive equipment and intensive human labor to obtain.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2659