PhysMotion: Physics-Grounded Dynamics From a Single Image

Xiyang Tan, Ying Jiang, Xuan Li, Tianyi Xie, Zeshun Zong, Yin Yang, Chenfanfu Jiang

Keywords: Generative Dynamics, Physics Based Simulation

TL;DR: PhysMotion is a novel framework that leverages principled physics-based simulations to guide intermediate 3D representations generated from a single image and input conditions, producing high-quality, physically plausible video generation.

Abstract:

We introduce PhysMotion, a novel framework that leverages principled physics-based simulations to guide intermediate 3D representations generated from a single image and input conditions (e.g., applied force and torque), producing high-quality, physically plausible video generation. By utilizing continuum mechanics-based simulations as a prior knowledge, our approach addresses the limitations of traditional data-driven generative models and result in more consistent physically plausible motions. Our framework begins by reconstructing a feed-forward 3D Gaussian from a single image through geometry optimization. This representation is then time-stepped using
Material Point Method (MPM) with continuum mechanics-based elastoplasticity models, which provides a strong foundation for realistic dynamics, albeit at a coarse level of detail. To enhance the geometry, appearance, and ensure spatiotemporal consistency, we refine the initial simulation using a text-to-image (T2I) diffusion model with cross-frame attention, resulting in a physically plausible video that retains intricate details comparable to the input image. We conduct comprehensive qualitative and quantitative evaluations to validate the efficacy of our method.

Submission Number: 268