Generative 3D Object Particle Dynamics

ICLR 2026 Conference Submission14797 Authors

Keywords: Dynamics Model

Abstract: We introduce ParticleDiffuser, a particle-based 3D trajectory diffusion model that represents scenes as evolving particle graphs, enabling the capture of complex action–object interactions and object deformations. Unlike existing 3D particle dynamics models, which typically rely on deterministic action-conditioned predictors constrained to narrow domains (e.g., individual cloth or soft-body objects), ParticleDiffuser adopts a generative approach trained on large-scale simulated data of deformable and soft objects, and capturing multimodality of future particle tra- jectories. To support efficient spatiotemporal reasoning, ParticleDiffuser introduces learnable latent vectors that fuse information across particles and employs autoregressive rollouts with latent-variable attention across sequential frame seg- ments, enabling long-horizon 3D video generation. We present two variants: (i) an action-conditioned particle trajectory generator, and (ii) a joint action–object particle trajectory generator. By directly modeling the joint distribution of object particles and actions within a single diffusion process, ParticleDiffuser allows goal-conditioned action generation by steering diffusion toward desired future configurations, eliminating the costly trajectory searches required by traditional MPC methods. Experiments show that ParticleDiffuser generalizes to diverse objects and actions in simulated and real-world settings where deterministic graph-based particle networks quickly fail. It also substantially outperforms MPC baselines in both accuracy and efficiency on manipulation tasks involving a broad spectrum of object types, including rigid and deformable bodies.

Primary Area: generative models

Submission Number: 14797