DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation

Jiangran Lyu

Keywords: World Action Model, Non-prehensile Manipulation

Abstract:

Nonprehensile manipulation is crucial for handling objects that are
too thin, large, or otherwise ungraspable in unstructured environments. While
conventional planning-based approaches struggle with complex contact modeling,
learning-based methods have recently emerged as a promising alternative. How-
ever, existing learning-based approaches face two major limitations: they heavily
rely on multi-view cameras and precise pose tracking, and they fail to generalize
across varying physical conditions, such as changes in object mass and table fric-
tion. To address these challenges, we propose the Dynamics-Adaptive World Ac-
tion Model (DyWA), a novel framework that enhances action learning by jointly
predicting future states while adapting to dynamics variations based on histori-
cal trajectories. By unifying the modeling of geometry, state, physics, and robot
actions, DyWA enables more robust policy learning under partial observability.
Compared to baselines, our method improves the success rate by 31.5% using
only single-view point cloud observations in the simulation. Furthermore, DyWA
achieves an average success rate of 68% in real-world experiments, demonstrat-
ing its ability to generalize across diverse object geometries, adapt to varying table
friction, and robustness in challenging scenarios such as half-filled water bottles
and slippery surfaces.

Submission Number: 9