Learning Multimodal Probabilistic Models of Manipulation Skill Effects

Siyeon Kim, Tucker Hermans, Yixuan Huang, Mohanraj Devendran Shanthi

Keywords: Robot learning, Probabilistic skill effect models, Mixture density networks (MDNs), State Uncertainty, Sim-to-real transfer

TL;DR: We learn probabilistic skill effect models to capture state uncertainty in dynamics for a single object, given a partial-view point cloud.

Abstract: Robotic manipulation has advanced in simulations, yet real-world deployments face challenges due to uncertainty. This challenge is significant for sim-to-real learning due to the reality gap caused by modeling approximations and parameter mismatches between simulation and the real world. Uncertainty arises not only from sensor noise and recognition errors but also from unmodeled dynamics, which are often overlooked. In simple tasks like pushing, small changes in push height or contact points can lead to different outcomes, highlighting the need for models that capture multimodality. We thus propose a probabilistic skill effect model with latent dynamics and a mixture density network for single-step outcome prediction, which takes a partial point cloud and an action to predict a multimodal distribution of object pose changes. Tested on three YCB objects, our model outperformed deterministic and unimodal baselines in pose prediction and uncertainty quality. Additionally, it successfully separates pose modes and aligns predicted topple probabilities with empirical data.

Serve As Reviewer: ~Siyeon_Kim1

Submission Number: 22