Point Policy: Unifying Observations and Actions with Key Points for Robot Manipulation
Workshop Statement: The workshop aims to discuss the ideal data types for learning embodied interactions and ways of acquiring such data efficiently and safely. In this work, we present Point Policy, a method that explores key points as a representation for enabling the learning of robot policies directly from human videos. Key points are easy to obtain, requiring only a few seconds of human effort for each task, and exhibit better sample efficiency than image-based policy learning methods. Further, by design, key points enable generalization to novel object instances as well as robustness to significant environmental variations.
Overall, this work represents key points as a unified observation and action space for learning more capable robot policies. We believe that this aligns well with the theme of this workshop, and the workshop will present a suitable audience for this work.
Keywords: Imitation Learning, Key Points, Human Demonstrations
TL;DR: We present Point Policy, a framework that uses key point based representations for learning robot policies solely from human demonstrations.
Abstract: Building robotic agents capable of operating across diverse environments and object types remains a significant challenge, often requiring extensive data collection. This is particularly restrictive in robotics, where each data point must be physically executed in the real world. Consequently, there is a critical need for alternative data sources for robotics and frameworks that enable learning from such data. In this work, we present Point Policy, a new method for learning robot policies exclusively from offline human demonstration videos and without any teleoperation data. Point Policy leverages state-of-the-art vision models and policy architectures to translate human hand poses into robot poses while capturing object states through semantically meaningful key points. This approach yields a morphology-agnostic representation that facilitates effective policy learning. Our experiments on 8 real-world tasks demonstrate an overall 75% absolute improvement over prior works when evaluated in identical settings as training. Further, Point Policy exhibits a 74% gain across tasks for novel object instances and is robust to significant background clutter. Videos of the robot are best viewed at point-policy.github.io.
Submission Number: 10
Loading