Way-Tu: A Framework for Tool Selection and Manipulation Using Waypoint Representations

Ece Kunduracioglu, Ozgur S. Oguz

Keywords: Learning Robot Fine Manipulation Skills, Tool Manipulation and Selection, Learning Waypoint Representations, Motion Optimization

TL;DR: We present Way-TU, a framework that jointly addresses tool selection and manipulation by learning waypoint (3D oriented keypoint) representations for motion optimization.

Abstract: The ability to manipulate tools is essential for integrating intelligent robots in real-world settings, allowing them to significantly expand the range of tasks they can perform in daily life. To address this challenge, we introduce Way-TU, a novel framework that learns to generate waypoint representations (3D oriented keypoints) for motion planning in tool-use tasks. Our approach perceives the full environment, reasons over object geometry, and generates waypoints to guide the motion optimizer toward task completion, simultaneously enabling tool selection by identifying the most suitable tool among candidates. We evaluated our framework on three tasks—minigolf, lifting, and hammering—and demonstrated competitive manipulation performance against baselines and effective tool selection capabilities.

Submission Number: 22