Toward Affordance Detection and Ranking on Novel Objects for Real-World Robotic Manipulation

Fu-Jen Chu, Ruinian Xu, Landan Seguin, Patricio A. Vela

Abstract: This letter presents a framework to detect and rank affordances of novel objects to assist with robotic manipulation tasks. The framework segments the affordance map of unseen objects using region-based affordance segmentation. Detected affordances define an initial state from which to generate action primitives for manipulation via the planning domain definition language (PDDL). The proposed category-agnostic affordance segmentation approach generalizes learned affordances to unseen objects by utilizing binary classification on proposed instance masks. The predicted pixel-wise level affordances are ranked by KL-divergence, augmenting the available affordance choices for manipulation tasks with non-primary affordances of an object. Experimental results show that the proposed method achieves state-of-the-art performance on affordance segmentation of novel objects, and outperforms baselines on affordance ranking. Actual robotic manipulation scenarios demonstrate the use of affordance detection with PDDL-generated action primitives for task execution. Prediction of ranked affordances on unseen objects provides flexibility to accomplish goal-oriented tasks.