IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Models

Yiyang Ling, Karan Sharad Owalekar, Oluwatobiloba Adesanya, Erdem Biyik, Daniel Seita

Keywords: safe physical interaction, contact-rich manipulation, motion planning, contextual reasoning, anisotropic cost map, semantic understanding, foundation models for robotics

TL;DR: We use Vision-Language Models to infer which objects a robot can contact, how and where to safely push them, encoding this into a directional cost map that enables contact-aware motion planning in clutter.

Abstract: Motion planning involves determining a sequence of robot configurations to reach a desired pose, subject to movement and safety constraints. Traditional motion planning finds collision-free paths, but this is overly restrictive in clutter, where it may not be possible for a robot to accomplish a task without contact. In addition, contacts range from relatively benign (eg brushing a soft pillow) to more dangerous (eg toppling a glass vase), making it difficult to characterize which may be acceptable. In this paper, we propose IMPACT, a novel motion planning framework that uses Vision-Language Models (VLMs) to infer environment semantics, identifying which parts of the environment can best tolerate contact based on object properties and locations. Our approach generates an anisotropic cost map that encodes directional push safety. We pair this map with a contact-aware A* planner to find stable contact-rich paths. Our results over 3200 simulation and 200 real-world trials suggest that IMPACT enables efficient contact-rich motion planning in cluttered settings while outperforming alternative methods and ablations.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 2