From Appearance to Motion: Aligning Visual Representations for Robotic Manipulation

Eadom T Dessalene, Dehao Yuan, Michael Maynord, Nathan S. Pan, Cornelia Fermuller, Yiannis Aloimonos

Keywords: pretrained, frozen, motion, features, policy, behavioral-cloning

TL;DR: We pre-train motion sensitive representations, finding they outperform generic vision backbones in behavioral cloning.

Abstract: Pre-trained vision models used in robotics often misalign with manipulation tasks due to the loss used to train these vision models being focused on appearance rather than motion. In order to enhance motion encoding within vision models, we introduce a simple novel contrastive training framework that operates over predictions of motion. After training over EPIC Kitchens, model evaluations on behavioral cloning show a improvement in success rate over state-of-the-art methods across a benchmark of $3$ environments and $21$ object manipulation tasks.

Primary Area: applications to robotics, autonomy, planning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8950