Learning to See Physical Properties with Active Sensing Motor Policies

Gabriel B. Margolis, Xiang Fu, Yandong Ji, Pulkit Agrawal

Keywords: Locomotion, Vision, Navigation, Reinforcement Learning

TL;DR: A legged robot learns to measure terrain properties more accurately by intentionally feeling the ground; its feelings serve as self-supervised labels for a visual perception module; this facilitates locomotion planning in diverse configurations.

Abstract: To plan efficient robot locomotion, we must use the information about a terrain’s physics that can be inferred from color images. To this end, we train a visual perception module that predicts terrain properties using labels from a small amount of real-world proprioceptive locomotion. To ensure label precision, we introduce Active Sensing Motor Policies (ASMP). These policies are trained to prefer motor skills that facilitate accurately estimating the environment’s physics, like swiping a foot to observe friction. The estimated labels supervise a vision model that infers physical properties directly from color images and can be reused for different tasks. Leveraging a pretrained vision backbone, we demonstrate robust generalization in image space, enabling path planning from overhead imagery despite using only ground camera images for training.

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Website: https://gmargo11.github.io/active-sensing-loco/

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/learning-to-see-physical-properties-with/code)