Learning to See Physical Properties with Active Sensing Motor PoliciesDownload PDF

Published: 30 Aug 2023, Last Modified: 18 Oct 2023CoRL 2023 PosterReaders: Everyone
Keywords: Locomotion, Vision, Navigation, Reinforcement Learning
TL;DR: A legged robot learns to measure terrain properties more accurately by intentionally feeling the ground; its feelings serve as self-supervised labels for a visual perception module; this facilitates locomotion planning in diverse configurations.
Abstract: To plan efficient robot locomotion, we must use the information about a terrain’s physics that can be inferred from color images. To this end, we train a visual perception module that predicts terrain properties using labels from a small amount of real-world proprioceptive locomotion. To ensure label precision, we introduce Active Sensing Motor Policies (ASMP). These policies are trained to prefer motor skills that facilitate accurately estimating the environment’s physics, like swiping a foot to observe friction. The estimated labels supervise a vision model that infers physical properties directly from color images and can be reused for different tasks. Leveraging a pretrained vision backbone, we demonstrate robust generalization in image space, enabling path planning from overhead imagery despite using only ground camera images for training.
Student First Author: yes
Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)
Website: https://gmargo11.github.io/active-sensing-loco/
Publication Agreement: pdf
Poster Spotlight Video: mp4
16 Replies