Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience

Haresh Karnan, Elvin Yang, Daniel Farkash, Garrett Warnell, Joydeep Biswas, Peter Stone

Keywords: Vision-based Navigation, Representation Learning, Learning from Experience

TL;DR: We propose a novel self-supervised terrain representation learning algorithm that can learn relevant representations from unconstrained, unlabelled robot experience.

Abstract: Terrain awareness, i.e., the ability to sufficiently represent key differences in terrain, is a critical ability that robots must have in order to be able to succeed at autonomous off-road navigation. Current approaches that provide robots with this awareness are prohibitively expensive, requiring curated datasets with extensive human labeling effort or vast amounts of data gathered during expert-level driving. Towards endowing robots with terrain awareness without these expenses, we introduce Self-supervised TErrain Representation LearnING (STERLING), a novel approach for learning terrain representations that relies solely on easy-to-collect, unconstrained (e.g., non-expert), and unlabelled robot experience. STERLING employs a novel multi-modal self-supervision objective through non-contrastive representation learning to learn relevant terrain representations for terrain-aware navigation. Through physical robot experiments in off-road environments, we evaluate STERLING features on the task of operator-preference-aligned visual navigation and find that STERLING features perform on par with fully-supervised approaches and outperform other state-of-the-art methods with respect to preference alignment. Additionally, we perform a large-scale experiment of autonomously hiking a 3-mile long trail which STERLING completes successfully with only two manual interventions, demonstrating its robustness to real-world off-road conditions.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/self-supervised-terrain-representation/code)