Behavioral Exploration: Learning to Explore via In-Context Adaptation

Andrew Wagenmaker, Zhiyuan Zhou, Sergey Levine

TL;DR: We train adaptive policies that utilize in-context learning to enable online exploration.

Abstract: Developing autonomous agents that quickly explore an environment and adapt their behavior online is a canonical challenge in robotics and machine learning. While humans are able to achieve such fast online exploration and adaptation, often acquiring new information and skills in only a handful of interactions, existing algorithmic approaches tend to rely on random exploration and slow, gradient-based behavior updates. How can we endow autonomous agents with such capabilities on par with humans? Taking inspiration from recent progress on both in-context learning and large-scale behavioral cloning, in this work we propose behavioral exploration: training agents to internalize what it means to explore and adapt in-context over the space of ''expert'' behaviors. To achieve this, given access to a dataset of expert demonstrations, we train a long-context generative model to predict expert actions conditioned on a context of past observations and a measure of how ''exploratory'' the expert's behaviors are relative to this context. This enables the model to not only mimic the behavior of an expert, but also, by feeding its past history of interactions into its context, to select different expert behaviors than what have been previously selected, thereby allowing for fast online adaptation and targeted, ''expert-like'' exploration. We demonstrate the effectiveness of our method in both simulated locomotion and manipulation settings, as well as on real-world robotic manipulation tasks, illustrating its ability to learn adaptive, exploratory behavior.

Lay Summary: In this work we propose a method to train a policy that learns to explore from demonstration data. In particular, our method learns to infer what actions taken by the demonstrator are likely to lead to observing novel states in the future, relative to what has been observed in the past.

Primary Area: Reinforcement Learning

Keywords: in-context learning, exploration, adaptive agents, behavior cloning

Submission Number: 7433