Proto Successor Measure: Representing the Behavior Space of an RL Agent

Siddhant Agarwal, Harshit Sikchi, Peter Stone, Amy Zhang

TL;DR: A zero shot RL method that represents all possible policies in the MDP

Abstract: Having explored an environment, intelligent agents should be able to transfer their knowledge to most downstream tasks within that environment without additional interactions. Referred to as "zero-shot learning", this ability remains elusive for general-purpose reinforcement learning algorithms. While recent works have attempted to produce zero-shot RL agents, they make assumptions about the nature of the tasks or the structure of the MDP. We present *Proto Successor Measure*: the basis set for all possible behaviors of a Reinforcement Learning Agent in a dynamical system. We prove that any possible behavior (represented using visitation distributions) can be represented using an affine combination of these policy-independent basis functions. Given a reward function at test time, we simply need to find the right set of linear weights to combine these bases corresponding to the optimal policy. We derive a practical algorithm to learn these basis functions using reward-free interaction data from the environment and show that our approach can produce the near-optimal policy at test time for any given reward function without additional environmental interactions. Project page: agarwalsiddhant10.github.io/projects/psm.html.

Lay Summary: Any human can perform a large number of tasks in any environment by simply exploring it initially. Humans have a wonderful ability to form an abstraction about their surroundings, which enables them to solve any task very quickly. We investigate whether this ability can be transferred to any reinforcement learning agent. Using the mathematical properties of the environment, we construct an abstract space where any behavior of the agent can be represented. Given any task, the agent simply needs to search in this space to find the most performant behavior that solves the task.

Primary Area: Reinforcement Learning

Keywords: Unsupervised Reinforcement Learning, Zero Shot Reinforcement Learning, Representation Learning

Submission Number: 7354