Exploration by Running Away from the Past

Paul-Antoine LE TOLGUENEC, Yann Besse, Florent Teichteil-Königsbuch, Dennis George Wilson, Emmanuel Rachelson

Keywords: Reinforcement Learning, Exploration, Deep Learning

TL;DR: This study presents a new exploratory algorithm in Reinforcement Learning, where the agent explores by moving away from its past experiences using two distinct approaches.

Abstract:

The ability to explore efficiently and effectively is a central challenge of reinforcement learning.
In this work, we consider exploration through the lens of information theory.
Specifically, we cast exploration as a problem of maximizing the Shannon entropy of the state occupation measure.
This is done by maximizing a sequence of divergences between distributions representing an agent's past behavior and its current behavior.
Intuitively, this encourages the agent to explore new behaviors that are distinct from past behaviors.
Hence, we call our method RAMP, for ``$\textbf{R}$unning $\textbf{A}$way fro$\textbf{m}$ the $\textbf{P}$ast.''
A fundamental question of this method is the quantification of the distribution change over time.
We consider both the Kullback-Leibler divergence and the Wasserstein distance to quantify divergence between successive state occupation measures, and explain why the former might lead to undesirable exploratory behaviors in some tasks.
We demonstrate that by encouraging the agent to explore by actively distancing itself from past experiences, it can effectively explore mazes and a wide range of behaviors on robotic manipulation and locomotion tasks.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7454