Online Laplacian-Based Representation Learning in Reinforcement Learning

Maheed H. Ahmed, Jayanth Bhargav, Mahsa Ghasemi

TL;DR: We propose an online method for learning the Laplacian representation in reinforcement learning, and show theoretically and empirically it converges.

Abstract: Representation learning plays a crucial role in reinforcement learning, especially in complex environments with high-dimensional and unstructured states. Effective representations can enhance the efficiency of learning algorithms by improving sample efficiency and generalization across tasks. This paper considers the Laplacian-based framework for representation learning, where the eigenvectors of the Laplacian matrix of the underlying transition graph are leveraged to encode meaningful features from raw sensory observations of the states. Despite the promising algorithmic advances in this framework, it remains an open question whether the Laplacian-based representations can be learned online and with theoretical guarantees along with policy learning. We address this by formulating an online optimization approach using the Asymmetric Graph Drawing Objective (AGDO) and analyzing its convergence via online projected gradient descent under mild assumptions. Our extensive simulation studies empirically validate the convergence guarantees to the true Laplacian representation. Furthermore, we provide insights into the compatibility of different reinforcement learning algorithms with online representation learning.

Lay Summary:

Reinforcement learning (RL) teaches AI agents to make decisions through interacting with an environment, but it could struggle in complex situations with a large amount of data. One way to improve RL is to teach the agents how to better understand the data by compressing what they see into simpler, more useful pieces. This method is called representation learning.

Our work explores a method inspired by how networks or maps are drawn using something called the graph Laplacian to help the agents create better representations of their environments that capture the environments' dynamics. While prior methods could only build this representation ahead of time before training of the agents starts, we explore how to build these representations on the fly in an online manner. We show theoretically that learning these representations in real-time is possible and that it works reliably in practical conditions.

Link To Code: https://github.com/MaheedHatem/online_laplacian_representation

Primary Area: Reinforcement Learning

Keywords: Reinforcement Learning, Representation learning, Online Learning, Graph Laplacian

Submission Number: 13699