Revisiting Laplacian Representations for Value Function Approximation in Deep RL

Priyesh Vijayan, Padideh Nouri, Rishav Rishav, Sarath Chandar, Yash Chandak, Mathieu Reymond, Samira Ebrahimi Kahou, Doina Precup

Keywords: Laplacian, Eigenvectors, Successor Representations, Proto-value networks, PQN, ALE

TL;DR: Laplacian Eigenvectors as features for Parallelized Q Networks

Abstract: Proto-value functions (PVFs) introduced Laplacian embeddings as an effective feature basis for value-function approximation; however, their utility remained limited to small, fully known state spaces. Recent work has scaled Laplacian embeddings to high-dimensional inputs, using them for reward shaping and option discovery in goal-directed tasks, yet only as auxiliary signals, rather than directly using them as features for value functions. In this paper, we learn Laplacian eigenvectors online and employ them as features for Q-learning in 23 Atari games. We empirically demonstrate that these online–learned embeddings substantially improve model-free RL in large, high-dimensional domains. We demonstrate that enriching state representations with action embeddings yields additional gains under both behavior-policy and uniform-random policies. Additionally, we introduce the Fusion architecture, which augments the representation with useful inductive bias at the embedding level. To assess the usefulness of each embedding used in the Fusion architecture, we use Shapley values analysis.

Submission Number: 10