Diffusion-Based Approximate Value Functions

Martin Klissarov, Doina Precup

Keywords: spectral graph theory, geometric deep learning, graph convolutional network, reinforcement learning, deep geometric learning

TL;DR: Solving sparse rewards environments by diffusing the reward signal through Graph Convolutional Networks.

Abstract: We present a novel model-based framework inspired by spectral graph theory and deep geometric learning: the Diffusion-based Approximate Value Function. Our approach efficiently approximates the graph Laplacian of an MDP's underlying graph by using Graph Convolutional Networks (GCN). By generating an approximate value function, we diffuse the reward signal much faster than traditional Reinforcement Learning algorithms such as TD(0). This leads to substantial improvements on sparse rewards environments where efficient credit assignment is most demanding.