XLVIN: eXecuted Latent Value Iteration Nets

Andreea Deac, Petar Veličković, Ognjen Milinković, Pierre-Luc Bacon, Jian Tang, Mladen Nikolić

Keywords: value iteration, graph neural networks, reinforcement learning

TL;DR: We combine contrastive self-supervised learning, graph representation learning and neural algorithm execution to perform value iteration in the latent space, generalising VINs to arbitrary domains.

Abstract: Value Iteration Networks (VINs) have emerged as a popular method to perform implicit planning within deep reinforcement learning, enabling performance improvements on tasks requiring long-range reasoning and understanding of environment dynamics. This came with several limitations, however: the model is not explicitly incentivised to perform meaningful planning computations, the underlying state space is assumed to be discrete, and the Markov decision process (MDP) is assumed fixed and known. We propose eXecuted Latent Value Iteration Networks (XLVINs), which combine recent developments across contrastive self-supervised learning, graph representation learning and neural algorithmic reasoning to alleviate all of the above limitations, successfully deploying VIN-style models on generic environments. XLVINs match the performance of VIN-like models when the underlying MDP is discrete, fixed and known, and provide significant improvements to model-free baselines across three general MDP setups.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2010.13146/code)