Knowledge Representation for Reinforcement Learning using General Value Functions

Gheorghe Comanici, Doina Precup, Andre Barreto, Daniel Kenji Toyama, Eser Aygün, Philippe Hamel, Sasha Vezhnevets, Shaobo Hou, Shibl Mourad

Abstract: Reinforcement learning (RL) is a very powerful approach for learning good control strategies from data. Value functions are a key concept for reinforcement learning, as they guide the search for good policies. A lot of effort has been devoted to designing and improving algorithms for learning value functions. In this paper, we argue that value functions are also a very natural way of providing a framework for knowledge representation for reinforcement learning agents. We show that generalized value functions provide a unifying lens for many algorithms, including policy gradient, successor features, option models and policies, and other forms of hierarchical reinforcement learning. We also demonstrate the potential of this representation to provide new, useful algorithms.

Keywords: Reinforcement Learning, General Value Functions, Policy Gradient, Hierarchical Reinforcement Learning, Successor Features