Successor Feature Representations

Chris Reinke, Xavier Alameda-Pineda

Abstract: Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

- updated Figure 2 a by depicting the agent the features
- added acknowledgments on page 11
- modified equations 16 and 41 to contain the indicator function, instead of two cases, for the observed feature and the unobserved features
- minor updates to proofs and explanations according to the comments of the third reviewer (see https://openreview.net/forum?id=MTFf1rDDEI&noteId=ELzMKfMCPa for details)

Video: https://youtu.be/RYemlh2NSzQ

Code: https://gitlab.inria.fr/robotlearn/sfr_learning

Assigned Action Editor: ~Marcello_Restelli1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 698