Temporally Abstract Partial Models

Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, Doina Precup

Keywords: reinforcement learning, model-based reinforcement learning, temporal abstraction, options, partial models, affordances

TL;DR: TLDR: We propose temporally abstract partial options models via the notion of affordances, with theoretical guarantees and empirical analysis demonstrating improvement in final performance and sample efficiency.

Abstract: Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup \& Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant or feasible in a given situation, sometimes termed affordable actions. In this paper, we define a notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only in certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we empirically demonstrate the ability to learn both affordances and partial option models online resulting in improved sample efficiency and planning time in the Taxi domain.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/deepmind/affordances_option_models