Know Your Action Set: Learning Action Relations for Reinforcement Learning

Ayush Jain, Norio Kosaka, Kyung-Min Kim, Joseph J Lim

Keywords: reinforcement learning, varying action space, relational reasoning

Abstract: Intelligent agents can solve tasks in various ways depending on their available set of actions. However, conventional reinforcement learning (RL) assumes a fixed action set. This work asserts that tasks with varying action sets require reasoning of the relations between the available actions. For instance, taking a nail-action in a repair task is meaningful only if a hammer-action is also available. To learn and utilize such action relations, we propose a novel policy architecture consisting of a graph attention network over the available actions. We show that our model makes informed action decisions by correctly attending to other related actions in both value-based and policy-based RL. Consequently, it outperforms non-relational architectures on applications where the action space often varies, such as recommender systems and physical reasoning with tools and skills. Results and code at https://sites.google.com/view/varyingaction .

One-sentence Summary: Learning action interdependence for reinforcement learning under a varying action space.