Inter-Task Action Correlation for Reinforcement Learning Tasks

Matthew E. Taylor, Peter Stone

Abstract: Introduction Reinforcement learning (RL) problems (Sutton & Barto 1998) are characterized by agents making decisions attempting to maximize total reward, which may be time delayed. RL problems contrast with classical planning problems in that agents do not know a priori how their actions will affect the world. RL differs from supervised learning because agents are never given training examples with the correct action labeled. An RL agent operating successfully in an environment could have its available actions disabled, degraded, or even augmented. Instead of learning from scratch with its new action set, we would like the agent to identify which of the actions in the new set, A′, are most similar to the actions in the original set, A, and then leverage its existing knowledge. In the general problem of transfer learning, a learner trains in a source task and then utilizes its learned knowledge to speed up learning or increase performance in a target task. In this paper, we consider pairs of tasks with different action sets. We present Inter-Task Action Correlation (I-TAC), a method for identifying actions in the target task which are most similar to actions in the source task. This mapping can then be used to automate knowledge transfer by identifying similarities between pairs of RL tasks. We evaluate our algorithm by fully implementing and testing it in the RoboCupsoccer Keepaway domain.