Representation Transfer via Elaboration

Matthew E. Taylor, Peter Stone

Abstract: Introduction A key component of any reinforcement learning (RL) algorithm is the underlying representation used by the agent for learning (e.g. the parameterization of its function approximator). Transfer learning tasks typically look at speeding up a target task after learning in a source task. This paper considers a different, but related, question: is it possible, and desirable, for agents to transfer from a source representation to a target representation? Elaboration, presented below, is a representation transfer (RT) algorithm that may allow an agent to learn faster than learning with a single representation. One motivation for such flexibility is learning speed: changing representations partway through learning may allow agents to achieve better performance in less time. SOAR (Laird, Newell, & Rosenbloom 1987) can use multiple descriptions of planning problems and search problems, generated by a human user, for just this reason. Other work (Sherstov & Stone 2005) suggests that gradually reducing the generalization of a function approximator may improve the speed of learning. In this paper we suggest that it is advantageous to modify the internal representation while learning in some RL tasks, relative to using a fixed representation, so that higher performance is achieved more quickly.