Retrieval-Augmented Reinforcement Learning

Anirudh Goyal, Abram L. Friesen, Theophane Weber, Andrea Banino, Nan Rosemary Ke, Adria Puigdomenech Badia, Ksenia Konyushkova, Michal Valko, Simon Osindero, Timothy P Lillicrap, Nicolas Heess, Charles Blundell

Keywords: replay buffer, reinforcement learning, offline RL, attention

Abstract: Most deep reinforcement learning (RL) algorithms distill experience into parametric behavior policies or value functions via gradient updates. While effective, this approach has several disadvantages: (1) it is computationally expensive, (2) it can take many updates to integrate experiences into the parametric model, (3) experiences that are not fully integrated do not appropriately influence the agent's behavior, and (4) behavior is limited by the capacity of the model. In this paper we explore an alternative paradigm in which we train a network to map a dataset of past experiences to optimal behavior. Specifically, we augment an RL agent with a retrieval process (parameterized as a neural network) that has direct access to a dataset of experiences. This dataset can come from the agent's past experiences, expert demonstrations, or any other relevant source. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context, to help the agent achieve its goal faster and more efficiently. We integrate our method into two different RL agents: an offline DQN agent and an online R2D2 agent. In offline multi-task problems, we show that the retrieval-augmented DQN agent avoids task interference and learns faster than the baseline DQN agent. On Atari, we show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores. We run extensive ablations to measure the contributions of the components of our proposed method.

One-sentence Summary: Augment the RL agent with the helper process which helps the RL agent achieve goals faster and in a more efficient way

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2202.08417/code)