Influence-aware Memory for Deep Reinforcement Learning

Miguel Suau, Elena Congeduti, Rolf A.N. Starre, Aleksander Czechowski, Frans A. Oliehoek

Keywords: Deep Reinforcement Learning, POMDP, Influence, Memory, Recurrent Neural Networks

Abstract: Making the right decisions when some of the state variables are hidden, involves reasoning about all the possible states of the environment. An agent receiving only partial observations needs to infer the true values of these hidden variables based on the history of experiences. Recent deep reinforcement learning methods use recurrent models to keep track of past information. However, these models are sometimes expensive to train and have convergence difficulties, especially when dealing with high dimensional input spaces. Taking inspiration from influence-based abstraction, we show that effective policies can be learned in the presence of uncertainty by only memorizing a small subset of input variables. We also incorporate a mechanism in our network that learns to automatically choose the important pieces of information that need to be remembered. The results indicate that, by forcing the agent's internal memory to focus on the selected regions while treating the rest of the observable variables as Markovian, we can outperform ordinary recurrent architectures in situations where the amount of information that the agent needs to retain represents a small fraction of the entire observation input. The method also reduces training time and obtains better scores than methods that use a fixed window of experiences as input to remove partial observability in domains where long-term memory is required.

Code: https://www.dropbox.com/home/InfluenceNet