Foundation Models for History Compression in Reinforcement Learning

Fabian Paischer, Thomas Adler, Andreas Radler, Markus Hofmarcher, Sepp Hochreiter

Keywords: Deep Reinforcement Learning, Language Models, History Compression, Partial Observability, Foundation Models

Abstract:

Agents interacting under partial observability require access to past observations via a memory mechanism in order to approximate the true state of the environment.
Recent work suggests that leveraging language as abstraction provides benefits for creating a representation of past events.
History Compression via Language Models (HELM) leverages a pretrained Language Model (LM) for representing the past.
It relies on a randomized attention mechanism to translate environment observations to token embeddings.
In this work, we show that the representations resulting from this attention mechanism can collapse under certain conditions.
This causes blindness of the agent to subtle changes in the environment that may be crucial for solving a certain task.
We propose a solution to this problem consisting of two parts.
First, we improve upon HELM by substituting the attention mechanism with a feature-wise centering-and-scaling operation.
Second, we take a step toward semantic history compression by leveraging foundation models, such as CLIP, to encode observations, which further improves performance.
By combining foundation models, our agent is able to solve the challenging MiniGrid-Memory environment.
Surprisingly, however, our experiments suggest that this is not due to the semantic enrichment of the representation presented to the LM, but rather due to the discriminative power provided by CLIP.
We make our code publicly available at https://github.com/ml-jku/helm.