Informed POMDP: Leveraging Additional Information in Model-Based RL

Gaspard Lambrechts, Adrien Bolland, Damien Ernst

Keywords: POMDP, RNN, Sufficient Statistic, Model-Based, Privileged Information, Asymmetric Learning

TL;DR: We introduce the informed POMDP to account for additional information during training, propose an objective for learning a sufficient statistic and a world model using this information, which increases the convergence speed of the policies in POMDPs.

Abstract: In this work, we generalize the problem of learning through interaction in a POMDP by accounting for eventual additional information available at training time. First, we introduce the informed POMDP, a new learning paradigm offering a clear distinction between the information at training and the observation at execution. Next, we propose an objective that leverages this information for learning a sufficient statistic of the history for the optimal control. We then adapt this informed objective to learn a world model able to sample latent trajectories. Finally, we empirically show a learning speed improvement in several environments using this informed world model in the Dreamer algorithm. These results and the simplicity of the proposed adaptation advocate for a systematic consideration of eventual additional information when learning in a POMDP using model-based RL.

Submission Number: 105