Partially Observable Reinforcement Learning with Memory Traces

Onno Eberhard, Michael Muehlebach, Claire Vernade

Keywords: Reinforcement learning theory, Partial observability, Memory

TL;DR: Eligibility traces are more effective than sliding windows as a memory mechanism for RL in POMDPs.

Abstract: Partially observable environments present a considerable computational challenge in reinforcement learning due to the need to consider long histories. Learning with a finite window of observations quickly becomes intractable as the window length grows. In this work, we introduce *memory traces*. Inspired by eligibility traces, these are compact representations of the history of observations in the form of exponential moving averages. We prove sample complexity bounds for the problem of offline on-policy evaluation that quantify the return errors achieved with memory traces for the class of Lipschitz continuous value estimates. We establish a close connection to the window approach, and demonstrate that, in certain environments, learning with memory traces is significantly more sample efficient. Finally, we underline the effectiveness of memory traces empirically in online reinforcement learning experiments for both value prediction and control.

Confirmation: I understand that authors of each paper submitted to EWRL may be asked to review 2-3 other submissions to EWRL.

Serve As Reviewer: ~Onno_Eberhard1

Track: Fast Track: published work

Publication Link: [e-mail]

Submission Number: 77