Automata Conditioned Reinforcement Learning with Experience Replay

Beyazit Yalcinkaya, Niklas Lauffer, Marcell Vazquez-Chanlatte, Sanjit Seshia

Confirmation: I have read and confirm that at least one author will be attending the workshop in person if the submission is accepted

Keywords: reinforcement learning, goal-conditioned reinforcement learning, formal methods, experience replay

TL;DR: We show that experience replay helps for goal-conditioned RL with automata-based goals.

Abstract: We explore the problem of goal-conditioned reinforcement learning (RL) where goals are represented using deterministic finite state automata (DFAs). Due to the sparse and binary nature of automata-based goals, we hypothesize that experience replay can help an RL agent learn more quickly and consistently in this setting. To enable the use of experience replay, we introduce a novel end-to-end neural architecture, including a graph neural network (GNN) to encode the DFA goal before passing it to a feed-forward policy network. Experimental results in a gridworld domain demonstrate the efficacy of the model architecture and highlight the significant role of experience replay in enhancing the learning speed and reducing the variance of RL agents for DFA tasks.

Submission Number: 31