Automata Conditioned Reinforcement Learning with Experience Replay

Published: 03 Nov 2023, Last Modified: 27 Nov 2023GCRL WorkshopEveryoneRevisionsBibTeX
Confirmation: I have read and confirm that at least one author will be attending the workshop in person if the submission is accepted
Keywords: reinforcement learning, goal-conditioned reinforcement learning, formal methods, experience replay
TL;DR: We show that experience replay helps for goal-conditioned RL with automata-based goals.
Abstract: We explore the problem of goal-conditioned reinforcement learning (RL) where goals are represented using deterministic finite state automata (DFAs). Due to the sparse and binary nature of automata-based goals, we hypothesize that experience replay can help an RL agent learn more quickly and consistently in this setting. To enable the use of experience replay, we introduce a novel end-to-end neural architecture, including a graph neural network (GNN) to encode the DFA goal before passing it to a feed-forward policy network. Experimental results in a gridworld domain demonstrate the efficacy of the model architecture and highlight the significant role of experience replay in enhancing the learning speed and reducing the variance of RL agents for DFA tasks.
Submission Number: 31