LTL-Constrained Policy Optimization with Cycle Experience Replay
Abstract: Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many settings, LTL is insufficient for task specification; LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. Prior methods for this constrained problem are restricted to finite state spaces, limiting its applicability in deep Reinforcement Learning (DRL) settings. In this work, we present Cycle Experience Replay (CyclER), a reward-shaping approach to this problem that succeeds in using DRL to learn performant policies in continuous state and action spaces. CyclER guides a policy towards satisfaction by encouraging partial behaviors compliant with the LTL constraint, using the structure of the constraint. In doing so, it addresses the optimization challenges stemming from the sparse nature of LTL satisfaction. We evaluate CyclER in three continuous control domains. On these tasks, CyclER outperforms existing reward-shaping methods at finding effective and LTL-satisfying policies.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Matteo_Papini1
Submission Number: 3426
Loading