Formal Specification Embeddings for Neuro-Symbolic Autonomy

Beyazit Yalcinkaya, Marcell Vazquez-Chanlatte, Sanjit A. Seshia

Keywords: Reinforcement Learning, Formal Specifications, Representation Learning

Abstract: Formal specifications have long been studied as a means of defining objectives in Reinforcement Learning (RL), largely due to their well-defined operational semantics and compositional nature, which enable correctness guarantees for learned policies. However, most approaches have either been confined to systems trained for a single fixed objective, greatly inhibiting generalization, or required symbolic planning over the induced automata of given specifications, resulting in sub-optimal behaviors. On the other hand, the recent success of foundation models has popularized natural language and demonstrations as expressive instruction modalities. One approach to using these modalities in multi-task RL has been to utilize pretrained text and image embeddings to learn task-conditioned policies. While this approach enables generalization in multi-task systems, it sacrifices any formal correctness guarantees for learned policies. To bridge the gap between the traditional use of formal specifications and more recent techniques that incorporate instruction embeddings in policy learning, in our previous work, we have introduced RAD Embeddings, provably correct pretrained automata embeddings. RAD Embeddings distinguish distinct tasks and encode semantic similarities across a large class of temporal specifications. In turn, policies conditioning on RAD Embeddings provide both formal correctness guarantees on their behaviors and optimal behavior across a wide range of tasks. In this paper, we present our approach for pretraining RAD Embeddings and show its use cases in both single-agent and multi-agent RL settings.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 4