Handling Open-Vocabulary Constructs in Formalizing Specifications: Retrieval Augmented Parsing with Expert Knowledge
Research Area: Alignment, Data, Evaluation, Learning algorithms for LMs, LMs with tools and code
Keywords: Open Vocabulary Constructs, Retrieval Augmented Parsing,
TL;DR: We propose a ROLex, a retrieval augmented parsing mechanism, to handle open vocabulary construct (constructs not known before encounter) problem in formalizing specifications such as Linear Temporal Logic, Python code etc.
Abstract: We study the problem of Open-vocabulary constructs (OVCs), ones that are not known beforehand, in the context of converting natural
language (NL) specification sentences into formal languages (e.g., LTL or code). Models tend to fare poorly on such OVCs, since they do
not have the necessary knowledge a priori. In such settings, a domain expert can provide the correct constructs based on their
preference or domain knowledge at inference time. Our goal is to effectively reuse this inference-time, expert-provided knowledge in future specification sentences without having to retrain the model. To this end, we first present a new parsing setting---\emph{dynamic knowledge-augmented parsing} (DKAP)---where, in addition to the input sentence, the model is given (dynamically growing) expert knowledge in the form of a key-value lexicon that associates NL phrases with correct OVC constructs. To address the DKAP problem, we propose ROLex, a retrieval-augmented parsing approach that uses the dynamic expert lexicon. ROLex consists of a retriever and a generator that are trained to find and use the relevant subset of the key-value store to produce the correct parse. One key challenge in realizing this solution is the lack of training data for the retrieval-augmented parsing. We show how we can make use of synthetic data generation, along with original task-level training data---i.e., the (NL sentence, FL statement) pairs---to carry out the requisite training for the retrieval-augmented parsing setting. Further, to improve training effectiveness, we have devised multiple strategies for focusing the model on the relevant subset of retrieved knowledge. Finally, we introduce a new evaluation paradigm designed to address the DKAP problem by simulating the dynamic expert-provided knowledge in three different formalization settings (NL2LTL, NL2Code, and NL2CMD). Our evaluations show that DKAP is a difficult challenge, and ROLex helps improve the performance of baseline models by using dynamic expert knowledge effectively.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html
Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html
Submission Number: 487
Loading