Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue

Anonymous

Keywords: Dialogue, Task Oriented Dialogue, User Simulator, Neural Augmentation

TL;DR: Generate synthetic task-oriented conversations fully neurally with neural user simulator and assistant models; self-train on the successful conversations. Repeat. Profit.

Abstract:

We investigate the ability of large language models to neurally generate Task Oriented Dialogues in novel domains, provided only with an API implementation and a list of goals.
We show these simulations formulate online, automatic metrics that correlate well with human evaluations. Furthermore, by filtering for dialogues where goals are met, we can use simulation to repeatedly generate training data and improve the quality of the dialogues themselves. With no human intervention or domain-specific training data, our simulations bootstrap end-to-end models which achieve a 37\% error reduction over baseline in previously unseen domains. By including as few as 32 domain-specific conversations bootstrapped models can match the performance of a fully-supervised model with $10\times$ more data.