Making Task-Oriented Dialogue Datasets More Natural by Synthetically Generating Indirect User Requests
Abstract: Existing benchmark corpora of task-oriented dialogue are collected using either a "machines talking to machines" approach or by giving template-based goal descriptions to crowdworkers. These methods, however, often produce utterances that are markedly different from natural human conversations in which people often convey their preferences in indirect ways, such as through small talk. We term such utterances as indirect utterances (iius). Understanding such iius demands considerable world knowledge and reasoning capabilities on the listener's part. Our study introduces a large language model (LLM)-based pipeline to automatically generate realistic, high-quality iius for a given domain, with the ultimate goal of improving the robustness of dialogue state tracking (DST) models for task-oriented dialogue systems. Our findings show that while large LLMs such as GPT-3.5 and GPT-4 generate high-quality iius, achieving similar quality with smaller models is more challenging. We release IndirectRequests, a dataset of iius based off the SGD dataset, and show that it provides a challenging testbed to evaluate the "in the wild" performance of natural language understanding (NLU) and DST models.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: evaluation and metrics, task-oriented, commonsense reasoning, dialogue state tracking, automatic evaluation, NLP datasets
Contribution Types: NLP engineering experiment, Data resources, Data analysis
Languages Studied: English
Section 2 Permission To Publish Peer Reviewers Content Agreement: Authors grant permission for ACL to publish peer reviewers' content
Submission Number: 83
Loading