Causal Document-Grounded Dialogue Pre-training

Yingxiu Zhao, Bowen Yu, Bowen Li, Haiyang Yu, Jinyang Li, Chao Wang, Fei Huang, Yongbin Li, Nevin L. Zhang

Submission Type: Regular Long Paper

Submission Track: Dialogue and Interactive Systems

Submission Track 2: Natural Language Generation

Keywords: Document-grounded dialogue, task specific pretraining, causal effect

TL;DR: We construct a causally complete dataset for document-grounded dialogue, and propose a causally-perturbed pre-training strategy.

Abstract: The goal of document-grounded dialogue (DocGD) is to generate a response by anchoring the evidence in a supporting document in accordance with the dialogue context. This entails four causally interconnected variables. While task-specific pre-training has significantly enhanced performances on numerous downstream tasks, existing DocGD methods still rely on general pre-trained language models without a specifically tailored pre-training approach that explicitly captures the causal relationships. To address this, we present the first causally-complete dataset construction strategy for developing million-scale DocGD pre-training corpora. Additionally, we propose a causally-perturbed pre-training strategy to better capture causality by introducing perturbations on the variables and optimizing the overall causal effect. Experiments conducted on three benchmark datasets demonstrate that our causal pre-training yields substantial and consistent improvements in fully-supervised, low-resource, few-shot, and zero-shot settings.

Submission Number: 2668