An Empirical Study of Multitask Learning to Improve Open Domain Dialogue Systems

Mehrdad Farahani, Richard Johansson

Keywords: auxiliary tasks, open-domain dialogue, multitask learning

TL;DR: auxiliary tasks in decoder-only open-domain dialogue models

Abstract: Autoregressive models used to generate responses in open-domain dialogue systems often struggle to take long-term context into account and to maintain consistency over a dialogue. Previous research in open-domain dialogue generation has shown that the use of \emph{auxiliary tasks} can introduce inductive biases that encourage the model to improve these qualities. However, most previous research has focused on encoder-only or encoder/decoder models, while the use of auxiliary tasks in \emph{encoder-only} autoregressive models is under-explored. This paper describes an investigation where four different auxiliary tasks are added to small and medium-sized GPT-2 models fine-tuned on the PersonaChat and DailyDialog datasets. The results show that the introduction of the new auxiliary tasks leads to small but consistent improvement in evaluations of the investigated models.

Student Paper: Yes, the first author is a student