Dialogue Specific Pre-training Tasks for Improved Dialogue State Tracking

Jinwon An, Misuk Kim

Abstract: Although Pre-trained language models are widely used in dialogue state tracking, there exists little work on pre-training tasks that are designed for dialogue state tracking. In this perspective, we propose simple and effective pre-training tasks of language models that are specifically designed for dialogue state tracking. The first is modified slot prediction, which is a pre-training task that makes a binary prediction for detecting slots that change their value from the previous turn in the current turn of the dialogue. The second is next dialogue prediction, which is also a binary prediction pre-training task of finding whether a part of a given recent dialogue context is replaced with excerpts from other dialogues or not. Experimental results suggest that combing our pre-training tasks achieves a significant improvement of 4.51% point over a model trained without additional pre-training. This is impressive in that additional data is not used for the pre-training. In addition, ablation studies show how each pre-training task affects the performance. Specifically, our pre-training tasks works best when it is used in the pre-training phase rather than in the fine-tuning phase. Also, longer pre-training helps fine-tuning performance.