Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation

Xuxin Cheng, Zhihong Zhu, Wanshi Xu, Yaowei Li, Hongxiang Li, Yuexian Zou

Submission Type: Regular Long Paper

Submission Track: Dialogue and Interactive Systems

Keywords: Multiple Intent Detection and Slot Filling, Knowledge Distillation, Non-Autoregressive

Abstract: Recent non-autoregressive Spoken Language Understanding (SLU) models attracts increasing attention owing to the high inference speed. However, most of them still (1) suffer from the multi-modality problem since the prior knowledge about the reference is relatively poor during inference; (2) fail to achieve a satisfactory inference speed limited by their complex frameworks. To tackle these problems, in this paper, we propose a $\textbf{T}$argeted $\textbf{K}$nowledge $\textbf{D}$istillation $\textbf{F}$ramework (TKDF), which applies knowledge distillation to improve the performance. Specifically, we first train an SLU model as a teacher model, which has higher accuracy while slower inference speed. Then we introduce an evaluator and utilize the curriculum learning strategy to select proper targets for the student model. Experiment results on two public multi-intent SLU datasets demonstrate that our method can realize a flexible trade-off between inference speed and accuracy, achieving comparable performance to the state-of-the-art models while speeding up by over 4.5 times.

Submission Number: 838