Exploring transfer learning for end-to-end spoken language understanding

Chengwei Su, Wael Hamza, Subendhu Rongali, Liwei Cai, Konstantine Arkoudas

Abstract:

Voice Assistants such as Alexa, Siri, and Google Assistant
typically use a two-stage Spoken Language Understanding
pipeline; first, an Automatic Speech Recognition (ASR) com-
ponent to process customer speech and generate text tran-
scriptions, followed by a Natural Language Understanding
(NLU) component to map transcriptions to an actionable hy-
pothesis. An end-to-end (E2E) system that goes directly from
speech to a hypothesis is a more attractive option. These sys-
tems were shown to be smaller, faster, and better optimized.
However, they require massive amounts of end-to-end train-
ing data and in addition, don’t take advantage of the already
available ASR and NLU training data.
In this work, we propose an E2E system that is designed
to jointly train on multiple speech-to-text tasks, such as
ASR (speech-transcription) and SLU (speech-hypothesis),
and text-to-text tasks, such as NLU (text-hypothesis). We call
this the Audio-Text All-Task (AT-AT) Model and we show
that it beats the performance of E2E models trained on in-
dividual tasks, especially ones trained on limited data. We
show this result on an internal music dataset and two public
datasets, FluentSpeech and SNIPS Audio, where we achieve
state-of-the-art results. Since our model can process both
speech and text input sequences and learn to predict a target
sequence, it also allows us to do zero-shot E2E SLU by train-
ing on only text-hypothesis data (without any speech) from
a new domain. We evaluate this ability of our model on the
Facebook TOP dataset and set a new benchmark for zeroshot
E2E performance. We release the audio data collected for the
TOP dataset for future research.