Uncertainty Estimation in Autoregressive Structured Prediction

Andrey Malinin, Mark Gales

Keywords: ensembles, structures prediction, uncertainty estimation, knowledge uncertainty, autoregressive models, information theory, machine translation, speech recognition.

Abstract: Uncertainty estimation is important for ensuring safety and robustness of AI systems. While most research in the area has focused on un-structured prediction tasks, limited work has investigated general uncertainty estimation approaches for structured prediction. Thus, this work aims to investigate uncertainty estimation for structured prediction tasks within a single unified and interpretable probabilistic ensemble-based framework. We consider: uncertainty estimation for sequence data at the token-level and complete sequence-level; interpretations for, and applications of, various measures of uncertainty; and discuss both the theoretical and practical challenges associated with obtaining them. This work also provides baselines for token-level and sequence-level error detection, and sequence-level out-of-domain input detection on the WMT’14 English-French and WMT’17 English-German translation and LibriSpeech speech recognition datasets.

One-sentence Summary: A Deep Investigation of Ensemble-based Uncertainty Estimation for Autoregressive ASR and NMT models.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [LibriSpeech](https://paperswithcode.com/dataset/librispeech)