Detecting Hallucinated Content in Conditional Neural Sequence Generation

Chunting Zhou, Jiatao Gu, Mona T. Diab, Paco Guzmán, Luke Zettlemoyer, Marjan Ghazvininejad

Keywords: conditional text generation, hallucination detection, sequence generation evaluation, neural machine translation, abstractive text summarization

Abstract:

Neural sequence models can generate highly fluent sentences but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input, which can cause a lack of trust in the model.
To better assess the faithfulness of the machine outputs, we propose a new task to predict whether each token in the output sequence is hallucinated conditioned on the source input, and collect new manually annotated evaluation sets for this task.
We also introduce a novel method for learning to model hallucination detection, based on pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations.
Experiments on machine translation and abstract text summarization demonstrate the effectiveness of our proposed approach -- we obtain an average F1 of around 0.6 across all the benchmark datasets.
Furthermore, we demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in the low-resource machine translation and achieve significant improvements over strong baseline methods.
We will also release our annotated data and code for future research.

One-sentence Summary: We propose a new task to predict hallucinated words in the machine output conditioned on the source input and a novel general-purpose method to learn this task.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2011.02593/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=xPZLjwJU3