Detecting Hallucinated Content in Conditional Neural Sequence Generation

Abstract

Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety of fluent but wrong outputs are particularly problematic, as it will not be possible for users to tell they are being presented incorrect content. To detect these errors, we propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually annotated evaluation sets for this task. We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations. Experiments on machine translation (MT) and abstractive summarization demonstrate that our proposed approach consistently outperforms strong baselines on all benchmark datasets. We further demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods. We also apply our method to word-level quality estimation for MT and show its effectiveness in both supervised and unsupervised settings 1 . * Most work was done during an internship at FAIR.

Retrieval Augmentation Reduces Hallucination in Conversation

“…Hallucination in text-generation models is a topic that has received attention recently, particularly in the settings of summarization (Maynez et al, 2020), machine translation (Zhou et al, 2021), and news generation (Zellers et al, 2019). For dialogue, it has been observed in state-of-the-art models (Roller et al, 2021) and studied in depth (Mielke et al, 2020), but so far without resolution.…”

Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization

“…There has been a lot of recent work in abstractive summarization showing that state-of-the-art systems suffer from generating inconsistent information with respect to the source article, despite their improved success in producing fluent summaries (Falke et al, 2019;Lux et al, 2020). Since word-overlap based metrics such as ROUGE have low correlation with human scores of faithfulness (Kryscinski et al, 2019;Fabbri et al, 2020), there has been significant effort to develop automated metrics that can detect such errors (Zhou et al, 2021;Gabriel et al, 2021;Pagnoni et al, 2021a). For example, Falke et al (2019), Maynez et al (2020) and Goyal and Durrett (2020) have proposed to assess faithfulness using entailment models, where a faithful summary should be assigned a high entailment score with respect to the original article.…”

Miðeind's WMT 2021 submission

“…We note the relatively small training time of the mBART adaptation and the lack of Icelandic data in the pretraining task for mBART as primary factors that can be addressed for improving results. Additionally online (or semi-online) self-training instead of train-then-translate would also improve results, especially with selective loss truncation as described in (Zhou et al, 2021). The data selected for backtranslation should also be expanded for greater diversity of both genre and vocabulary.…”