Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

Abstract

Despite significant progress in neural abstractive summarization, recent studies have shown that the current models are prone to generating summaries that are unfaithful to the original context. To address the issue, we study contrast candidate generation and selection as a model-agnostic post-processing technique to correct the extrinsic hallucinations (i.e. information not present in the source text) in unfaithful summaries. We learn a discriminative correction model by generating alternative candidate summaries where named entities and quantities in the generated summary are replaced with ones with compatible semantic types from the source document. This model is then used to select the best candidate as the final output summary. Our experiments and analysis across a number of neural summarization systems show that our proposed method is effective in identifying and correcting extrinsic hallucinations. We analyze the typical hallucination phenomenon by different types of neural summarization systems, in hope to provide insights for future work on the direction.

Survey of Hallucination in Natural Language Generation

“…Multiple terminologies, such as faithfulness [20,22,50,117,133,144,144,163,172,195,219], factual consistency [18,19,24,154,157,194], fidelity [23], factualness 4 [146], factuality 4 [33],…”

“…or on the other hand, hallucination [40,73,107,154,158], fact contradicting [129] are used in the human evaluation of hallucination to rate whether the generated text is in accord with the source input. Chen et al [22], Nie et al [130] use finer-grained metrics for intrinsic hallucination and extrinsic hallucination separately. Moreover, there are some broad metrics, such as Correctness [7,12,98,182], Accuracy [97,203], and Informativeness [102] considering both missing and additional contents (extrinsic hallucinations) compared to the input source.…”

“…Consequently, a better semantic understanding helps alleviate the divergence issue from the source. For example, augmented with entity information [107], extracted relation triples from source document [20,73] obtained by Fact Description Extraction, synthetic data generated through replacement or perturbation [22,91], retrieved external knowledge [12,45,65,158,222], and retrieved similar training samples [13].…”

Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization

“…Improving faithfulness of summarization systems is essential for deploying these systems in realworld scenarios, as such recent work has studied methods to improve the faithfulness of abstractive summarization systems (Zhao et al, 2020;Dong et al, 2020;Goyal and Durrett, 2021;Xu et al, 2020;Chen et al, 2021;Zhu et al, 2021). For example, Goyal and Durrett (2021) train summarization systems by modifying the training objective to maximize the likelihood of the subset of summary tokens that are considered faithful according to their factuality detection model.…”

“…Order determined by coin flip. as well as methods to improve faithfulness of generated summaries (Kang and Hashimoto, 2020;Chen et al, 2021). Intuitively, one straightforward way of improving faithfulness of generated summaries is to copy a larger amount of content from the source article (i.e.…”