Focus Attention: Promoting Faithfulness and Diversity in Summarization

Abstract

Professional summaries are written with document-level information, such as the theme of the document, in mind. This is in contrast with most seq2seq decoders which simultaneously learn to focus on salient content, while deciding what to generate, at each decoding step. With the motivation to narrow this gap, we introduce Focus Attention Mechanism, a simple yet effective method to encourage decoders to proactively generate tokens that are similar or topical to the input document. Further, we propose a Focus Sampling method to enable generation of diverse summaries, an area currently understudied in summarization. When evaluated on the BBC extreme summarization task, two state-of-the-art models augmented with Focus Attention generate summaries that are closer to the target and more faithful to their input documents, outperforming their vanilla counterparts on ROUGE and multiple faithfulness measures. We also empirically demonstrate that Focus Sampling is more effective in generating diverse and faithful summaries than top-k or nucleus samplingbased decoding methods.

Survey of Hallucination in Natural Language Generation

“…The encoder is expected to turn input text in meaningful representations so a model can comprehend the input. When encoders learn wrong correlations between different parts of the training data, it could result in erroneous generation that diverges from the input [2,48,98,172].…”

“…Attention Mechanism is an integral implement selectively concentrating on relevant parts while ignoring others based on dependencies in neural networks [4,177]. In order to encourage the generator to pay more attention to the source, Aralikatte et al [2] introduce a short circuit from the input document to the vocabulary distribution via source-conditioned bias. Krishna et al [85] employ sparse attention to improve the model's long-range dependencies in the hope of modeling more retrieved documents to mitigate the hallucination in the answer.…”

“…1 Hallucination Definition in Data-to-Text Generation 10. 2 Hallucination Metrics in Data-to-Text Generation 10. 3 Hallucination Mitigation in Data-to-Text Generation 10.…”

“…1 Hallucinations Definition and Categories in NMT 11. 2 Hallucination Metrics in NMT 11.3 Hallucination Mitigation Methods in NMT 11.4 Future Directions in NMT 12 Conclusion References…”

CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

“…First, a separate correction model is learned to fix errors made by the summarizers (Zhao et al, 2020;, including replacing entities absent from the source or revising all possible errors (Cao et al, 2020). The second type targets at modifying the sequence-to-sequence architecture to incorporate relation triplets (Cao et al, 2018), knowledge graphs (Zhu et al, 2021), and topics (Aralikatte et al, 2021) to inform the summarizers of article facts. Yet additional engineering efforts and model retraining are often needed.…”