Highlight-Transformer: Leveraging Key Phrase Aware Attention to Improve Abstractive Multi-Document Summarization

Abstract

ive multi-document summarization aims to generate a comprehensive summary covering salient content from multiple input documents. Compared with previous RNNbased models, the Transformer-based models employ the self-attention mechanism to capture the dependencies in input documents and can generate better summaries. Existing works have not considered key phrases in determining attention weights of self-attention. Consequently, some of the tokens within key phrases only receive small attention weights. It can affect completely encoding key phrases that convey the salient ideas of input documents. In this paper, we introduce the Highlight-Transformer, a model with the highlighting mechanism in the encoder to assign greater attention weights for the tokens within key phrases. We propose two structures of highlighting attention for each head and the multihead highlighting attention. The experimental results on the Multi-News dataset show that our proposed model significantly outperforms the competitive baseline models.

Towards Legal Judgment Summarization: A Structure-Enhanced Approach

“…BERT [5] and BART [22], two famous language models, utilize the Transformer Encoder-Decoder structure to obtain summaries. Based on the structure, previous research introduce Entity Aggregation [10], Key Phrases Detection [14], Sentence Structure Relations [1], and Time Content Selection [4] to generate summaries.…”

Generating a Structured Summary of Numerous Academic Papers: Dataset and Method

“…To compose the pairs of input and output, they match body sections with abstract's sections by section titles. Meng et al [2021] with target summary's sections. These SDS works can utilize the explicit formats (e.g., the division of sections) of input documents and target summaries to determine the alignment relationships between the input and output.…”

“…But writing a survey paper needs a lot of time and effort, making it difficult to cover the latest papers and all the research topics. The multidocument summarization (MDS) techniques [Liu et al, 2018;Fabbri et al, 2019;Liu et al, 2021;Liu et al, 2022] can be utilized to automatically produce summaries as a supplement to human-written summaries. To cover the latest papers and more research topics at a low cost, people can flexibly adjust the input papers and let the summarization methods produce summaries for these papers.…”

Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness

“…However, extractive methods often suffer the coherence problem (Wu and Hu, 2018). Therefore, instead of directly extracting sentences from the articles, abstractive methods that can rewrite the articles achieve great success with the advantages of large annotated corpora (Pang et al, 2021;Zhou et al, 2021;Liu et al, 2021a;Zhong et al, 2020;Liu and Lapata, 2019a).…”

“…We compare REFLECT with several strong baselines (Liu and Lapata, 2019a;Gehrmann et al, 2018;Fabbri et al, 2019;Perez-Beltrachini and Lapata, 2021;Liu et al, 2021a;Zhong et al, 2020;Zhang et al, 2020a;Pasunuru et al, 2021) on Multi-News (Fabbri et al, 2019), Multi-XScience (Lu et al, 2020) and WikiCat-Sum (Perez-Beltrachini et al, 2019) corpora, derived from news, academic domains and Wikipedia, respectively. Due to space limit, the results of Multi-XScience and WikiCatSum are provided in the Appendix A.…”