Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters

Abstract

This paper presents an efficient graphenhanced approach to multi-document summarization (MDS) with an encoder-decoder Transformer model. This model is based on recent advances in pre-training both encoder and decoder on very large text data , and it incorporates an efficient encoding mechanism (Beltagy et al., 2020) that avoids the quadratic memory growth typical for traditional Transformers. We show that this powerful combination not only scales to large input documents commonly found when summarizing news clusters; it also enables us to process additional input in the form of auxiliary graph representations, which we derive from the multi-document clusters. We present a mechanism to incorporate such graph information into the encoder-decoder model that was pre-trained on text only. Our approach leads to significant improvements on the Multi-News dataset, overall leading to an average 1.8 ROUGE score improvement over previous work . We also show improvements in a transfer-only setup on the DUC-2004 dataset. The graph encodings lead to summaries that are more abstractive. Human evaluation shows that they are also more informative and factually more consistent with their input documents. 1

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

“…512/256 and 1024/1024 respectively) on all of the datasets (Except for the zero-shot experiments, the details can be found in Sec.3.3). 10 We use the same length limit as our 9 pilot experiments show that simple truncation results in inferior performance, which is also in line with Pasunuru et al (2021) 10 Regarding the length limit of inputs for PEGASUS, we do experiments with 512, 1024, 4096 on Multi-News dataset, and the model with length limit 512 achieves the best performance, model for the LED model, i.e. 4096/1024 for input and output respectively, for all the datasets.…”

“…PEGASUS (Zhang et al, 2020) a pre-trained model designed for abstractive summarization as downstream task, especially for the single document input. It is trained on the objective of Gap Similar to Pasunuru et al (2021), the inputs of all the models are the concatenations of the documents within the clusters (in the same order), each document is truncated based on the input length limit divided by total number of documents so that all documents are represented in the input 9 . To preserve the same format as the corresponding pre-trained models we set the inference length limit of input and output for BART and PEGASUS exactly as their pre-trained settings (i.e.…”

“…Neural Multi-Document Summarization Neural multi-document summarizerization models can be roughly categorized into two classes, graphbased models (Liao et al, 2018;Pasunuru et al, 2021) and hierarchical models (Liu and Lapata, 2019a;Fabbri et al, 2019;Jin et al, 2020). The graph-based models typically leverage additional information, e.g.…”

“…State-of-the-art approaches to multi-document summarization are mostly graph-based (Liao et al, 2018;Pasunuru et al, 2021), leveraging graph neural networks to connect information between the documents, or hierarchical (Liu and Lapata, 2019a;Fabbri et al, 2019;Jin et al, Figure 1: Prior work on pre-training mostly focuses on single document representation and does not target a specific downstream task. In contrast, PRIMER is a pre-trained model for multi-document processing with focus on summarization.…”

Pretrained Language Models for Text Generation: A Survey

“…In order to adapt to long-form text and alleviate quadratic complexity of full-attention operation, sparse attention is applied in the self-attention module for long-form input. Rather than attending to all other tokens, every token only attends to specific tokens with strategies such as window attention [130,142,209], global attention [142,209], random attention [209] and Sinkhorn attention [228].…”