Long-Span Summarization via Local Attention and Content Selection

Abstract

Transformer-based models have achieved state-of-the-art results in a wide range of natural language processing (NLP) tasks including document summarization. Typically these systems are trained by fine-tuning a large pretrained model to the target task. One issue with these transformer-based models is that they do not scale well in terms of memory and compute requirements as the input length grows. Thus, for long document summarization, it can be challenging to train or fine-tune these models. In this work, we exploit large pre-trained transformer-based models and address long-span dependencies in abstractive summarization using two methods: local self-attention; and explicit content selection. These approaches are compared on a range of network configurations. Experiments are carried out on standard long-span summarization tasks, including Spotify Podcast, arXiv, and PubMed datasets. We demonstrate that by combining these methods, we can achieve state-of-the-art results on all three tasks in the ROUGE scores. Moreover, without a large-scale GPU card, our approach can achieve comparable or better results than existing approaches. 1

Pretrained Language Models for Text Generation: A Survey

“…Compared with sequence, graph can aggregate relevant disjoint context by uniformly representing them as nodes and their relations as edges [136,191]. As a representative example, Wu et al [191] [76]; DANCER [53]; Manakul et al [130] Preservation CSP [203]; mRASP [112]; Wada et al [181] Fig. 2.…”

“…Efficiency is an important factor to consider for modeling long documents, especially when generating long text. Since the self-attention mechanism grows quadratically with sequence length, many works aim to improve the encoding efficiency of selfattention [76,130]. A representative example is Manakul et al [130] proposed two methods: local self-attention, allowing longer input spans during training; and explicit content selection, reducing memory and compute requirements.…”

Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems

“…Vanilla Transformers. We use BART (Lewis et al, 2020) and local-attention BART (LoBART) (Manakul and Gales, 2021) as our base models. BART's maximum input length is 1024, while that of LoBART is 4096 with attention width of 1024.…”

“…Time and memory are dominated by the encoder self-attention, and models such as LoBART adopt local attention in its encoder to mitigate this bottleneck, while keeping the original decoder (Manakul and Gales, 2021). Training is fast because attention is highly parallelizable.…”

“…A.1 Models BART/LoBART: We use the HuggingFace's implementation (Wolf et al, 2020), including BART models fine-tuned to CNNDM 5 and XSum 6 . We take LoBART from Manakul and Gales (2021), including LoBART(4k)+MCS fine-tuned to Podcast and arXiv. MCS is the multitask content selection system for handling the Podcast/arXiv input documents that exceed 4096 words.…”