Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents

Zhang, Yusen; Ni, Ansong; Mao, Ziming; Wu, Chen; Zhu, Chenguang; Deb, Budhaditya; Awadallah, Ahmed Hassan; Radev, Dragomir; Zhang, Rui

doi:10.48550/arxiv.2110.10150

Cited by 5 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hierarchical models [ 23 ] iteratively merge paragraph-level dependencies. Segmentation-based approaches [ 24 , 25 , 26 ] with fusion-in-decoder [ 27 ] and marginalized decoding [ 28 ] divide the input into meaningful units to produce a summary. Extract-then-abstract procedures [ 29 ] pick a subset of relevant sentences from the source to generate the outline, eventually relying on marginalization [ 30 , 31 ].…”

Section: Related Workmentioning

confidence: 99%

Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes

Moro

Ragazzi

Valgimigli

et al. 2023

Sensors

View full text Add to dashboard Cite

Long document summarization poses obstacles to current generative transformer-based models because of the broad context to process and understand. Indeed, detecting long-range dependencies is still challenging for today’s state-of-the-art solutions, usually requiring model expansion at the cost of an unsustainable demand for computing and memory capacities. This paper introduces Emma, a novel efficient memory-enhanced transformer-based architecture. By segmenting a lengthy input into multiple text fragments, our model stores and compares the current chunk with previous ones, gaining the capability to read and comprehend the entire context over the whole document with a fixed amount of GPU memory. This method enables the model to deal with theoretically infinitely long documents, using less than 18 and 13 GB of memory for training and inference, respectively. We conducted extensive performance analyses and demonstrate that Emma achieved competitive results on two datasets of different domains while consuming significantly less GPU memory than competitors do, even in low-resource settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes

Moro

Ragazzi

Valgimigli

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…While there do exist more powerful dialogue summarization models such as DialogLM [29] and Summ [28], we use the BART (Bidirectional and Auto-Regressive Transformers) model [12] due to its speed and high performance in long document summarization tasks [11]. In addition, there has been previous research in assessing different topic segmentation methods on the BART model, so this allows us to evaluate our techniques.…”

Section: Bart Model For Meeting Summarizationmentioning

confidence: 99%

Action-Item-Driven Summarization of Long Meeting Transcripts

Golia,

Kalita

2023

Proceedings of the 2023 7th International Conference on Natural Language Processing and Information Retrieval

View full text Add to dashboard Cite

The increased prevalence of online meetings has significantly enhanced the practicality of a model that can automatically generate the summary of a given meeting. This paper introduces a novel and effective approach to automate the generation of meeting summaries. Current approaches to this problem generate general and basic summaries, considering the meeting simply as a long dialogue. However, our novel algorithms can generate abstractive meeting summaries that are driven by the action items contained in the meeting transcript. This is done by recursively generating summaries and employing our action-item extraction algorithm for each section of the meeting in parallel. All of these sectional summaries are then combined and summarized together to create a coherent and action-item-driven summary. In addition, this paper introduces three novel methods for dividing up long transcripts into topic-based sections to improve the time efficiency of our algorithm, as well as to resolve the issue of large language models (LLMs) forgetting long-term dependencies. Our pipeline achieved a BERTScore of 64.98 across the AMI corpus, which is an approximately 4.98% increase from the current state-of-the-art result produced by a fine-tuned BART (Bidirectional and Auto-Regressive Transformers) model. 1

show abstract

“…The simplest approach involves truncating the lengthy input text into a shorter sequence within a predefined maximum length [9 , 10]. While this allows the use of off-the-shelf LLMs, it is heavily influenced by lead bias and can lead to significant information loss as the document length increases [11,12,13]. Another approach, text chunking, involves breaking down a long document into smaller, semantically similar segments and processing each segment independently before aggregation [8,14].…”

Section: Related Workmentioning

confidence: 99%

Information Extraction from Lengthy Legal Contracts: Leveraging Query-Based Summarization and GPT-3.5

Zin,

Nguyen,

Satoh

et al. 2023

Frontiers in Artificial Intelligence and Applications

View full text Add to dashboard Cite

In the legal domain, extracting information from contracts poses significant challenges, primarily due to the scarcity of annotated data. In such situations, leveraging large language models (LLMs), such as the Generative Pretrained Transformer (GPT) models, offers a promising solution. However, the inherent token limitations of these models can be a bottleneck for processing lengthy legal contracts. This paper presents an unsupervised two-step approach to address these challenges. First, we propose a query-based summarization model that extracts sentences pertinent to predefined queries, concisely representing lengthy contracts. This summarization ensures that the core information remains intact while simultaneously addressing the token limitation issue. Subsequently, the generated summary is fed to GPT-3.5 for precise information extraction. Our approach effectively overcomes the challenges of token limitations and zero resources, enabling efficient and scalable information extraction from legal contracts. We compare our results with those obtained from supervised models that have been fine-tuned on domain-specific annotated data. Experimental results demonstrate the remarkable effectiveness of our approach, as it achieves state-of-the-art performance without the need for domain-specific training data.

show abstract

Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents

Cited by 5 publications

References 33 publications

Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes

Efficient Memory-Enhanced Transformer for Long-Document Summarization in Low-Resource Regimes

Action-Item-Driven Summarization of Long Meeting Transcripts

Information Extraction from Lengthy Legal Contracts: Leveraging Query-Based Summarization and GPT-3.5

Contact Info

Product

Resources

About