An Exploratory Study on Long Dialogue Summarization: What Works and What’s Next

Abstract

Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series. However, real-world dialogues pose a great challenge to current summarization models, as the dialogue length typically exceeds the input limits imposed by recent transformer-based pretrained models, and the interactive nature of dialogues makes relevant information more context-dependent and sparsely distributed than news articles. In this work, we perform a comprehensive study on long dialogue summarization by investigating three strategies to deal with the lengthy input problem and locate relevant information: (1) extended transformer models such as Longformer, (2) retrieve-then-summarize pipeline models with several dialogue utterance retrieval methods, and (3) hierarchical dialogue encoding models such as HMNet. Our experimental results on three long dialogue datasets (QMSum, Me-diaSum, SummScreen) show that the retrievethen-summarize pipeline models yield the best performance. We also demonstrate that the summary quality can be further improved with a stronger retrieval model and pretraining on proper external summarization datasets. * Equal Contribution. ‡ The work was done when Asli was at MSR.

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

“…This extra step reduces the burden on the neural summarizers that have to generate an abstract summary and select important content at the same time. Some also refer models that utilize the hybrid approach as retrieve-the-summarize model because it involves retrieving a subset of long document text before summarizing it [131]. TLM+Ext [94] irst implemented this method by limiting inputs of the scientiic articles in arXiv datasets as the introduction of the document, a subset of carefully selected sentences of the original article using extractive summarization approach, and, inally, include the remaining text if there remains extra space for Transformer-based decoder.…”

“…Other than the discourse bias mechanism, we observe that (a) eicient attention and (b) content selection mechanisms are the two most notable long document mechanisms. As the content selection mechanism requires a separate retriever to extract salient content from the source (i.e., the hybrid approach), we distinguish Transformer models with content selection mechanism as the retrieve-then-summarize model [131] and the pure encoder-decoder Transformer without this mechanism as an end-to-end model for the rest of this work. Lastly, it is also important to note that both mechanisms can be jointly implemented within a single architecture, where the content selection mechanism will extract a longer subset of input to be processed by a Transformer with eicient attention [77].…”

“…Each approach has its advantages and limitations that may suit certain summarization tasks better. For example, extractive summarization may be suicient in summarizing certain news articles [15,128] but inadequate to summarize a long dialogue where salient content are sparsely distributed [131]. This is because while the extractive summarization approach is always factually consistent with the source document, it does not modify the original text and thus lacks the ability to generate luent and concise summary [120].…”

Topic Knowledge Based Controlled Generation for Long Documents Using Retrieval-Based Language Models

“…(3) Our summaries adhere to people-personalized preferences. In comparisons with strong LLM summarization methods on the standard datasets MACSum [4] and arXiv [5], which contain long document summarization examples, our approach demonstrated clear advantages.…”

DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization

“…The results of the Longformer-based model in AMI and ICSI are from(Fabbri et al 2021), and the results of it in QMSum come from(Zhang et al 2021b). In screenplay domain, the results of Longformer are fromChen et al (2021a).3 BART-large-CNN refers to further fine-tuning BART-large on the news summarization dataset CNN/DailyMail.…”