What Have We Achieved on Text Summarization?

Abstract

Deep learning has led to significant improvement in text summarization with various methods investigated and improved ROUGE scores reported over the years. However, gaps still exist between summaries produced by automatic summarizers and human professionals. Aiming to gain more understanding of summarization systems with respect to their strengths and limits on a fine-grained syntactic and semantic level, we consult the Multidimensional Quality Metric 1 (MQM) and quantify 8 major sources of errors on 10 representative summarization models manually. Primarily, we find that 1) under similar settings, extractive summarizers are in general better than their abstractive counterparts thanks to strength in faithfulness and factual-consistency; 2) milestone techniques such as copy, coverage and hybrid extractive/abstractive methods do bring specific improvements but also demonstrate limitations; 3) pre-training techniques, and in particular sequence-to-sequence pre-training, are highly effective for improving text summarization, with BART giving the best results. * Equal contribution. † Corresponding author. 1 MQM is a framework for declaring and describing human writing quality which stipulates a hierarchical listing of error types restricted to human writing and translation.

Keep It Simple: Unsupervised Simplification of Multi-Paragraph Text

“…Modern text generation models are known to hallucinate facts (Huang et al, 2020), which has led the community to create models to detect and correct hallucinations (Cao et al, 2020;.…”

WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation

“…notation task followed Huang et al (2020) and consisted of relevance, consistency, fluency, and coherency.…”

“…In this paper, we focus on coherent paragraph summarization datasets. Automatic evaluation of summarization systems, e.g., by using the ROUGE metric, is challenging (Lloret et al, 2018) and is often inconsistent with human evaluation (Liu and Liu, 2008;Cohan and Goharian, 2016;Tay et al, 2019;Huang et al, 2020). To understand -and later improve -the quality of summarization systems, it is necessary to conduct a human evaluation.…”

A Case Study on User Evaluation of Scientific Publication Summarization by Japanese Students

“…The authors of [11] also compared human responses with automatic evaluation metrics used in text summarization research [12]. Huang et al [19] defined eight errors including missing key points or unnecessary repetition and asked users to manually select errors in computer generated summaries to investigate the limitations of prevailing automatic summarization methods. In these studies, the CNN/DM dataset [20], which consists of news articles, is used for manual evaluation of summaries.…”

“…We employed an unsupervised algorithm, called TextRank [27], to compute sentence importance based on sentence connectivity and generate summaries that consist of important sentences only. TextRank is still a strong baseline which shows competitive performance to a recent summarization method [19]. Note that we exclude the "abstract" and the "reference" sections in the original papers and use only the paper bodies as inputs.…”