How well do you know your summarization datasets?

Abstract

State-of-the-art summarization systems are trained and evaluated on massive datasets scraped from the web. Despite their prevalence, we know very little about the underlying characteristics (data noise, summarization complexity, etc.) of these datasets, and how these affect system performance and the reliability of automatic metrics like ROUGE. In this study, we manually analyse 600 samples from three popular summarization datasets. Our study is driven by a six-class typology which captures different noise types (missing facts, entities) and degrees of summarization difficulty (extractive, abstractive). We follow with a thorough analysis of 27 state-of-the-art summarization models and 5 popular metrics, and report our key insights: (1) Datasets have distinct data quality and complexity distributions, which can be traced back to their collection process. (2) The performance of models and reliability of metrics is dependent on sample complexity. (3) Faithful summaries often receive low scores because of the poor diversity of references. We release the code, annotated data and model outputs. 1

An Empirical Survey on Long Document Summarization: Datasets, Models, and Metrics

“…Glaringly, in our experiment, the problem of noisy data afects more than 60% of the annotated ground truth summaries in the randomly sampled arXiv test set. This is greater than 54% detected in XSUM dataset [111]. While many of the errors and noises are minor, more than 15% of the reference summaries have signiicant errors where at least half of the summary contains errors, rendering the summaries to be unreadable.…”

“…For example, Beltagy et al [1] ine-tuned BERT on large-scale scientiic paper datasets and have found its performance to improve in scientiic domains as compared to the BERT-base model. Evidenced by Tejaswin et al [111]'s experimental result where BERTScore is found not to discriminate summaries with and without errors well, this questions the use of BERT-base model as the "independent evaluator" of candidate summaries across all domains.…”

“…Publicly available benchmark datasets have been introduced to evaluate the performance of summarization models. Nonetheless, the benchmark datasets have diferent intrinsic characteristics that have been found to be crucial in the understanding of model performances [81,111], summarization approach suitability (i.e., extractive or abstractive approach) [104,128] and evaluation metrics efectiveness [31,89]. Hence, only through a comprehensive understanding of the benchmark datasets, one can assess the underlying performance and applicability of a summarization model in the real-world settings [61].…”

“…Further, insights drawn from benchmark datasets have led to the introduction of state-of-the-art models across a wide range of natural language processing (NLP) tasks [13,123], including the text summarization task [25,37,77]. In response, intrinsic dataset evaluation through large-scale automatic evaluation [4] or more ine-grained human evaluation at a smaller scale [111] has also been performed to enhance the understanding of various benchmarks. Nevertheless, none of the aforementioned works performed a large-scale automatic evaluation analysis nor a thorough human evaluation of benchmark datasets in the long document text summarization domain.…”