What Have We Achieved on Text Summarization?

Huang, Dandan; Cui, Leyang; Yang, Shicheng; Bao, Guangsheng; Wang, Kun; Xie, Jun; Zhang, Yue

doi:10.48550/arxiv.2010.04529

Cited by 3 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The author further argues that community has exerted over crafting algorithms to improve performance evaluation scores on the benchmark data-sets, thereby limiting progress in the science of automatic extractive document summarization. Similar searching questions have been asked by Huang et al (2020), who design a multi-dimensional quality metric and quantify major sources of errors on well-known summarization models. The authors underscore faithfulness and factual-consistency of extractive summaries compared to abstractive counterparts, based on the designed metric.…”

Section: Introductionmentioning

confidence: 91%

“…During the last two years, there has been a spurt in research related to metrics for summary quality (Peyrard, 2019b;Bhandari et al, 2020a;Huang et al, 2020;Vasilyev & Bohannon, 2020;Fabbri et al, 2020;Bhandari et al, 2020b). Most of these works have argued against the ROUGE metric because it fails to robustly match paraphrases resulting in misleading scores, which do not correlate well with human judgements (Zhang et al, 2019;Huang et al, 2020). Zhong et al (2020a) argue that high semantically similarity with the source document is highly desirable for a good summary.…”

Section: Evaluation Metricsmentioning

confidence: 99%

“…Further, recent debate and consequent surge in study of evaluation metrics for automatic summaries is a clear and strong testimony to the considerable complexity of the task (Peyrard, 2019a,b;Ermakova et al, 2019;Bhandari et al, 2020a;Vasilyev & Bohannon, 2020;Fabbri et al, 2020;Huang et al, 2020;Bhandari et al, 2020b).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Investigating Entropy for Extractive Document Summarization

Khurana,

Bhatnagar

2021

Preprint

View full text Add to dashboard Cite

Automatic text summarization aims to cut down readers' time and cognitive effort by reducing the content of a text document without compromising on its essence. Ergo, informativeness is the prime attribute of document summary generated by an algorithm, and selecting sentences that capture the essence of a document is the primary goal of extractive document summarization.In this paper, we employ Shannon's entropy to capture informativeness of sentences. We employ Non-negative Matrix Factorization (NMF) to reveal probability distributions for computing entropy of terms, topics, and sentences in latent space. We present an information theoretic interpretation of the computed entropy, which is the bedrock of the proposed E-Summ algorithm, an unsupervised method for extractive document summarization. The algorithm systematically applies information theoretic principle for selecting informative sentences from important topics in the document. The proposed algorithm is generic and fast, and hence amenable to use for summarization of documents in real time. Furthermore, it is domain-, collection-independent and agnostic to the language of the document. Benefiting from strictly positive NMF factor matrices, E-Summ algorithm is transparent and explainable too.We use standard ROUGE toolkit for performance evaluation of the proposed method on four well known public data-sets. We also perform quantitative as- *

show abstract

Section: Introductionmentioning

confidence: 91%

Section: Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Investigating Entropy for Extractive Document Summarization

Khurana,

Bhatnagar

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There has been much work analyzing Seq2Seq models which is always task-specific and based on automatic or human evaluation. For example, Huang et al (2020) analyze the common models' performance on summarization.…”

Section: Related Workmentioning

confidence: 99%

“…We want to know the pros and cons of different Seq2Seq models on this task, and the factors influencing the generation performance. Particularly, following Multidimensional Quality Metric(MQM) (Mariana, 2014), similar to the job on summarization evaluation (Huang et al, 2020), we use 8 metrics on the Accuracy and Fluency aspects to count errors, respectively. Therefore, compared with existing manual evaluation reports, it is more informative and objective.…”

Section: Introductionmentioning

confidence: 99%

How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?

Xunjian¹,

Wan²

2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

With the rapid development of deep learning, Seq2Seq paradigm has become prevalent for end-to-end data-to-text generation, and the BLEU scores have been increasing in recent years. However, it is widely recognized that there is still a gap between the quality of the texts generated by models and the texts written by human. In order to better understand the ability of Seq2Seq models, evaluate their performance and analyze the results, we choose to use Multidimensional Quality Metric(MQM) to evaluate several representative Seq2Seq models on end-to-end data-to-text generation. We annotate the outputs of five models on four datasets with eight error types and find that 1) copy mechanism is helpful for the improvement in Omission and Inaccuracy Extrinsic errors but it increases other types of errors such as Addition; 2) pre-training techniques are highly effective, and pre-training strategy and model size are very significant; 3) the structure of the dataset also influences the model's performance greatly; 4) some specific types of errors are generally challenging for seq2seq models.

show abstract

A Survey of Recent Error Annotation Schemes for Automatically Generated Text

Huidrom¹,

Belz²

2022

Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)

View full text Add to dashboard Cite

While automatically computing numerical scores remains the dominant paradigm in NLP system evaluation, error annotation and analysis is receiving increasing attention, with several error annotation schemes recently proposed for automatically generated text. However, there is little agreement about what error annotation schemes should look like, how many different types of errors should be distinguished and at what level of granularity. In this paper, our aim is to map out work on annotating errors in human and machine generated text, with a particular focus on error taxonomies. We describe our paper selection process, and survey the error annotation schemes reported in the papers, drawing out similarities and differences between them. Finally, we characterise the issues that would make it difficult to move from the current situation to a standardised error taxonomy for annotating errors in automatically generated text.

show abstract

What Have We Achieved on Text Summarization?

Cited by 3 publications

References 35 publications

Investigating Entropy for Extractive Document Summarization

Investigating Entropy for Extractive Document Summarization

How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?

A Survey of Recent Error Annotation Schemes for Automatically Generated Text

Contact Info

Product

Resources

About