Improving ROUGE for Timeline Summarization

Martschat, Sebastian; Markert, Katja

doi:10.18653/v1/e17-2046

Cited by 18 publications

(18 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, it is worth mentioning the work of (Martschat and Markert, 2017), where a variant of ROUGE that allows for evaluation of timeline summarization is presented. This novel metric takes into account both temporal and semantic similarity of daily summaries.…”

Section: Task Based Evaluationmentioning

confidence: 99%

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

Evaluation is crucial in the research and development of automatic summarization applications, in order to determine the appropriateness of a summary based on different criteria, such as the content it contains, and the way it is presented. To perform an adequate evaluation is of great relevance to ensure that automatic summaries can be useful for the context and/or application they are generated for. To this end, researchers must be aware of the evaluation metrics, approaches, and datasets that are available, in order to decide which of them would be the most suitable to use, or to be able to propose new ones, overcoming the possible limitations that existing methods may present. In this article, a critical and historical analysis of evaluation metrics, methods, and datasets for automatic summarization systems is presented, where the strengths and weaknesses of evaluation efforts are discussed and the major challenges to solve are identified. Therefore, a clear up-to-date overview of the evolution and progress of summarization evaluation is provided, giving the reader useful insights into the past, present and latest trends in the automatic evaluation of summaries.

show abstract

Section: Task Based Evaluationmentioning

confidence: 99%

The challenging task of summary evaluation: an overview

Lloret

Plaza

Aker

2017

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…Therefore, researchers developed automated methods to evaluate summaries. Most of these methods are based on the similarity measure between a summary and its original text, but they do not relate the judgment with human judgment [28]. Hence, a recall-oriented method named ROUGE was developed to evaluate the quality of the summary [28].…”

Section: Evaluation Methodsmentioning

confidence: 99%

“…Most of these methods are based on the similarity measure between a summary and its original text, but they do not relate the judgment with human judgment [28]. Hence, a recall-oriented method named ROUGE was developed to evaluate the quality of the summary [28]. Here, the system summary and the reference summary (human-generated) are compared to evaluate the summary quality.…”

Section: Evaluation Methodsmentioning

confidence: 99%

Multi‐layered attentional peephole convolutional LSTM for abstractive text summarization

Rahman

Siddiqui

2020

ETRI Journal

View full text Add to dashboard Cite

ive text summarization is a process of making a summary of a given text by paraphrasing the facts of the text while keeping the meaning intact. The manmade summary generation process is laborious and time-consuming. We present here a summary generation model that is based on multilayered attentional peephole convolutional long short-term memory (MAPCoL; LSTM) in order to extract abstractive summaries of large text in an automated manner. We added the concept of attention in a peephole convolutional LSTM to improve the overall quality of a summary by giving weights to important parts of the source text during training. We evaluated the performance with regard to semantic coherence of our MAPCoL model over a popular dataset named CNN/Daily Mail, and found that MAPCoL outperformed other traditional LSTM-based models. We found improvements in the performance of MAPCoL in different internal settings when compared to state-of-the-art models of abstractive text summarization.

show abstract

“…Automatic evaluation of TLS is done by ROUGE (Lin, 2004). We report ROUGE-1 and ROUGE-2 F 1 scores for the concat, agreement and align+ m:1 metrics for TLS we presented in Martschat and Markert (2017). These metrics perform evaluation by concatenating all daily summaries, evaluating only matching days and evalu-ating aligned dates based on date and content similarity, respectively.…”

Section: Evaluation Metricsmentioning

confidence: 99%

A Temporally Sensitive Submodularity Framework for Timeline Summarization

Martschat¹,

Markert²

2018

Proceedings of the 22nd Conference on Computational Natural Language Learning

Self Cite

View full text Add to dashboard Cite

Timeline summarization (TLS) creates an overview of long-running events via dated daily summaries for the most important dates. TLS differs from standard multi-document summarization (MDS) in the importance of date selection, interdependencies between summaries of different dates and by having very short summaries compared to the number of corpus documents. However, we show that MDS optimization models using submodular functions can be adapted to yield wellperforming TLS models by designing objective functions and constraints that model the temporal dimension inherent in TLS. Importantly, these adaptations retain the elegance and advantages of the original MDS models (clear separation of features and inference, performance guarantees and scalability, little need for supervision) that current TLS-specific models lack. An open-source implementation of the framework and all models described in this paper is available online. 1 16 Predecessors of this task were the update and temporal summarization tasks (Aslam et al., 2015)

show abstract

Improving ROUGE for Timeline Summarization

Cited by 18 publications

References 16 publications

The challenging task of summary evaluation: an overview

The challenging task of summary evaluation: an overview

Multi‐layered attentional peephole convolutional LSTM for abstractive text summarization

A Temporally Sensitive Submodularity Framework for Timeline Summarization

Contact Info

Product

Resources

About