A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries

Abstract

ABSTRACTere is growing interest in systems that generate timeline summaries by ltering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task depend on standardized and reproducible evaluation methodologies for comparing systems. However, timeline summary evaluation is still in its infancy, with competing methodologies currently being explored in international evaluation forums such as TREC. One area of active exploration is how to explicitly represent the units of information that should appear in a "good" summary. Currently, there are two main approaches, one based on identifying nuggets in an external "ground truth", and the other based on clustering system outputs. In this paper, by building test collections that have both nugget and cluster annotations, we are able to compare these two approaches. Speci cally, we address questions related to evaluation e ort, di erences in the nal evaluation products, and correlations between scores and rankings generated by both approaches. We summarize advantages and disadvantages of nuggets and clusters to o er recommendations for future system evaluations.

Exploitation of Social Media for Emergency Relief and Preparedness: Recent Research and Trends

“…Evaluation of summarization algorithms is traditionally done using ROUGE scores (based on unigram/bigram overlap with gold standard summaries), but these measures are not sufficient for timeline summarization methods. Nugget-based or cluster-based evaluation methods have recently been shown to be more effective, but they require lot of annotation effort (Baruah et al 2017).…”

On enhancing the robustness of timeline summarization test collections

“…of systems. As a result, it is unclear to what extent the test collections produced during these tracks can be used to evaluate the quality of new systems that were not pooled for judging [5].…”

“…Second as information clusters within the TREC Real-time Summarization track during 2016 and 2017. We choose to use the TREC Temporal Summarization implementation as the basis for the study in this paper as it is the more complex/costly to deploy of the two (due to the more fine-grained definition of atomic information units used) and because it enables a more detailed comparison of systems [5]. We discuss this implementaton below.…”

Explicit Diversification of Event Aspects for Temporal Summarization

“…More precisely, events [1][2][3][4][5][6][7][8][9][10] 18 are assigned to a 'TTG-2013' label set, events 11-25 to a 'TTG-2014' label set and events 26-46 to a 'TTG-2015' label set. We use these label sets later to provide an approximate comparison of the performance of our proposed approaches to the TREC best participating systems for each year.…”

“…In particular, the labeling methodologies used to create the nuggets and matches, the interfaces and support tools used to do the matching, as well as the assessor profiles differ between the TREC-TS original assessments ('TREC-TS-201X' label sets) and the label sets derived from 'TTG-All' ('TTG-201X' label sets). For those interested in examining the differences between these methodologies in more detail, we recommend reading the study by Baruah et al [4].…”