Topic-focused multi-document summarization using an approximate oracle score

Conroy, John M.; Schlesinger, Judith D.; O’Leary, Dianne P.

doi:10.3115/1273073.1273093

Cited by 69 publications

(51 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Summarization systems that directly optimize for more topic signatures during content selection have fared very well in evaluations (Conroy et al, 2006). Hence the number of topic signatures from the input present in a summary might be a good indicator of summary content quality.…”

Section: Use Of Topic Words In the Summarymentioning

confidence: 99%

Automatically evaluating content selection in summarization without human models

Louis

Nenkova

2009

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09

View full text Add to dashboard Cite

We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen-Shannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations. Automatically Evaluating Content Selection in Summarization without Human ModelsAnnie Louis University of Pennsylvania lannie@seas.upenn.edu Ani NenkovaUniversity of Pennsylvania nenkova@seas.upenn.edu AbstractWe present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, JensenShannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.

show abstract

Section: Use Of Topic Words In the Summarymentioning

confidence: 99%

Automatically evaluating content selection in summarization without human models

Louis

Nenkova

2009

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09

View full text Add to dashboard Cite

show abstract

“…The scored sentences (a ranked list) are passed to an antiredundancy component for summary sentence selection. TsSum [2] relies on the computation of topic words [10], which are words that occur more often in the input text than in a large background corpus. The log-likelihood ratio test is applied, with a threshold parameter used to determine topic words from non-topic words.…”

Section: Reproducible Experimental Setupmentioning

confidence: 99%

Experiments in Newswire Summarisation

Mackie

McCreadie

Macdonald

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. In this paper, we investigate extractive multi-document summarisation algorithms over newswire corpora. Examining recent findings, baseline algorithms, and state-of-the-art systems is pertinent given the current research interest in event tracking and summarisation. We first reproduce previous findings from the literature, validating that automatic summarisation evaluation is a useful proxy for manual evaluation, and validating that several state-of-the-art systems with similar automatic evaluation scores create different summaries from one another. Following this verification of previous findings, we then reimplement various baseline and state-of-the-art summarisation algorithms, and make several observations from our experiments. Our findings include: an optimised Lead baseline; indication that several standard baselines may be weak; evidence that the standard baselines can be improved; results showing that the most effective improved baselines are not statistically significantly less effective than the current state-of-the-art systems; and finally, observations that manually optimising the choice of anti-redundancy components, per topic, can lead to improvements in summarisation effectiveness.

show abstract

“…Number of topic signature words (Lin and Hovy, 2000;Conroy et al, 2006) and percentage of signature words in the vocabulary. Document similarity in the input set These features apply to multi-document summarization only.…”

Section: Log-likelihood Ratio For Words In the Inputmentioning

confidence: 99%

Performance confidence estimation for automatic summarization

Louis¹,

Nenkova²

2009

Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on - EACL '09

View full text Add to dashboard Cite

We address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single-or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for single-and 74% for multi-document summarization. AbstractWe address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single-or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine different formulations of the classification task which considerably influence performance. The best results are 84% prediction accuracy for single-and 74% for multi-document summarization.

show abstract

Topic-focused multi-document summarization using an approximate oracle score

Cited by 69 publications

References 7 publications

Automatically evaluating content selection in summarization without human models

Automatically evaluating content selection in summarization without human models

Experiments in Newswire Summarisation

Performance confidence estimation for automatic summarization

Contact Info

Product

Resources

About