Summarization system evaluation revisited

Abstract

This article presents a novel automatic method (AutoSummENG) for the evaluation of summarization systems, based on comparing the character n-gram graphs representation of the extracted summaries and a number of model summaries. The presented approach is language neutral, due to its statistical nature, and appears to hold a level of evaluation performance that matches and even exceeds other contemporary evaluation methods. Within this study, we measure the effectiveness of different representation methods, namely, word and character n-gram graph and histogram, different n-gram neighborhood indication methods as well as different comparison methods between the supplied representations. A theory for the a priori determination of the methods' parameters along with supporting experiments concludes the study to provide a complete alternative to existing methods concerning the automatic summary system evaluation process.

The challenging task of summary evaluation: an overview

“…Another statistical method is the AutoSummENG proposed by Giannakopoulos et al (2008). This method is based on n-grams graphs, and takes into account within-a-window co-occurrence of either word n-grams or character n-grams.…”

A BFS-Based Similar Conference Retrieval Framework

“…For example, given a sentence "I like the information retrieval course", when = 2(also known as bigrams), then the n-grams can be generated as: "I like", "like the", "the information", "information retrieval", "retrieval course". In this task, we use N-Gram distance [1] to compute the similarity between two conferences ( ( , )). Nevertheless, the details about this technique would not covered in this report.…”

Detecting Human Features in Summaries – Symbol Sequence Statistical Regularity

“…Then, recall and precision related measures can be used to evaluate the given text. These methods however, together with they Pyramid method [20] and the AutoSummENG family of methods [8,7], are mostly meant to evaluate content and not grammaticality.…”