Evaluating Web Search Result Summaries

Liang, Shao Fen; Devlin, Siobhan; Tait, John

doi:10.1007/11735106_10

Cited by 7 publications

(2 citation statements)

References 8 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, the features that are important for the web summary are not likely to be included in FOG and other models. More recently a feature-based approach has been used to model answer quality [13,2,19]. Our work, however, focuses on the issue of modeling readability itself and using it for realtime monitoring of readability of web result summaries.…”

Section: Related Workmentioning

confidence: 99%

Predicting the readability of short web summaries

Kanungo

Orr

2009

Proceedings of the Second ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Readability is a crucial presentation attribute that web summarization algorithms consider while generating a querybaised web summary. Readability quality also forms an important component in real-time monitoring of commercial search-engine results since readability of web summaries impacts clickthrough behavior, as shown in recent studies, and thus impacts user satisfaction and advertising revenue.The standard approach to computing the readability is to first collect a corpus of random queries and their corresponding search result summaries, and then each summary is then judged by a human for its readabilty quality. An average readability score is then reported. This process is time consuming and expensive. Besides, the manual evaluation process can not be used in the real-time summary generation process. In this paper we propose a machine learning approach to the problem. We use the corpus as described above and extract summary features that we think may characterize readability. We then estimate a model (gradient boosted decision tree) that predicts human judgments given the features. This model can then be used in real time to estimate the readability of new (unseen) web search summaries and also be used in the summary generation process.We present results on approximately 5000 editorial judgments collected over the course of a year and show examples where the model predicts the quality well and where it disagrees with human judgments. We compare the results of the model to previous models of readability, most notably Collins-Thompson-Callan, Fog and Flesch-Kincaid, and see that our model shows substantially better correlation with editorial judgments as measured by Pearson's correlation coefficient. The learning algorithm also provides us with the relative importance of the features used.

show abstract

Section: Related Workmentioning

confidence: 99%

Predicting the readability of short web summaries

Kanungo

Orr

2009

Proceedings of the Second ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…As the previous research showed that query length is generally short, processing the QTO algorithm for online summarisation is not complex and can generate a set of weighting terms from the input query terms to enhance weighting effectiveness. Although our proposed Query Term Order algorithm proved effective for producing search result summaries with English web documents (Liang, 2006), we wished to triangulate the study to establish the algorithm's effectiveness using different sets of data. Document Understand Conference (DUC, 2004) data was used for this experiment.…”

Section: Query Term Order Examination With Ducmentioning

confidence: 99%

Investigating sentence weighting components for automatic summarisation

Liang

Devlin

Tait

2007

Information Processing & Management

Self Cite

View full text Add to dashboard Cite

The work described here initially formed part of a triangulation exercise to establish the effectiveness of the Query Term Order algorithm. The methodology produced subsequently proved to be a reliable indicator of quality for summarising English web documents. We utilised the human summaries from the Document Understanding Conference data, and generated queries automatically for testing the QTO algorithm. Six sentence weighting schemes that made use of Query Term Frequency and QTO were constructed to produce system summaries, and this paper explains the process of combining and balancing the weighting components. We also examined the five automatically generated query terms in their different permutations to check if the automatic generation of query terms resulting bias. The summaries produced were evaluated by the ROUGE-1 metric, and the results showed that using QTO in a weighting combination resulted in the best performance. We also found that using a combination of more weighting components always produced improved performance compared to any single weighting component.

show abstract