What Snippets Say about Pages in Federated Web Search

Abstract

Abstract.What is the likelihood that a Web page is considered relevant to a query, given the relevance assessment of the corresponding snippet? Using a new federated IR test collection that contains search results from over a hundred search engines on the internet, we are able to investigate such research questions from a global perspective. Our test collection covers the main Web search engines like Google, Yahoo!, and Bing, as well as a number of smaller search engines dedicated to multimedia, shopping, etc., and as such reflects a realistic Web environment. Using a large set of relevance assessments, we are able to investigate the connection between snippet quality and page relevance. The dataset is strongly inhomogeneous, and although the assessors' consistency is shown to be satisfying, care is required when comparing resources. To this end, a number of probabilistic quantities, based on snippet and page relevance, are introduced and evaluated.

Distributed deep web search

“…A result (i.e., a filled out form) denotes a set of field-value pairs. On a result level, the agreement of our judgments and those of the participants is 0.33, which is consistent with the "key" agreement reported in (Demeester et al, 2012). Though it might seem low, it is a direct result of the strict comparison: one slightly different field value causes results to disagree completely.…”

Snippet-Based Relevance Predictions for Federated Web Search

“…The FWS collection presented in [4] contains several levels of relevance judgments, but here we will only consider strict binary relevance levels, for which snippets were judged to be definitely relevant 3 , and pages were indicated as highly relevant or better. In [3] it was shown that even for the best resources only about 2/3 of all 'definitely relevant' snippets corresponded with a highly relevant result page.…”

“…In [3] it was shown that even for the best resources only about 2/3 of all 'definitely relevant' snippets corresponded with a highly relevant result page. A classifier trained on snippet labels would hence not be able to overcome this intrinsic gap in relevance when used for the prediction of the page labels.…”

“…A number of the used resources contain very few highly relevant pages [3], due to the nature of the query set, which has been designed for the 2010 TREC Web Track [2]. Therefore, we will only consider the resources from the following verticals: General Web Search, Multimedia, Encyclopedia/Dictionaries, Blogs, Books Libraries, News, and Shopping.…”

“…This gap was filled by the dataset introduced in [4], designed for research on Federated Web Search (FWS) and made available to researchers. It contains a large amount of sampled data from over a hundred diverse online search engines, and relevance judgments of snippets and pages for the results from these resources in response to the 50 reference topics [3].…”