IQN Routing: Integrating Quality and Novelty in P2P Querying and Ranking

Abstract

Abstract. We consider a collaboration of peers autonomously crawling the Web. A pivotal issue when designing a peer-to-peer (P2P) Web search engine in this environment is query routing: selecting a small subset of (a potentially very large number of relevant) peers to contact to satisfy a keyword query. Existing approaches for query routing work well on disjoint data sets. However, naturally, the peers' data collections often highly overlap, as popular documents are highly crawled. Techniques for estimating the cardinality of the overlap between sets, designed for and incorporated into information retrieval engines are very much lacking. In this paper we present a comprehensive evaluation of appropriate overlap estimators, showing how they can be incorporated into an efficient, iterative approach to query routing, coined Integrated Quality Novelty (IQN). We propose to further enhance our approach using histograms, combining overlap estimation with the available score/ranking information. Finally, we conduct a performance evaluation in MINERVA, our prototype P2P Web search engine.

DAW: Duplicate-AWare Federated Query Processing over the Web of Data

“…Overlap estimation among data sources have been used in a number of approaches in the area of distributed and P2P information retrieval [4,10,15,18,23,24]. COSCO [10] gathers statistics about coverage and overlap from past queries and uses them to determine in which order the overlapping collections should be accessed to retrieve the most new results in the least number of collections.…”

“…MIPs have been shown to be the provide a good tradeoff between estimation error and space requirements [15,6]. In addition, MIPs of different lengths can be compared, which can be beneficial for datasets of different sizes.…”

“…In general, if two MIPs have different sizes, we can always use the smaller number of permutations as a common denominator. This incurs in a loss of accuracy in the result MIPs, but still yields to a more flexible setting, where the different collections do not have to agree on a predefined MIPs size [15].…”

Peer-to-peer information retrieval

“…(Cuenca- Acuna et al, 2003;Reynolds and Vahdat, 2003;Suel et al, 2003;Zhang and Suel, 2005;Michel et al, 2005aMichel et al, , 2006 When using a distributed global index, a multiterm query requires multiple lookups in the distributed hash table. The posting lists for all terms need to be intersected to find the documents that contain all query terms.…”

Efficient peer-to-peer semantic overlay networks based on statistical language models

“…Many proposals have been made in the literature, for example: globally available term statistics about the peers' contents [8,16,27,2], epidemic routing using gossiping strategies [18], routing indices with peer summaries from local neighborhoods [10,21], statistical synopses such as Bloom filters or hash sketches maintained in a directory based on distributed hash tables (DHT) [5,28,32], randomized expander graphs with low-diameter guarantees [25,26] and randomized rendezvous [31], clustering of thematically related peers [11,12,23], superpeer-based hierarchical networks [24,22], cost/benefit optimization based on coarse-grained global knowledge [29,30], and many more.…”