A Suite of Testbeds for the Realistic Evaluation of Peer-to-Peer Information Retrieval Systems

Abstract

Abstract. Peer-to-peer (P 2 P) networking continuously gains popularity among computing science researchers. The problem of information retrieval (IR) over P 2 P networks is being addressed by researchers attempting to provide valuable insight as well as solutions for its successful deployment. All published studies have, so far, been evaluated by simulation means, using well-known document collections (usually acquired from TREC). Researchers test their systems using divided collections whose documents have been previously distributed to a number of simulated peers. This practice leads to two problems: First, there is little justification in favour of the document distributions used by relevant studies and second, since different studies use different experimental testbeds, there is no common ground for comparing the solutions proposed. In this work, we contribute a number of different document testbeds for evaluating P 2 P IR systems. Each of these has been deduced from TREC's WT10g collection and corresponds to different potential P 2 P IR application scenarios. We analyse each methodology and testbed with respect to the document distributions achieved as well as to the location of relevant items within each setting. This work marks the beginning of an effort to provide more realistic evaluation environments for P 2 P IR systems as well as to create a common ground for comparisons of existing and future architectures.

Clustering-Based Query Routing in Cooperative Semi-Structured Peer to Peer Networks

“…Experimental Setup: We use three P2PIR testbed categories, DL*, ASIS*, and U*, with 1500, 11680 and 11680 peers respectively, which are derived from the WT10g collection [12]; the WR and WOR variants indicate where there is content replication across peers or not. The standard query set of TREC topics 451-550 2 is used and we report performance measurements averaged across the 100 queries in the set.…”

Evaluating Document Retrieval Methods for Resource Selection in Clustered P2P IR

“…The final merged result set is evaluated against classical IR parameters such as precision and MAP, as well as recall since labelled relevance information is available for the query-testbed combination we use. In addition to popular P2P IR testbeds comprising ≈1.6 million documents listed from [11], we also evaluate our approach on a metasearch task on the FedWeb 2013 dataset [6] (13k documents, 200 queries) consisting of results across 157 search engines, each of which are modeled as a separate peer. Since the result trends across methods were found to be consistent across varying values of pre-specified percentages of peers to be selected at each super-peer, we will report the average of results from 10 settings with the selectivity parameter varying from 5% to 50% in increments of 5%.…”

“…Since the result trends across methods were found to be consistent across varying values of pre-specified percentages of peers to be selected at each super-peer, we will report the average of results from 10 settings with the selectivity parameter varying from 5% to 50% in increments of 5%. Table 3 summarizes the results on the IR testbeds viz., DLWOR, DLWR, ASISWOR and ASISWR (details in [11]); DL* and ASIS* simulate digital library and file-sharing scenarios respectively. We report precision, recall and MAP figures evaluated over the top-1000 documents, for each technique-testbed combination.…”

An Evaluation of a Cluster-Based Architecture for Peer-to-Peer Information Retrieval

“…We performed our evaluation using the testbeds proposed in [6]. These testbeds are based on TREC's WT10g collection and are designed to address a number of P2P IR applications through different document distributions and concentrations of relevant documents.…”

“…In this paper we provide a wide-scale experimental evaluation of a cluster-based P2P IR architecture [2], using a set of testbeds that were devised for this purpose [6]. Through clustering, this architecture attempts to organise the shared content into semantically-related peer-groups.…”