On the Feasibility of Peer-to-Peer Web Indexing and Search

Abstract

This paper discusses the feasibility of peer-to-peer full-text keyword search of the Web. Two classes of keyword search techniques are in use or have been proposed: flooding of queries over an overlay network (as in Gnutella), and intersection of index lists stored in a distributed hash table. We present a simple feasibility analysis based on the resource constraints and search workload. Our study suggests that the peer-to-peer network does not have enough capacity to make naive use of either of search techniques attractive for Web search. The paper presents a number of existing and novel optimizations for P2P search based on distributed hash tables, estimates their effects on performance, and concludes that in combination these optimizations would bring the problem to within an order of magnitude of feasibility. The paper suggests a number of compromises that might achieve the last order of magnitude. Comments Postprint version. Published in AbstractThis paper discusses the feasibility of peer-to-peer full-text keyword search of the Web. Two classes of keyword search techniques are in use or have been proposed: flooding of queries over an overlay network (as in Gnutella), and intersection of index lists stored in a distributed hash table. We present a simple feasibility analysis based on the resource constraints and search workload. Our study suggests that the peer-to-peer network does not have enough capacity to make naive use of either of search techniques attractive for Web search. The paper presents a number of existing and novel optimizations for P2P search based on distributed hash tables, estimates their effects on performance, and concludes that in combination these optimizations would bring the problem to within an order of magnitude of feasibility. The paper suggests a number of compromises that might achieve the last order of magnitude.

IQN Routing: Integrating Quality and Novelty in P2P Querying and Ranking

“…A conceptually global but physically distributed directory, which is layered on top of Chord [33], holds compact, aggregated information about the peers' local indexes, to the extent that the individual peers are willing to disclose. Unlike [23], we use the Chord DHT to partition the term space, such that every peer is responsible for the statistics and metadata of a randomized subset of terms within the directory. For failure resilience and availability, the responsibility for a term can be replicated across multiple peers.…”

Arpeggio: Metadata Searching and Content Sharing with Chord

“…It is not clear how to perform searches without sacrificing scalability or query completeness. Indeed, the obvious approaches to distributed full-text document search scale poorly [9].In this paper, however, we consider systems, such as file sharing, that search only over a relatively small amount of metadata associated with each file, but that have to support highly dynamic and unstable network topology, content, …”

Aspects of Adaptivity in P2P Information Retrieval

“…However, more research is needed for evaluating if the cost of distributed processing of relevance feedback queries is within reasonable bounds. [13] suggest that naive use of inverted files in large networks is beyond reasonable communication cost even when proceeding queries with few query terms.…”