How to run experiments with large peer-to-peer data structures

Abstract

Distributed Hash Tables (DHT) promise to administer huge sets of (key, value)-pairs under high workloads. DHT currently are a hot topic of research in various disciplines of computer science. Experimental results that are convincing require evaluations with large DHT (i.e., more than 100,000 nodes). However, many studies confine themselves to (less convincing) experimental examinations with much fewer nodes. Information on how to run experiments with DHT with many nodes is not available. Based on experience gained with a DHT implementation of our own [4], this article describes how to carry out such experiments successfully. The infrastructure used is a cluster of 32 commodity workstations. The article starts by compiling requirements regarding such experiments. We then identify the various bottlenecks that may be the result of a naive implementation, and we describe their negative effects. The article proposes various countermeasures, e.g., an experiment clock, and a component that maintains persistent network connections between cluster nodes. The features proposed are beneficial: A naive experimental setup allows for 10,000 peers maximum and a total of 20 operations per second, a sophisticated one following our proposal for 1,000,000 peers and 150 operations per second. Furthermore, we say why experimental results gained in such a way are meaningful in many situations.

On the Usage of Global Document Occurrences in Peer-to-Peer Information Systems

“…An interesting observation was that the GDO-based query execution results in less top-100 documents, compared to the GDO-free counterpart; thus, the best approach for retrieving top-100 documents appears to be when employing the GDO-based scores only for query routing. We expect that this behaviour is due to some very frequent low-ranked top-100 documents 4 . These documents get replaced from some less popular documents, slightly less relevant, which do not belong in the top-100 documents.…”

“…Recent research on P2P systems, such as Chord [1], CAN [2], Pastry [3], P2P-Net [4], or P-Grid [5] is based on various forms of distributed hash tables (DHTs) and supports mappings from keys, e.g., titles or authors, to locations in a decentralized manner such that routing scales well with the number of peers in the system. Typically, in a network of n nodes, an exact-match key lookup can be routed to the proper peer(s) in at most O(log n) hops, and no peer needs to maintain more than O(log n) routing information.…”

HyCube: A distributed hash table based on a variable metric

“…The created library and simulator were designed based on several guidelines presented in [3] -processing events for individual nodes was performed by only a certain number of threads on each simulator instance, which allowed simulations of very large systems (even much larger than 100'000 nodes) to be preformed on a limited number of physical machines.…”

“…However, too large values of λ could cause an increase in path lengths. Figure 2 presents the static resilience of HyCube consisting of 10'000 nodes, using the Euclidean metric, for several values of the λ parameter 3 . For better readability, the failed paths figure was scaled and cropped -the most significant differences were observed below 70 % of failed nodes.…”

The MINERVA Project: Towards Collaborative Search in Digital Libraries Using Peer-to-Peer Technology

“…Recent research on P2P systems, such as Chord [27], CAN [24], Pastry [26], P2P-Net [5], or P-Grid [1], is based on various forms of distributed hash tables (DHTs) and supports mappings from keys, e.g., titles or authors, to locations in a decentralized manner such that routing scales well with the number of peers in the system.…”