Fast replication in content distribution overlays

Abstract

Abstract-We present SPIDER -a system for fast replication or distribution of large content from a single source to multiple sites interconnected over Internet or via a private network. In order to exploit spatial diversity of the underlying network, SPIDER uses an overlay structure composed of dedicated Transit Nodes (TNs). The data transport mechanism in SPIDER leverages this overlay structure to provide a coordinated approach that minimizes the maximum time to replicate to all destination sites (the makespan of content replication). In order to achieve this objective, SPIDER employs two orthogonal components: a) creation of multiple dynamic distribution trees using the transit nodes b) end-to-end reliable data transport with flow control on these trees by chaining point-to-point TCPs. We further present simulations based results to quantify benefits of tree construction algorithms in random topologies. We evaluate the real implementation of the SPIDER in PlanetLab and observe a 2-6 times speed up compared to different existing schemes.

Beyond Music Sharing: An Evaluation of Peer-to-Peer Data Dissemination Techniques in Large Scientific Collaborations

“…To add flexibility, various peer-to-peer data distribution solutions split files into blocks and transfer these blocks independently (e.g., BitTorrent [11], Bullet [12], SPIDER [30] and many other systems). Much like the aforementioned applicationlevel routing, this approach allows applications a greater degree of control over data distribution.…”

A Simulation Study of Data Distribution Strategies for Large-Scale Scientific Data Collaborations

“…When providing data dissemination functionality, overlays often serve as support for extracting dissemination trees used to send data from each source to all destinations. Among the solutions in this category sit: application level multicasting ALM [5], which constructs an overlay tree between participating end-nodes which is then used to disseminate data, and Spider [4], which builds multiple source-rooted trees.…”

Scalable Maintenance for Strong Web Consistency in Dynamic Content Delivery Overlays

“…But the performance of overlay changes with time as nodes dynamically join and leave the system. [19] proposed an overlay CDN architecture where a set of intermediaries act as TNs, which organize themselves into a content delivery overlay and are used to replicate and forward data Our work can be used in both in conventional CDNs and the improved Overlay CDNs. In this paper, it focuses on how to maintain the Web consistency and reduce the network traffic caused by consistency management in content delivery overlay.…”

“…More recently, some other researchers have also advocated using an overlay CDN structure composed of dedicated transit nodes to distribute the large contents [19].…”