Optimizing Data Distribution in Desktop Grid Platforms

Abstract

Current infrastructures for Volunteer Computing follow a centralized architecture for data distribution, creating a potential bottleneck when tasks require large input files or the central server has limited bandwidth. In this paper we propose two new data models for Berkeley Open Infrastructure for Network Computing (BOINC): an approach based on the popular BitTorrent protocol; and a Content Delivery Network approach. While the latter remains on a theoretical level, we developed a prototype that adds BitTorrent functionality for task distribution and conducted medium-scale tests of the environment. Our preliminary results indicate that the BitTorrent client had a negligible influence on the BOINC client's computation time. The BOINC server showed an unexpectedly low bandwidth output when seeding the file, as well as spikes on CPU usage. This paper discusses the tests that were performed, how they were evaluated, as well as some improvements that could be made in future research on both approaches.

Internet-scale support for map-reduce processing

“…Existing projects such as Climateprediction.net and MilkyWay@home have encountered problems when dealing with large files or having the same data shared by many clients [11]. Although some potential solutions have been proposed [12,13], they have not been deployed in the most widely used systems.…”

Towards MapReduce for Desktop Grid Computing

“…In [17] and [18], authors investigate the use of the BitTorrent protocol with the XtremWeb and BOINC Desktop Grid. If the peer-to-peer approach seems efficient, it assumes that volunteers would agree that their desktop PC connects directly to another participant's machine to exchange data.…”

Enabling High Data Throughput in Desktop Grids through Decentralized Data and Metadata Management: The BlobSeer Approach

“…One of them adopts a P2P strategy, where the input data gets spread across the distributed Desktop Grid (on the same physical resources that serve as workers) [4]. A central data server is used as an initial data source, from which data is first distributed at a large scale.…”

“…The workers can then download their input data from each other when needed, using for instance a BitTorrent-like mechanism. An alternative approach [4] proposes to use Content Distribution Networks (CDN) to improve the available download bandwidth by redirecting the requests for input data from the central data server to some appropriate surrogate data server, based on a global scheduling strategy able to take into account criteria such as locality or load balancing. The CDN approach is more costly then the P2P approach (as it relies on a set of data servers), however it is potentially more reliable (as the surrogate data servers are supposed to be stable enough).…”

“…Second, the whole metadata is managed by the central scheduler, which is put under high I/O pressure by all clients that concurrently read the metadata and may potentially become an I/O bottleneck. Whereas several recent approaches have proposed to decentralize data management in Desktop Grids through P2P or CDN-based strategies [4], to the best of our knowledge, none has explored the idea of decentralizing metadata. We specifically take this approach and propose an architecture based on the BlobSeer prototype, which implements an efficient lock-free, distributed metadata management scheme, in order to provide support for performant concurrent write accesses to data.…”