BitDew: A programmable environment for large-scale data management and distribution

Abstract

Abstract:Desktop Grids use the computing, network and storage resources from idle desktop PC's distributed over multiple-LAN's or the Internet to compute a large variety of resource-demanding distributed applications. While these applications need to access, compute, store and circulate large volumes of data, little attention has been paid to data management in such large-scale, dynamic, heterogeneous, volatile and highly distributed Grids. In most cases, data management relies on ad-hoc solutions, and providing general approach is still a challenging issue.To address this problem, we propose the BitDew framework, a programmable environment for automatic and transparent data management on computational Desktop Grids. This paper describes the BitDew programming interface, its architecture, and the performance evaluation of its runtime components. BitDew relies on a specific set of meta-data to drive key data management operations, namely life cycle, distribution, placement, replication and fault-tolerance with a high level of abstraction. The Bitdew runtime environment is a flexible distributed service architecture that integrates modular P2P components such as DHT's for a distributed data catalog and collaborative transport protocols for data distribution. Through several examples, we describe how application programmers and Bitdew users can exploit Bitdew's features. The performance evaluation demonstrates that the high level of abstraction and transparency is obtained with a reasonable overhead, while offering the benefit of scalability, performance and fault tolerance with little programming cost.

BIGhybrid -- A Toolkit for Simulating MapReduce in Hybrid Infrastructures

“…BitDew [7] is a middleware that exploits protocols like P2P, http, BitTorrent and ftp, and selects the best protocol depending on data size. BitDew architecture is decentralized and has independent services for Data Scheduler, Data Catalog, Data Repository and Data Transfer.…”

“…BitDew [7] is a middleware for large scale data management on hybid distributed computing infrastructures. Its runtime environment supports the use of multiple file transfer protocols: either client/server (http, ftp, scp), P2P (bittorrent), Grid (through the SAGA API) or Cloud (Amazon S3, Dropbox).…”

“…It is able to simulate two middleware for two distinct infrastructures: BitDew-MR [7], [8] for Desktop Grid Computing and Hadoop-Blobseer [9] for Cloud computing. BIGhybrid has several desirable features: it is built atop of MRSG, a validated Hadoop simulator [10], and MRA++, a simulator for heterogeneous environments [11]; it has a trace toolkit that enable analyze, monitor and graphically plot the task executions; it is a trace-base simulator able to process real infrastructure availability traces [12]; and its modular design allows for further extension.…”

DSL-Lab: A Low-Power Lightweight Platform to Experiment on Domestic Broadband Internet

“…This approach complements Grid'5000 where the experimental conditions are controlled and reproducible but far from real life Internet condition. BitDew [8] is a middleware dedicated to data management on Desktop Grid. In addition with traditional microbenchmarks performed on Grid'5000, several real-life scenarios have been run on DSL-Lab.…”

“…The experiments on the DSL-Lab platform have shown that the BitDew middleware was able to cope with complex scenarios in situations close to real-world deployment. We report on two scenarios that were previously published in [8] to illustrate the specificity of DSL-Lab with respect to quantitative performance evaluation.…”