Internet-scale support for map-reduce processing

Abstract

Volunteer Computing systems (VC) harness computing resources of machines from around the world to perform distributed independent tasks. Existing infrastructures follow a master/worker model, with a centralized architecture. This limits the scalability of the solution due to its dependence on the server. Our goal is to create a fault-tolerant VC platform that supports complex applications, by using a distributed model which improves performance and reduces the burden on the server. In this paper we present VMR, a VC system able to run MapReduce applications on top of volunteer resources, spread throughout the Internet. VMR leverages users' bandwidth through the use of inter-client communication, and uses a lightweight task validation mechanism. We describe VMR's architecture and evaluate its performance by executing several MapReduce applications on a wide area testbed. Our results show that VMR successfully runs MapReduce tasks over the Internet. When compared to an unmodified VC system, VMR obtains a performance increase of over 60% in application turnaround time, while reducing server bandwidth use by two orders of magnitude and showing no discernible overhead.

Volunteer Computing on Mobile Devices

“…Most of the existing volunteer computing systems, as well as the architectures described in Section IV, consider a centralized system, with communication going through a single server or coordinator that fulfills the role of job scheduler and handles the task distribution and result validation. As a result, these systems either do not implement any result aggregation and validation mechanisms or create a considerable overhead on the server and are thus limited to embarrassingly parallel applications [64]. Recent solutions tolerate clients' failures by assigning the same task to multiple devices and by replicating the intermediate and final output of the computations across different devices [64], [65].…”

Scadamar

“…Nevertheless, as MapReduce's popularity increased, some platforms decided to use available resources over the Internet to run MapReduce jobs. Therefore, solutions such as SCOLARS [5], MOON [11], Tang [15] and Marozzo [7] already support MapReduce applications.…”

“…With respect to the few solutions [11,15,7,5] that support MapReduce, we are able to point out some issues (more details on Section 4): data distribution could be improved, intermediate data availability is overlooked, and lack of support for multiple cycle MapReduce applications.…”

Availability and Network-Aware MapReduce Task Scheduling over the Internet

“…Another similar work is VMR [5], a volunteer computing system able to run MapReduce applications on top of volunteer resources, spread throughout the Internet. VMR leverages users' bandwidth through the use of inter-client communication, and uses a lightweight task validation mechanism.…”

“…MapReduce is an emerging programming model for large-scale data processing [6]. Recently, there are some MapReduce implementations that are designed for large-scale parallel data processing specialized on desktop grid or volunteer resources in Intranet or Internet, such as MOON [11], P2P-MapReduce [13], VMR [5], HybridMR [18], etc. In our previous work, we also implemented a MapReduce system called BitDew-MapReduce, specifically for desktop grid environment [19].…”