Towards MapReduce for Desktop Grid Computing

Tang, Bing; Moca, Mircea; Chevalier, Stephane; He, H. H.; Fedak, Gilles

doi:10.1109/3pgcic.2010.33

Cited by 39 publications

(28 citation statements)

References 23 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, Tang et al (2010) propose a system designed to support MapReduce applications, by exploiting the BitDew middleware (Fedak et al, 2008), which is a programmable environment for automatic and transparent data management on desktop grids (a particular case of pervasive grids aiming at the leverage of unused processing cycles and storage space available within the enterprise). Unfortunately, this study also requires the presence of a stable master node that runs BitDew services and coordinates data/task distribution.…”

Section: Science Publicationsmentioning

confidence: 99%

Mapreduce Challenges on Pervasive Grids

Steffenel¹,

Flauzac²,

Charão³

et al. 2014

Journal of Computer Science

View full text Add to dashboard Cite

This study presents the advances on designing and implementing scalable techniques to support the development and execution of MapReduce application in pervasive distributed computing infrastructures, in the context of the PER-MARE project. A pervasive framework for MapReduce applications is very useful in practice, especially in those scientific, enterprises and educational centers which have many unused or underused computing resources, which can be fully exploited to solve relevant problems that demand large computing power, such as scientific computing applications, big data processing, etc. In this study, we propose the study of multiple techniques to support volatility and heterogeneity on MapReduce, by applying two complementary approaches: Improving the Apache Hadoop middleware by including context-awareness and fault-tolerance features; and providing an alternative pervasive grid implementation, fully adapted to dynamic environments. The main design and implementation decisions for both alternatives are described and validated through experiments, demonstrating that our approaches provide high reliability when executing on pervasive environments. The analysis of the experiments also leads to several insights on the requirements and constraints from dynamic and volatile systems, reinforcing the importance of context-aware information and advanced fault-tolerance features to provide efficient and reliable MapReduce services on pervasive grids.

show abstract

Section: Science Publicationsmentioning

confidence: 99%

Mapreduce Challenges on Pervasive Grids

Steffenel¹,

Flauzac²,

Charão³

et al. 2014

Journal of Computer Science

View full text Add to dashboard Cite

show abstract

“…There are several approaches [67,52,22] to execute MapReduce tasks in a wide-area network. However, unlike our approach, they do not consider network bandwidth and just employ the simple scheduling approach that is used in localarea MapReduce systems.…”

Section: Other Mapreduce Job Scheduling Approachesmentioning

confidence: 99%

“…Since resources are widely dispersed, we cannot use the localityaware scheme of the traditional MapReduce system such that a task is assigned to a node in a local rack where the input data exists. There have been several studies [67,52,22] about developing a wide-area MapReduce system with a similar motivation to ours. However, none of them could effectively overcome the challenges above.…”

Section: Introductionmentioning

confidence: 99%

Decentralized Network Bandwidth Prediction and Node Search

Song

2011

2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and PHD Forum

View full text Add to dashboard Cite

As modern computing becomes increasingly data-intensive and distributed, it is becoming crucial to effectively manage and exploit end-to-end network bandwidth information from hosts on wide-area networks. Inspired by the finding that Internet bandwidth can be represented approximately in a tree metric space, we focus on three specific research problems. First, we have designed a decentralized algorithm for network bandwidth prediction. The algorithm embeds the bandwidth information as distance in an edgeweighted tree, without performing full n-to-n measurements. No central and fixed infrastructure is required. Each joining node performs a limited number of sampling measurements. Second, we designed a decentralized algorithm to search for a centroid node that has high-bandwidth connections with a given set of nodes.The algorithm can find a centroid accurately and efficiently using the bandwidth data produced by the prediction algorithm. Last, we have designed another type of decentralized search algorithm to find a cluster of nodes that have high-bandwidth interconnections. While the clustering problem is NP-complete in a general graph, our algorithm runs in polynomial time with the bandwidth data predicted in a tree metric space. We provide proofs that our algorithms for bandwidth prediction and node search have perfect accuracy and high scalability when a network is modeled as a tree metric space. Also, experimental results with real-world data sets validate the high accuracy and scalability of our approaches.

show abstract

“…During a network failure, many nodes leave the Hadoop cluster; this issue has been discussed in different publications [72,98,36,74], although for particular environments.…”

Section: Network Failurementioning

confidence: 99%

“…A later project similar to MOON is presented in [98]. Here, authors try to present a complete runtime environment to execute MapReduce applications on a Desktop Grid.…”

Section: Network Failurementioning

confidence: 99%

Optimizing the reliability and resource efficiency of MapReduce-based systems

Memishi¹

View full text Add to dashboard Cite

Due to the increase of huge data volumes, a new parallel computing paradigm to process big data in an efficient way has arisen. Many of these systems, called dataintensive computing systems, follow the Google MapReduce programming model. The main advantage of these systems is based on the idea of sending the computation where the data resides, trying to provide scalability and efficiency.In failure-free scenarios, these frameworks usually achieve good results. However, these ones are not realistic scenarios. Consequently, these frameworks exhibit some fault tolerance and dependability techniques as built-in features. On the other hand, dependability improvements are known to imply additional resource costs. This is reasonable and providers offering these infrastructures are aware of this. Nevertheless, not all the approaches provide the same tradeoff between fault tolerant capabilities (or more generally, reliability capabilities) and cost.In this thesis, we have addressed the coexistence between reliability and resource efficiency in MapReduce-based systems, looking for methodologies that introduce the minimal cost and guarantee an appropriate level of reliability. In order to achieve this, we have proposed: (i) a formalization of a failure detector abstraction; (ii) an alternative solution to single points of failure of these frameworks, and finally (iii) a novel feedback-based resource allocation system at the container level.Finally, our generic contributions have been instantiated for the Hadoop YARN architecture, which is the state-of-the-art framework in the data-intensive computing systems community nowadays. The thesis demonstrates how all our approaches outperform Hadoop YARN in terms of reliability and resource efficiency. 3 ResumenDebido al gran incremento de datos digitales que ha tenido lugar en los últimos años, ha surgido un nuevo paradigma de computación paralela para el procesamiento eficiente de grandes volúmenes de datos. Muchos de los sistemas basados en este paradigma, también llamados sistemas de computación intensiva de datos, siguen el modelo de programación de Google MapReduce. La principal ventaja de los sistemas MapReduce es que se basan en la idea de enviar la computación donde residen los datos, tratando de proporcionar escalabilidad y eficiencia.En escenarios libres de fallo, estos sistemas generalmente logran buenos resultados. Sin embargo, la mayoría de escenarios donde se utilizan, se caracterizan por la existencia de fallos. Por tanto, estas plataformas suelen incorporar características de tolerancia a fallos y fiabilidad. Por otro lado, es reconocido que las mejoras en confiabilidad vienen asociadas a costes adicionales en recursos. Esto es razonable y los proveedores que ofrecen este tipo de infraestructuras son conscientes de ello. No obstante, no todos los enfoques proporcionan la misma solución de compromiso entre las capacidades de tolerancia a fallo (o de manera general, las capacidades de fiabilidad) y su coste.Esta tesis ha tratado la problemática de la coexist...

show abstract

Towards MapReduce for Desktop Grid Computing

Cited by 39 publications

References 23 publications

Mapreduce Challenges on Pervasive Grids

Mapreduce Challenges on Pervasive Grids

Decentralized Network Bandwidth Prediction and Node Search

Optimizing the reliability and resource efficiency of MapReduce-based systems

Contact Info

Product

Resources

About