A Scheduling Strategy to Run Hadoop Jobs on Geodistributed Data

Cavallo, Marco; Cusmà, Lorenzo; Modica, Giuseppe; Polito, Carmelo; Tomarchio, Orazio

doi:10.1007/978-3-319-33313-7_1

Cited by 5 publications

(2 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The exact value of β app for a submitted application may not be known a priori. The work in (Cavallo et al, 2015) discusses how to get a good estimate for the β app .…”

Section: Scheduling Strategymentioning

confidence: 99%

“…The algorithm used to generate potential execution paths and to select the best one is described in Listing 1. It is based on the Integer Partitioning theory (Andrews, 1976); for a deeper description of the algorithm steps the reader may refer to (Cavallo et al, 2015 Let us now explain how to compute the execution time of a specific execution path in a reference scenario. Figure 4 depicts a scenario of four sites (S 0 through S 3 ) and a geographic network interconnecting the sites.…”

Section: Scheduling Strategymentioning

confidence: 99%

See 1 more Smart Citation

A Hadoop based Framework to Process Geo-distributed Big Data

Cavallo

Cusmà

Modica

et al. 2016

Proceedings of the 6th International Conference on Cloud Computing and Services Science

Self Cite

View full text Add to dashboard Cite

In many application fields such as social networks, e-commerce and content delivery networks there is a constant production of big amounts of data in geographically distributed sites that need to be timely elaborated. Distributed computing frameworks such as Hadoop (based on the MapReduce paradigm) have been used to process big data by exploiting the computing power of many cluster nodes interconnected through high speed links. Unfortunately, Hadoop was proved to perform very poorly in the just mentioned scenario. We designed and developed a Hadoop framework that is capable of scheduling and distributing hadoop tasks among geographically distant sites in a way that optimizes the overall job performance. We propose a hierarchical approach where a top-level entity, by exploiting the information concerning the data location, is capable of producing a smart schedule of low-level, independent MapReduce sub-jobs. A software prototype of the framework was developed. Tests run on the prototype showed that the job scheduler makes good forecasts of the expected job's execution time.

show abstract

“…The exact value of β app for a submitted application may not be known a priori. The work in (Cavallo et al, 2015) discusses how to get a good estimate for the β app .…”

Section: Scheduling Strategymentioning

confidence: 99%

Section: Scheduling Strategymentioning

confidence: 99%