Improving Hadoop performance in intercloud environments

Kim, Shin-gyu; Won, Junghee; Han, Hyuck; Eom, Hyeonsang; Yeom, Heon Y.

doi:10.1145/2160803.2160873

Cited by 22 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To reduce the overall job's average makespan, the single four steps must be differently coordinated. Some have proposed modified versions of Hadoop capable of enhancing only a single step [23,24]. In [7], the dynamics of the steps are analyzed, and a comprehensive end-to-end optimization of the job's execution flow is proposed.…”

Section: Geo-hadoop Approachmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

show abstract

Section: Geo-hadoop Approachmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

show abstract

“…In the aim of reducing the job's average makespan, phases and the relative timing must be adequately coordinated. Some researchers have proposed enhanced version of Hadoop capable of optimizing only a single phase (Kim et al, 2011;Mattess et al, 2013). Heintz et al(Heintz et al, 2014) analyze the dynamics of the phases and address the need of making a comprehensive, end-to-end optimization of the job's execution flow.…”

Section: Related Workmentioning

confidence: 99%

A Hadoop based Framework to Process Geo-distributed Big Data

Cavallo

Cusmà

Modica

et al. 2016

Proceedings of the 6th International Conference on Cloud Computing and Services Science

View full text Add to dashboard Cite

In many application fields such as social networks, e-commerce and content delivery networks there is a constant production of big amounts of data in geographically distributed sites that need to be timely elaborated. Distributed computing frameworks such as Hadoop (based on the MapReduce paradigm) have been used to process big data by exploiting the computing power of many cluster nodes interconnected through high speed links. Unfortunately, Hadoop was proved to perform very poorly in the just mentioned scenario. We designed and developed a Hadoop framework that is capable of scheduling and distributing hadoop tasks among geographically distant sites in a way that optimizes the overall job performance. We propose a hierarchical approach where a top-level entity, by exploiting the information concerning the data location, is capable of producing a smart schedule of low-level, independent MapReduce sub-jobs. A software prototype of the framework was developed. Tests run on the prototype showed that the job scheduler makes good forecasts of the expected job's execution time.

show abstract

“…Similarly, there are several approaches focused on enhancing the task scheduler of the Hadoop framework to reduce power consumption or network cost . Kim et al defined an intercloud as a federated environment of public cloud and private clusters and proposed a task scheduler to enhance performance. These approaches are different from the proposed method in that all of them require the scheduler modification of the Hadoop framework at source code level.…”

Section: Related Workmentioning

confidence: 99%

Enhancing network I/o performance for a virtualized Hadoop cluster

Jeong

Choi

2016

Concurrency and Computation

View full text Add to dashboard Cite

Summary A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware‐based single root I/O virtualization network architecture.

show abstract

Improving Hadoop performance in intercloud environments

Cited by 22 publications

References 2 publications

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

A Hadoop based Framework to Process Geo-distributed Big Data

Enhancing network I/o performance for a virtualized Hadoop cluster

Contact Info

Product

Resources

About