Application profiling in hierarchical Hadoop for geo-distributed computing environments

Cavallo, Marco; Modica, Giuseppe; Polito, Carmelo; Tomarchio, Orazio

doi:10.1109/iscc.2016.7543796

Cited by 9 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The result of the estimate constitutes what we call the job's Application Profile. Details on the elaboration of the Application Profile can be found in our previous work [18], where we studied the Application Profile of a well-known MapReduce applications.…”

Section: Modeling Job's Execution Pathsmentioning

confidence: 99%

“…In its turn, InvertedIndex is as CPU intensive as the other applications but stays in the middle for what concerns interaction I/O. The reader may refer to our former work [18] for a detailed characterization of these applications in terms of β app and Throughput.…”

Section: Experiment: H2f Vs Hadoopmentioning

confidence: 99%

“…In this work, we assume that all Sites are equipped with mass-storage devices ensuring comparable I/O performance. The procedure we designed to make an estimate of the application profile [18] leverages this assumption.…”

mentioning

confidence: 99%

See 2 more Smart Citations

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

Self Cite

View full text Add to dashboard Cite

In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

show abstract

Section: Modeling Job's Execution Pathsmentioning

confidence: 99%

Section: Experiment: H2f Vs Hadoopmentioning

confidence: 99%

See 1 more Smart Citation

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

Self Cite

View full text Add to dashboard Cite

show abstract

“…A simple extension to HMR is proposed in [134], where the authors suggested to consider the amount of data to be moved and the resources required to produce the final output at the global reducer. However, like HMR, this extension does not consider heterogeneous inter-DC bandwidth and available resources at the clusters.…”

Section: Global Controller Layermentioning

confidence: 99%

A Survey on Geographically Distributed Big-Data Processing Using MapReduce

Dolev

Florissi

Gudes

et al. 2019

IEEE Trans. Big Data

View full text Add to dashboard Cite

Abstract-Hadoop and Spark are widely used distributed processing frameworks for large-scale data processing in an efficient and fault-tolerant manner on private or public clouds. These big-data processing systems are extensively used by many industries, e.g., Google, Facebook, and Amazon, for solving a large class of problems, e.g., search, clustering, log analysis, different types of join operations, matrix multiplication, pattern matching, and social network analysis. However, all these popular systems have a major drawback in terms of locally distributed computations, which prevent them in implementing geographically distributed data processing. The increasing amount of geographically distributed massive data is pushing industries and academia to rethink the current big-data processing systems. The novel frameworks, which will be beyond state-of-the-art architectures and technologies involved in the current system, are expected to process geographically distributed data at their locations without moving entire raw datasets to a single location. In this paper, we investigate and discuss challenges and requirements in designing geographically distributed data processing frameworks and protocols. We classify and study batch processing (MapReduce-based systems), stream processing (Spark-based systems), and SQL-style processing geo-distributed frameworks, models, and algorithms with their overhead issues.

show abstract