DRASH: A Data Replication-Aware Scheduler in Geo-Distributed Data Centers

Convolbo, Moïse W.; Chou, Jerry; Lu, Shihyu; Chung, Yeh Ching

doi:10.1109/cloudcom.2016.0056

Cited by 12 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also considered works in the literature not focusing on MapReduce, although focusing on similar scenarios. To optimize the makespan of a job in a geo-distributed system in [32] a data-replication-aware scheduling algorithm is proposed. The approach is based on a global scheduler that maintains a local queue for all submitted jobs.…”

Section: Geo-hadoop Approachmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

show abstract

Section: Geo-hadoop Approachmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

show abstract

“…The close-to-files algorithm [30] schedules tasks with file data replication transfer to the least-loaded data center to balance the loading. Similarly, Convolbo et al, presented a data-replication aware scheduler [31] (DRASH), which partially replicates file data to data centers. Clearly, the advantage of this approach is that it generates less cost in terms of data transfer.…”

Section: ) Task-oriented Scheduling Algorithmsmentioning

confidence: 99%

JHTD: An Efficient Joint Scheduling Framework Based on Hypergraph for Task Placement and Data Transfer Across Geographically Distributed Data Centers

Jing

Dan

2022

IEEE Access

View full text Add to dashboard Cite

As the explosive growth of the data volume, data center is playing a critical role to store and process huge amount of data. Traditional single data center can no longer to adapt into incredibly fastgrowing data. Recently, some researches have extended the tasks such data processing to geographically distributed data centers. However, since the joint consideration of task placement and data transfer, it is complex and difficult to design a proper scheduling approach with the goal of minimizing makespan under the constraint of task dependencies, processing capability and network, etc. Therefore, our work proposes JHT D : an efficient joint scheduling framework based on hypergraph for task placement and data transfer across geographically distributed data centers. Generally, there are two crucial stages in JHT D. Initially, due to the outstanding of hypergraphs in modeling complex problems, we have leveraged a hypergraph-based model to establish the relationship between tasks, data files, and data centers. Thereafter, a hypergraph-based partition method has been developed for task placement within the first stage. In the second stage, a task reallocation scheme has been devised in terms of each task-to-data dependency. Meanwhile, a data dependency aware transferring scheme has been designed to minimize the makespan. Last, the real-world model China-VO project has been used to conduct a variety of simulation experiments. The results have demonstrated that JHT D effectively optimizes the problems of task placement and data transfer across geographically distributed data centers. JHT D has been compared with three other stateof-the-art algorithms. The results have demonstrated that JHT D can reduce the makespan by up to 20.6%. Also, various impacts (data transfer volume and load balancing) have been taken into account to show and discuss the effectiveness of JHT D.INDEX TERMS Big data processing, Geographically distributed data centers, Joint scheduling framework, Hypergraph, Task placement, Data transferring. I. INTRODUCTIONWith the advent of Big Data era, the rate of data generation is dramatically increasing. For example, Internet giants such as Google and Facebook crunch more than 10 PB of data a day [1]. As a result, it is essential to improve the efficiency of data processing in the face the huge amount of data.MapReduce [2] and Spark [3] have been widely adopted to deal with large amounts of data. These frameworks usually process data analytic jobs characterized by data-dependency awareness. These jobs can be divided into a set of dependent tasks. The execution of a task not only requires the outcome of the parent tasks, but also the data. Normally, the data and

show abstract

“…Convolbo, M. W., Chou, J., Lu, S., & Chung, Y. C. [4] proposed a novel information replication mindful booking calculation which can use the current copies to limit the normal fulfillment time of occupation submitted to a geo-disseminated framework. As it has turned into a true approach for cloud applications to imitate their information crosswise over server farms to forestall information misfortune and assurance administration accessibility, there is a need of adjusting the planning strategies to these geoconveyed server farms.…”

Section: Background Workmentioning

confidence: 99%

“…In this module, We are rearranging the cloned parts by the Fs-drops calculation. By utilizing this calculation, cloned parts are rearranged like (1)(2)(3)(4).Once the client mentioned for the data, It will recover the important sections in the successive request. When every one of the parts are gathered, will create a whole data to the client.…”

Section: C) Data Replication and Data Encryptionmentioning

confidence: 99%

Efficient and Privacy for Data Integrity and Data Replication in Cloud Computing

Gokulakrishnan*¹,

Gnanasekar²

2019

IJITEE

View full text Add to dashboard Cite

Distributed computing and capacity administrations enable customers to move their server farm and applications to unified huge server farms and in this manner keep away from the weight of neighborhood information stockpiling and upkeep. In any case, this stances new moves identified with making secure and solid information stockpiling over temperamental specialist organizations. In this examination, we address the issues of guaranteeing the uprightness of information stockpiling and information replication in distributed computing. To begin with, specifically, we consider strategies for decreasing the weight of producing a steady measure of metadata at the customer side. what's more, Second we proposed and set forth the aftereffects of a framework which will execute a capacity technique that uses the Division and Replication procedure for putting away the information. In this framework, the record will be divided and these parts will be duplicated by the replication factor before putting away it upon the cloud. Replication of a prevalent document and putting away its imitation in the areas closer to the customers who are settling on solicitations is the best decision to lessen the execution time. Despite the fact that replication helps in expanding accessibility, the topic of how to choose an ideal replication number and right areas to put the copies are open difficulties. By abusing some great characteristics of the bilinear gathering, we can devise a basic and effective review administration for open confirmation of untrusted and reappropriated capacity, which can be significant for accomplishing broad organization of distributed computing. While numerous earlier examinations on guaranteeing remote information uprightness did not consider the weight of creating check metadata at the customer side, the target of this investigation is to determine this issue. In addition, our plan likewise underpins information elements and open obviousness. Broad security and execution investigation demonstrates that the proposed plan is profoundly effective and provably secure.

show abstract

DRASH: A Data Replication-Aware Scheduler in Geo-Distributed Data Centers

Cited by 12 publications

References 15 publications

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

JHTD: An Efficient Joint Scheduling Framework Based on Hypergraph for Task Placement and Data Transfer Across Geographically Distributed Data Centers

Efficient and Privacy for Data Integrity and Data Replication in Cloud Computing

Contact Info

Product

Resources

About