GEODIS: towards the optimization of data locality-aware job scheduling in geo-distributed data centers

Convolbo, Moïse W.; Chou, Jerry; Hsu, Ching‐Hsien; Chung, Yeh Ching

doi:10.1007/s00607-017-0564-7

Cited by 30 publications

(20 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The cloud platform provides users with a variety of interfaces to access services, such as web portals, web services, and command lines. Users can view the service catalog provided by each layer of the platform according to their own needs and subscribe to management and services on-demand [ 23 , 24 ]. Monitor Management .…”

Section: Cloud Computingmentioning

confidence: 99%

Cloud Computing to Tourism Economic Data Scheduling Algorithm under the Background of Image and Video

Zhang

Feng

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

With the rapid development of image video and tourism economy, tourism economic data are gradually becoming big data. Therefore, how to schedule between data has become a hot topic. This paper first summarizes the research results on image video, cloud computing, tourism economy, and data scheduling algorithms. Secondly, the origin, structure, development, and service types of cloud computing are expounded in detail. And in order to solve the problem of tourism economic data scheduling, this paper regards the completion time and cross-node transmission delay as the constraints of tourism economic data scheduling. The constraint model of data scheduling is established, the fitness function is improved on the basis of an artificial immune algorithm combined with the constraint model, and the directional recombination of excellent antibodies is carried out by using the advantages of gene recombination so as to obtain the optimal solution to the problem more appropriately. When the resource node scale is 100, the response time of EDSA is 107.92 seconds.

show abstract

Section: Cloud Computingmentioning

confidence: 99%

Cloud Computing to Tourism Economic Data Scheduling Algorithm under the Background of Image and Video

Zhang

Feng

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…The performance of this method can be still improved by data grouping based on semantics. Convolbo et al [21] proposed a heuristic scheduling algorithm called GeoDis to optimize the makespan for data intensive jobs in geo distributed clouds. Authors formulated the task placement and data access as a linear programming problem and used heuristics linear problem solver to find optimal task placement schedule.…”

Section: B Task Managementmentioning

confidence: 99%

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

2024

IJIES

View full text Add to dashboard Cite

Huge demand on business intelligence applications over large volume of enterprise data has resulted in rapid adoption of High performance data analytics. Hadoop based high performance computing environment are optimized for large files. Data centric execution with localization of computing proximal to data provided higher performance for large files. But for small files, the performance reduced and overhead increased due to the way Hadoop handles files. Resource allocation and scheduling policies of Hadoop has to be improvised to handle small files. This work proposes an integrated data, task, and resource management technique to speed up processing of small files in Hadoop based high performance computing environment. As part of data management, the data placement is made dynamic to access frequency and inherent data semantics. As part of task management, tasks were grouped based on fine grained data semantics correlation they process. As part of resource management, data blocks or storage nodes are replicated to improve the access latency with minimal cost overheads. With this integrated management, the proposed solution is able to increase the speedup of handling small files by at least 8.7% compared to most recent works on individual management of data, task and resources.

show abstract

“…Foster's primary focus is on reducing access latency and bandwidth consumption. 13,14 The problem of job scheduling and replica selection in distributed data centers is modeled as static linear programming (LP) problem, 39 besides a hybrid online algorithm with linear complexity to optimize makespan. In another study, a cluster-based heuristic algorithm is developed.…”

Section: Combined Job Scheduling and Data Replication Algorithmsmentioning

confidence: 99%

Minimizing data access latency in data grids by neighborhood‐based data replication and job scheduling

Beigrezaei

Haghighat

Mirtaheri

2020

Int J Communication

View full text Add to dashboard Cite

Summary In Data Grid systems, quick data access is a challenging issue due to the high latency. The failure of requests is one of the most common matters in these systems that has an impact on performance and access delay. Job scheduling and data replication are two main techniques in reducing access latency. In this paper, we propose two new neighborhood‐based job scheduling strategies and a novel neighborhood‐based dynamic data replication algorithm (NDDR). The proposed algorithms reduce the access latency by considering a variety of practical parameters for decision making and the access delay by considering the failure probability of a node in job scheduling, replica selection, and replica placement. The proposed neighborhood concept in job scheduling includes all the nodes with low data transmission costs. Therefore, we can select the best computational node and reduce the search time by running a hierarchical and parallel search. NDDR reduces the access latency through selecting the best replica by performing a hierarchical search established based on the access time, storage queue workload, storage speed, and failure probability. NDDR improves the load balancing and data locality by selecting the best replication place considering the workload, temporal locality, geographical locality, and spatial locality. We evaluate our proposed algorithms by using Optorsim Simulator in two scenarios. The simulations confirm that the proposed algorithms improve the results compared with similar existing algorithms by 11%, 15%, 12%, and 10% in terms of mean job time, replication frequency, mean data access latency, and effective network usage, respectively.

show abstract

GEODIS: towards the optimization of data locality-aware job scheduling in geo-distributed data centers

Cited by 30 publications

References 45 publications

Cloud Computing to Tourism Economic Data Scheduling Algorithm under the Background of Image and Video

Cloud Computing to Tourism Economic Data Scheduling Algorithm under the Background of Image and Video

Integrated Data, Task and Resource Management to Speed Up Processing Small Files in Hadoop Cluster

Minimizing data access latency in data grids by neighborhood‐based data replication and job scheduling

Contact Info

Product

Resources

About