A data locality based scheduler to enhance MapReduce performance in heterogeneous environments

Naik, Nenavath Srinivas; Negi, Atul; Bapu, B. R. Tapas; Anitha, R.

doi:10.1016/j.future.2018.07.043

Cited by 48 publications

(15 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, data locality is only one aspect that can be taken into account when scheduling work. Other aspects such as load distribution and heterogeneity of the available resources on different nodes need to be balanced together with data locality to perform the work effectively [14]. There exist studies proposing advanced scheduling strategies to balance the reduction of data transfer with load distribution (e.g., [6,19]).…”

Section: Data Localitymentioning

confidence: 99%

Locality-Aware Workflow Orchestration for Big Data

Corodescu

Nikolov

Khan

et al. 2021

Proceedings of the 13th International Conference on Management of Digital EcoSystems

View full text Add to dashboard Cite

Section: Data Localitymentioning

confidence: 99%

Locality-Aware Workflow Orchestration for Big Data

Corodescu

Nikolov

Khan

et al. 2021

Proceedings of the 13th International Conference on Management of Digital EcoSystems

View full text Add to dashboard Cite

“…Naik et al [21] developed a novel scheduling scheme for assigning data blocks based on node processing power. The proposed scheme schedules and manages map/reduce tasks across the heterogeneous cluster's nodes according to their computational capabilities.…”

Section: Locality-aware Schedulementioning

confidence: 99%

Factors affecting cloud data-center efficiency: a scheduling algorithm-based analysis

Shehloo¹,

Butt²,

Zaman³

2021

IJATEE

View full text Add to dashboard Cite

Nowadays, users are required to cache, scrutinise, and process massive datasets from various fields, including science, business, and research. As a result, they require data-intensive platforms with ample storage and processing power. In addition, many of these kinds of platforms must-have features like parallel processing, fault tolerance, data dissemination, scalability, availability, and load balancing. Google developed the MapReduce programming paradigm to counter this problem, which served as the foundation for Apache's open-source Hadoop project. *Author for correspondence Hadoop relies upon a particular file system designated as HDFS, analogous to Google's File-System (GFS). It splits the massive data into equally sized segments and then places them across multiple nodes in a Hadoop cluster [1]. As a result, Hadoop is now widely accepted as a data analytics model [2]. Hadoop's fundamental operating principle is that "moving computation to data is less expensive than moving data to computation." As a result, Hadoop tries to schedule tasks on local data nodes to minimise network traffic [3]. Task scheduling is critical in Hadoop because it significantly impacts the framework's computation time and, thus, its overall performance [4]. However, given the dynamic nature of the cloud environment, proposing an effective task scheduling strategy is a constant challenge. Nevertheless, only a few studies have analyzed the proposed techniques and their overall effect on the Hadoop framework's

show abstract

“…Most of the efforts in scheduling are handling various priorities and the time estimation which is based on runtime running jobs. In [31], a data scheduling locality-based algorithm was proposed. It allocates the input data blocks to the proper nodes based on their processing capacity in order to enhance the performance of MapReduce in heterogeneous Hadoop clusters.…”

Section: B Hadoop Schedulersmentioning

confidence: 99%

LSTPD: Least Slack Time-Based Preemptive Deadline Constraint Scheduler for Hadoop Clusters

et al. 2020

View full text Add to dashboard Cite

Big data refers to numerous forms of complex and large datasets which need distinctive computational platforms in order to be analyzed. Hadoop is one of the popular frameworks for analytics of big data. In Hadoop, a big job is split into multiple small tasks and then they are distributed to worker nodes in a parallel way using MapReduce to speed up computational processes. In this aspect, it is important how to improve throughput performance. MapReduce jobs require quick responses from the worker nodes to complete them under their deadlines. The existing scheduling schemes for Hadoop such as FIFO, fair, and capacity schedulers cannot guarantee the quick response requirement satisfying a prior deadline. Thus, Hadoop system needs to improve response time and completion time for the heterogeneous MapReduce jobs. In this paper, we propose an efficient preemptive deadline constraint scheduler based on least slack time and data locality. In order for better allocation of tasks and load balancing, we first analyze the task scheduling behaviors of the Hadoop platform. Based on that, we propose a novel preemptive approach which considers the remaining execution time of the job being executed in deciding preemption. The experimental results show that the proposed scheme significantly reduces the job execution time and queue waiting time, compared to existing schemes.

show abstract

A data locality based scheduler to enhance MapReduce performance in heterogeneous environments

Cited by 48 publications

References 21 publications

Locality-Aware Workflow Orchestration for Big Data

Locality-Aware Workflow Orchestration for Big Data

Factors affecting cloud data-center efficiency: a scheduling algorithm-based analysis

LSTPD: Least Slack Time-Based Preemptive Deadline Constraint Scheduler for Hadoop Clusters

Contact Info

Product

Resources

About