Energy-Efficient Task Scheduling for CPU-Intensive Streaming Jobs on Hadoop

Jin, Peiquan; Hao, Xingjun; Wang, Xiaoliang; Yue, Lihua

doi:10.1109/tpds.2018.2881176

Cited by 23 publications

(7 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(1) After the node's memory processing of the map, the intermediate information among the locals is empty. After the sink ends, place the input and map's location (position, then Map 1, where Map 1 is the key pair value) area checkpoint file [28] .…”

Section: Improved Rescheduling Algorithmmentioning

confidence: 99%

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Kumar,

Varshney,

Bhatiya

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

We live in an age where everything around us is being created. Data generation rates are so scary, creating pressure to implement costly and straightforward data storage and recovery processes. MapReduce model functionality is used for creating a cluster parallel, distributed algorithm, and large datasets. The MapReduce strategy from Hadoop helps develop a community of non-commercial use to offer a new algorithm for resolving such problems for commercial applications as expected from this working algorithm with insights as a result of disproportionate or discriminatory Hadoop cluster results. Expected results are obtained in the work and the exam conducted under this job; many of them are scheduled to set schedules, match matrices' data positions, clustering before determining to click, and accurate mapping and internal reliability to be closed together to avoid running and execution times. Mapper output and proponents have been implemented, and the map has been used to reduce the function. The execution input key/value pair and output key/value pair have been set. This paper focuses on evaluating this technique for the efficient retrieval of large volumes of data. The technique allows for capabilities to inform a massive database of information, from storage and indexing techniques to the distribution of queries, scalability, and performance in heterogeneous environments. The results show that the proposed work reduces the data processing time by 30%.

show abstract

Section: Improved Rescheduling Algorithmmentioning

confidence: 99%

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Kumar,

Varshney,

Bhatiya

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

show abstract

“…Besides the above categories, a number of research works focus on energy-efficient workflow scheduling on specific applications. For example, as the streaming jobs on Hadoop can be formulated as a DAG, two types of energy-efficient workflow scheduling heuristics are proposed for the energy efficiency extension on YARN (yet another resource negotiator) [31]. To address the energy consumption of interconnection networks, a heuristic list-based network energy-efficient workflow scheduling (NEEWS) algorithm is proposed to investigate the efficiency of the computing nodes, as well as the interconnection networks in the HCS [32].…”

Section: Energy-efficient Workflow Schedulingmentioning

confidence: 99%

Bi-Objective Workflow Scheduling on Heterogeneous Computing Systems Using a Memetic Algorithm

Zhang

Tong

et al. 2021

Electronics

View full text Add to dashboard Cite

Due to the high power bills and the negative environmental impacts, workflow scheduling with energy consciousness has been an emerging need for modern heterogeneous computing systems. A number of approaches have been developed to find suboptimal schedules through heuristics by means of slack reclamation or trade-off functions. In this article, a memetic algorithm for energy-efficient workflow scheduling is proposed for a quality-guaranteed solution with high runtime efficiency. The basic idea is to retain the advantages of population-based, heuristic-based, and local search methods while avoiding their drawbacks. Specifically, the proposed algorithm incorporates an improved non-dominated sorting genetic algorithm (NSGA-II) to explore potential task priorities and allocates tasks to processors by an earliest finish time (EFT)-based heuristic to provide a time-efficient candidate. Then, a local search method integrated with a pruning technique is launched with a low possibility, to exploit the feasible region indicated by the candidate schedule. Experimental results on workflows from both randomly-generated and real-world applications suggest that the proposed algorithm achieves bi-objective optimization, improving makespan, and energy saving by 4.9% and 24.3%, respectively. Meanwhile, it has a low time complexity compared to the similar work HECS.

show abstract

“…The task completion time is estimated to minimize the task deadline to improve resource utilization. Jin et al 15 divided stream tasks into two types: batch stream tasks and online stream tasks and then designs different task scheduling algorithms for each stream task to improve execution efficiency. Gandomi et al 16 proposed a hybrid scheduling algorithm by using dynamic priority and positioning ID technology to reduce task completion time.…”

Section: Related Workmentioning

confidence: 99%

SMOSA: Spider monkey optimization‐based scheduling algorithm for heterogeneous Hadoop

Zhang

Guan

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Hadoop is a typical framework for processing big data. Task scheduling algorithms have a significant impact on the processing performance of Hadoop clusters. Existing scheduling algorithms of Hadoop fail to consider the performance differences between nodes in heterogeneous Hadoop clusters, causing problems such as uneven task allocation and low resource utilization. Aiming to solve this problem, we propose a spider monkey optimization-based scheduling algorithm (SMOSA) for heterogeneous Hadoop. First, the cluster heartbeat mechanism is used to obtain information such as memories and CPUs of nodes to comprehensively consider the actual load capacity of each node. Then, the spider monkey optimization algorithm is adopted to find the optimal mapping relationship between tasks and resources by taking the task completion time as the objective function and updating the position of the spider monkey.Finally, we calculate the remaining rate of node hardware resources, and according to the task type, the node with the higher remaining rate of resource will give priority to the task. Data are compressed for I/O type tasks to reduce disk operations and increase the speed of task execution. Experimental results show that, compared with existing scheduling algorithms, the SMOSA can effectively reduce task execution time and can significantly improve scheduling efficiency and task execution speed especially in heterogeneous Hadoop clusters. For different types of tasks, the execution time can be reduced by up to 19%.

show abstract

Energy-Efficient Task Scheduling for CPU-Intensive Streaming Jobs on Hadoop

Cited by 23 publications

References 28 publications

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Bi-Objective Workflow Scheduling on Heterogeneous Computing Systems Using a Memetic Algorithm

SMOSA: Spider monkey optimization‐based scheduling algorithm for heterogeneous Hadoop

Contact Info

Product

Resources

About