Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

Gautam, Jyoti V.; Prajapati, Harshadkumar B.; Dabhi, Vipul K.; Chaudhary, Sanjay

doi:10.1515/cait-2017-0012

Cited by 4 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be seen from formula (6) that there is a large deviation between the value of the measured impedance c Z and b Z at this time, and the error of the measured impedance c Z is large. As shown in formula (7), the P value deviates significantly from zero. Whether the measurement circuit is defective can be confirmed by the size of the P value.…”

Section: Fault Diagnosis Modelmentioning

confidence: 94%

“…Patel et al [6] according to the historical state and real-time state of the equipment, the state quantity is divided into static data and dynamic data, and the state estimation method based on the weight of the equipment is studied. Jyoti et al [7] treats the evaluation index as fuzzy, and the fuzzy comprehensive evaluation method of relay protection is proposed. Moses et al [8] equipment condition monitoring and secondary loop condition monitoring were analyzed, and a secondary equipment condition detection model was established.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Research on Fault Diagnosis Method of Secondary Equipment in Intelligent Substation

Kuang

Luo

et al. 2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

With the continuous development of power grid scale, traditional substations are gradually replaced by smart substations. In view of the high safety risks of the current power system secondary equipment maintenance and fault isolation work. A new smart substation network structure method for fault diagnosis of smart substations is proposed for the relay protection measurement circuit, and the fault status of the secondary equipment of the smart substation is analysed. First, the timeliness and reliability of relay protection are considered, and the intelligent substation network structure is established. Then, the polarity fault characteristics are analyzed, and the generalized transformation ratio is used to establish the fault diagnosis model of the secondary equipment in the smart substation. Finally, the effectiveness of the proposed method is verified by PSCAD simulation.

show abstract

Section: Fault Diagnosis Modelmentioning

confidence: 94%

Section: Introductionmentioning

confidence: 99%

Research on Fault Diagnosis Method of Secondary Equipment in Intelligent Substation

Kuang

Luo

et al. 2022

J. Phys.: Conf. Ser.

View full text Add to dashboard Cite

show abstract

“…However, the scheduler does not consider the user and job heterogeneity. In addition to that, because capacity scheduler uses the concept of queue capacity, queues may have less capacity to take more time to process the job [16].…”

Section: B Hadoop Schedulermentioning

confidence: 99%

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Khoiruddin¹,

Zakaria²,

Alhussian³

2020

Int. J. Adv. Sci. Eng. Inf. Technol.

View full text Add to dashboard Cite

Hadoop is widely adopted as a big data processing application as it can run on commercial hardware at a reasonable time. Hadoop uses asynchronous blocking concurrency using Thread and Future class. Therefore, in some cases such as network link or hardware failure, a running task may block other tasks from running (the task becomes straggler). Hadoop releases are equipped with algorithms to handle straggler tasks problem. However, the algorithms manage Map and Reduce task similarly, while the straggler root cause might be different for both tasks. In this paper, the Asynchronous Non-Blocking (ANB) method is proposed to improve the performance and avoid the blocking of Reduce task in Hadoop. Instead of using the single queue, our approach uses two queues, i.e. task queue and callback queue. When a task is not ready or detected as a straggler, it is removed from the main task queue and temporarily sent to the callback queue. When the task is ready to run, it will be sent back to the main task queue for running. The performance of the algorithm is compared with rTuner, the latest paper found on handling straggler task in Reduce task. From the comparison, it is shown that ANB consistently gives faster time to complete because any unready tasks will be directly put into the callback queue without blocking other tasks. Furthermore, the overhead time in rTuner is high as it needs to check the straggler status and to find the reason for a task to become straggler.

show abstract

“…One criterion could be the minimization of makespan -Cmax for example, where Cmax corresponds to the maximal time necessary for completion all jobs. This is chronologically the earliest developed model and the most often usedsee for example [10,21,18]. This model corresponds to real life production situations [37], where the machines could perform more than one operation (not at the same time) with corresponding different production times, i.e., some or all of the machines are multifunctional.…”

Section: Formulation Of Job Shop Scheduling Problem (Jssp)mentioning

confidence: 99%

Particle Swarm Optimization and Tabu Search Hybrid Algorithm for Flexible Job Shop Scheduling Problem – Analysis of Test Results

Toshev

2019

Cybernetics and Information Technologies

View full text Add to dashboard Cite

The paper presents a hybrid metaheuristic algorithm, including a Particle Swarm Optimization (PSO) procedure and elements of Tabu Search (TS) metaheuristic. The novel algorithm is designed to solve Flexible Job Shop Scheduling Problems (FJSSP). Twelve benchmark test examples from different reference sources are experimentaly tested to demonstrate the performance of the algorithm. The obtained mean error for the deviation from optimality is 0.044%. The obtained test results are compared to the results in the reference sources and to the results by a genetic algorithm. The comparison illustrates the good performance of the proposed algorithm. Investigations on the base of test examples with a larger dimension will be carried out with the aim of further improvement of the algorithm and the quality of the test results.

show abstract

Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

Cited by 4 publications

References 25 publications

Research on Fault Diagnosis Method of Secondary Equipment in Intelligent Substation

Research on Fault Diagnosis Method of Secondary Equipment in Intelligent Substation

Asynchronous Non-Blocking Algorithm to Handle Straggler Reduce Tasks in Hadoop System

Particle Swarm Optimization and Tabu Search Hybrid Algorithm for Flexible Job Shop Scheduling Problem – Analysis of Test Results

Contact Info

Product

Resources

About