Fast Recovery MapReduce (FAR-MR) to accelerate failure recovery in big data applications

Zhu, Yongqing; Samsudin, Juniarto; Kanagavelu, Renuga; Zhang, Weiwen; Wang, Long; Aye, Theint Theint; Goh, Rick Siow Mong

doi:10.1007/s11227-018-2716-8

Cited by 9 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Practical feedback confirms the feasibility of the method. Literature [22] introduces an optimized metadata recovery scheme that can retrieve tables faster than the contemporary scheme, which is corroborated by the simulation results, which confirms that the efficiency of the method is superior to other schemes. Literature [23] designed a method to recover missing or anomalous magnitude data (i.e., active power, reactive power, positive sequence current, and voltage magnitude) from PMU measurements based on historical PMU data obtained from both ends of the line.…”

Section: Introductionsupporting

confidence: 57%

Generative Adversarial Network-based Data Recovery Method for Power Systems

Yang,

Ji,

et al. 2024

Applied Mathematics and Nonlinear Sciences

View full text Add to dashboard Cite

Facing the problem of power system data loss, this paper proposes a power system data recovery method based on a generative adversarial network. The power system clustering method utilizes aggregated hierarchical clustering and takes into consideration the similarity between different power system data. To transform the power system data recovery problem into a data generation problem, an improved GAN network data analysis method is proposed that utilizes LSTM as a generator and discriminator. Through experimental tests, the LSTM-GAN method is tested with the LSTM method, interpolation method and low-rank method to compare its effect on lost data recovery under different signals of power system data static and dynamic and four fault scenarios. The results show that the root-mean-square errors of the LSTM-GAN method for recovering data under static-dynamic fluctuations are less than 1.2%, and the difference between the errors under 55% and 15% missing data conditions is only 0.77%, with the highest data recovery error of 2.32% in the power system fault scenarios. Therefore, the GAN-based power system data recovery method can effectively realize the recovery of lost data.

show abstract

Section: Introductionsupporting

confidence: 57%

Generative Adversarial Network-based Data Recovery Method for Power Systems

Yang,

Ji,

et al. 2024

Applied Mathematics and Nonlinear Sciences

View full text Add to dashboard Cite

show abstract

“…The conventional Hadoop MapReduce fault tolerance technique leads to significant performance disadvantages for processing tasks during failure recovery. For the betterment of this area, Fast Recovery MapReduce (FAR-MR), a fault recovery technique, is suggested to ensure effective recovery [61]. To facilitate rapid recovery from task failure and node failure, this incorporates a novel fault tolerance method that mixes decentralized checkpointing and proactive push mechanisms.…”

Section: Other Improved Algorithmsmentioning

confidence: 99%

An Overview of Hadoop Job Scheduling Algorithms for Big Data

Zameel

Zengi̇n

2022

Mugla Journal of Science and Technology

View full text Add to dashboard Cite

Rapid advancements in Big data systems have occurred over the last several decades. The significant element for attaining high performance is "Job Scheduling" in Big data systems which requires more utmost attention to resolve some challenges of scheduling. To obtain higher performance when processing the big data, proper scheduling is required. Apache Hadoop is most commonly used to manage immense data volumes in an efficient way and also proficient in handling the issues associated with job scheduling. To improve performance of big data systems, we significantly analyzed various Hadoop job scheduling algorithms. To get an overall idea about the scheduling algorithm, this paper presents a rigorous background. This paper made an overview on the fundamental architecture of Hadoop Big data framework, job scheduling and its issues, then reviewed and compared the most important and fundamental Hadoop job scheduling algorithms. In addition, this paper includes a review of other improved algorithms. The primary objective is to present an overview of various scheduling algorithms to improve performance when analyzing big data. This study will also provide appropriate direction in terms of job scheduling algorithm to the researcher according to which characteristics are most significant

show abstract

“…In MapReduce, the output of MT tasks is stored locally on the same node, which will be inaccessible when the node encounters a fail-stop. Therefore, research works by [ 23 , 24 , 25 ] proposed checkpointing algorithms to efficiently transfer the output to external storage to avoid faulty task re-execution from scratch. Although the proposed checkpointing algorithms improved the recovery process when the failed tasks are rescheduled on a different node, the re-computation still occurs because the stored checkpoints are not accessible by active nodes in the cluster.…”

Section: Related Workmentioning

confidence: 99%

“…Although the proposed checkpointing algorithms improved the recovery process when the failed tasks are rescheduled on a different node, the re-computation still occurs because the stored checkpoints are not accessible by active nodes in the cluster. Zhu et al [ 24 ] designed a novel fault-tolerance strategy that uses a combination of distributed checkpointing and a proactive push mechanism for low latency recovery. When a failure happens, the recovered task continues computing based on the last checkpoint without the necessity to re-compute the entire data block.…”

Section: Related Workmentioning

confidence: 99%

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques

Saadoon

Hamid

Sofian

et al. 2021

Sensors

View full text Add to dashboard Cite

Hadoop MapReduce reactively detects and recovers faults after they occur based on the static heartbeat detection and the re-execution from scratch techniques. However, these techniques lead to excessive response time penalties and inefficient resource consumption during detection and recovery. Existing fault-tolerance solutions intend to mitigate the limitations without considering critical conditions such as fail-slow faults, the impact of faults at various infrastructure levels and the relationship between the detection and recovery stages. This paper analyses the response time under two main conditions: fail-stop and fail-slow, when they manifest with node, service, and the task at runtime. In addition, we focus on the relationship between the time for detecting and recovering faults. The experimental analysis is conducted on a real Hadoop cluster comprising MapReduce, YARN and HDFS frameworks. Our analysis shows that the recovery of a single fault leads to an average of 67.6% response time penalty. Even though the detection and recovery times are well-turned, data locality and resource availability must also be considered to obtain the optimum tolerance time and the lowest penalties.

show abstract

Fast Recovery MapReduce (FAR-MR) to accelerate failure recovery in big data applications

Cited by 9 publications

References 16 publications

Generative Adversarial Network-based Data Recovery Method for Power Systems

Generative Adversarial Network-based Data Recovery Method for Power Systems

An Overview of Hadoop Job Scheduling Algorithms for Big Data

Experimental Analysis in Hadoop MapReduce: A Closer Look at Fault Detection and Recovery Techniques

Contact Info

Product

Resources

About