A classification framework for straggler mitigation and management in a heterogeneous Hadoop cluster: A state-of-art survey

Bawankule, Kamalakant Laxman; Dewang, Rupesh Kumar; Singh, Anil Kumar

doi:10.1016/j.jksuci.2022.02.021

Cited by 8 publications

(3 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are several surveys [3][4][5][6][7] of existing job schedulers that discuss their features, advantages, and limitations. They have classified job schedulers based on different aspects: strategy (static/dynamic), environment (homogenous/heterogeneous), time (deadline/ delay), etc.…”

Section: Related Workmentioning

confidence: 99%

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

et al. 2022

View full text Add to dashboard Cite

Scheduling of MapReduce jobs is an integral part of Hadoop and effective job scheduling has a direct impact on Hadoop performance. Data locality is one of the most important factors to be considered in order to improve efficiency, as it affects data transmission through the system. A number of researchers have suggested approaches for improving data locality, but few have considered cache locality. In this paper, we present a state-of-the-art job scheduler, CLQLMRS (Cache Locality with Q-Learning in MapReduce Scheduler) for improving both data locality and cache locality using reinforcement learning. The proposed algorithm is evaluated by various experiments in a heterogeneous environment. Experimental results show significantly decreased execution time compared with FIFO, Delay, and the Adaptive Cache Local scheduler.

show abstract

Section: Related Workmentioning

confidence: 99%

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The software as shown in Figure 4, composed of the following layers: an execution environment, a malfunction detection engine, an interface, and a user interface. The execution environment layer runs on a Java runtime environment (JRE) and consists of JRE code (that facilitates development in the Java language), the Hadoop platform (for the distributed processing of big data), Spark (for big data processing and real-time data streaming), and HBase (that facilitates non-stop data-saving for massive volumes of distributed data) [19][20][21][22][23][24]. The malfunction detection engine layer is composed of an ontology-based predictor (for deducing malfunctions using the RNN-based prediction model and information collected from smart livestock farms [25]), an RNN-based prediction model (for training and prediction using input data [26]), and a statistical computation module (that detects malfunctions using the basic statistics of the training data and the predicted values provided by each model).…”

Section: Software Architecturementioning

confidence: 99%

“…distributed processing of big data), Spark (for big data processing and real-time data streaming), and HBase (that facilitates non-stop data-saving for massive volumes of distributed data) [19][20][21][22][23][24]. The malfunction detection engine layer is composed of an ontology-based predictor (for deducing malfunctions using the RNN-based prediction model and information collected from smart livestock farms [25]), an RNN-based prediction model (for training and prediction using input data [26]), and a statistical computation module (that detects malfunctions using the basic statistics of the training data and the predicted values provided by each model).…”

Section: Software Architecturementioning

confidence: 99%

Design and Implementation of a Malfunction Detection System for Livestock Ventilation Devices in Smart Poultry Farms

Kim

Lee

2022

Agriculture

View full text Add to dashboard Cite

Smart livestock farming aims to improve the productivity of livestock through the provision of optimal housing, and it is developed using various sensors and actuators. Ventilation systems play a crucial role in smart livestock farming, including disease prevention and the processing of pollutants (ammonia and hydrogen sulfide) that are severely detrimental to livestock growth. Malfunctions in animal housing ventilation systems lead to mass mortality events. To address such issues, this study reports the design and implementation for a smart detection system for malfunctions in the ventilation devices installed in animal housing. This system is based on recurrent neural networks (RNNs) and implements the ontology method, considering sensor and controller data as the standard. A semantic sensor network ontology founded on a knowledge base was used to detect malfunctions, and stimulus-sensor-observation patterns were used to determine a sensor network within the smart barn. System activation and RNN model tests were used to test the malfunction detection system, and the error between actual data and predicted values was found to be 0.06889. These findings provide insight into the development of autonomous detection systems for device malfunctions and are essential for the development of smart livestock farming technologies.

show abstract