MapReduce scheduling algorithms in Hadoop: a systematic study

Hedayati, Soudabeh; Maleki, Neda; Olsson, Tobias; Ahlgren, Fredrik; Seyednezhad, Mahdi; Berahmand, Kamal

doi:10.1186/s13677-023-00520-9

Cited by 7 publications

(3 citation statements)

References 88 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concurrently, cloud computing has emerged to provide seamless access to extensive computing resources, networking, and storage capabilities, ensuring that applications can effectively handle large datasets. Owing to its wide range of applications and benefits, the MapReduce framework has been applied across various domains [ 64 , 65 ].…”

Section: Methodsmentioning

confidence: 99%

Streamline Intelligent Crowd Monitoring with IoT Cloud Computing Middleware

Gazis,

Katsiri

2024

Sensors

View full text Add to dashboard Cite

This article introduces a novel middleware that utilizes cost-effective, low-power computing devices like Raspberry Pi to analyze data from wireless sensor networks (WSNs). It is designed for indoor settings like historical buildings and museums, tracking visitors and identifying points of interest. It serves as an evacuation aid by monitoring occupancy and gauging the popularity of specific areas, subjects, or art exhibitions. The middleware employs a basic form of the MapReduce algorithm to gather WSN data and distribute it across available computer nodes. Data collected by RFID sensors on visitor badges is stored on mini-computers placed in exhibition rooms and then transmitted to a remote database after a preset time frame. Utilizing MapReduce for data analysis and a leader election algorithm for fault tolerance, this middleware showcases its viability through metrics, demonstrating applications like swift prototyping and accurate validation of findings. Despite using simpler hardware, its performance matches resource-intensive methods involving audiovisual and AI techniques. This design’s innovation lies in its fault-tolerant, distributed setup using budget-friendly, low-power devices rather than resource-heavy hardware or methods. Successfully tested at a historical building in Greece (M. Hatzidakis’ residence), it is tailored for indoor spaces. This paper compares its algorithmic application layer with other implementations, highlighting its technical strengths and advantages. Particularly relevant in the wake of the COVID-19 pandemic and general monitoring middleware for indoor locations, this middleware holds promise in tracking visitor counts and overall building occupancy.

show abstract

Section: Methodsmentioning

confidence: 99%

Streamline Intelligent Crowd Monitoring with IoT Cloud Computing Middleware

Gazis,

Katsiri

2024

Sensors

View full text Add to dashboard Cite

show abstract

“…While this examination delves into potential avenues for improving Hadoop's performance and resource allocation strategies in the event of unforeseen failures, it also underscores the need for additional exploration into the scalability of these strategies and their potential influence on the overall system performance. A study has been conducted on data placement policies in Hadoop, presenting a modified approach to enhance system performance, especially in environments characterized by heterogeneity [22]. However, striking a harmonious balance between optimizing data placement and accommodating real-time data access patterns may present its own set of challenges.…”

Section: Related Workmentioning

confidence: 99%

Enhancing Hadoop distributed storage efficiency using multi-agent systems

Mahdaoui,

Sais,

Abouchabaka

et al. 2024

IJEECS

View full text Add to dashboard Cite

<p>Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.</p>

show abstract

“…MapReduce [5,6] is a distributed parallel computing technology. The cores of Hadoop are the MapReduce computing framework and the distributed file system HDFS [7].…”

Section: Introductionmentioning

confidence: 99%

DAG-Based Formal Modeling of Spark Applications with MSVL

Fan,

Wang

2023

Information

View full text Add to dashboard Cite

Apache Spark is a high-speed computing engine for processing massive data. With its widespread adoption, there is a growing need to analyze its correctness and temporal properties. However, there is scarce research focused on the verification of temporal properties in Spark programs. To address this gap, we employ the code-level runtime verification tool UMC4M based on the Modeling, Simulation, and Verification Language (MSVL). To this end, a Spark program S has to be translated into an MSVL program M, and the negation of the property P specified by a Propositional Projection Temporal Logic (PPTL) formula that needs to be verified is also translated to an MSVL program M1; then, a new MSVL program “M and M1” can be compiled and executed. Whether program S violates the property P is determined by the existence of an acceptable execution of “M and M1”. Thus, the key issue lies in how to formalize model Spark programs using MSVL programs. We previously proposed a solution to this problem—using the MSVL functions to perform Resilient Distributed Datasets (RDD) operations and converting the Spark program into an MSVL program based on the Directed Acyclic Graph (DAG) of the Spark program. However, we only proposed this idea. Building upon this foundation, we implement the conversion from RDD operations to MSVL functions and propose, as well as implement, the rules for translating Spark programs to MSVL programs based on DAG. We confirm the feasibility of this approach and provide a viable method for verifying the temporal properties of Spark programs. Additionally, an automatic translation tool, S2M, is developed. Finally, a case study is presented to demonstrate this conversion process.

show abstract

MapReduce scheduling algorithms in Hadoop: a systematic study

Cited by 7 publications

References 88 publications

Streamline Intelligent Crowd Monitoring with IoT Cloud Computing Middleware

Streamline Intelligent Crowd Monitoring with IoT Cloud Computing Middleware

Enhancing Hadoop distributed storage efficiency using multi-agent systems

DAG-Based Formal Modeling of Spark Applications with MSVL

Contact Info

Product

Resources

About