A New Merging Numerous Small Files Approach for Hadoop Distributed File System

Ali, Adnan; Mirza, Nada Masood; Ishak, Mohamad Khairi

doi:10.1109/ecti-con54298.2022.9795369

Cited by 2 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such an extensive data set is a significant barrier to traditional data processing methods. Google launched one of the practical frameworks for processing massive data, MapReduce, in 2004 [10], [11]. It is scalable, dependable, and has excellent fault tolerance.…”

Section: Introductionmentioning

confidence: 99%

The scheduling techniques in the Hadoop and Spark of smart cities environment: a systematic review

Mirza,

Ali,

Ishak

2024

Bulletin EEI

View full text Add to dashboard Cite

Processing extensive and diverse data in real-time is a significant challenge in the context of smart cities. Timely access to information and efficient analytics is essential for smart city services to make data-driven decisions and enhance urban living. Scheduling algorithms play a crucial role in ensuring the prompt delivery of services and efficient task completion. This paper explores various scheduling techniques, including static, dynamic, and hybrid schedulers, and compares their objectives and performance. Additionally, the study examines two prominent data processing frameworks, Hadoop and Spark, and compares their capabilities in handling big data in smart cities. With its ability to process large amounts of data quickly and efficiently, Spark has shown superiority over Hadoop in real-time data processing and performance optimization. The paper concludes by highlighting the strengths and limitations of each framework. It discusses the need for further research in optimizing scheduling techniques and exploring hybrid artificial intelligence scheduling for Spark. Overall, the findings contribute to a better understanding of data processing in real-time and provide insights for researchers and practitioners in smart cities.

show abstract

Section: Introductionmentioning

confidence: 99%

The scheduling techniques in the Hadoop and Spark of smart cities environment: a systematic review

Mirza,

Ali,

Ishak

2024

Bulletin EEI

View full text Add to dashboard Cite

show abstract

“…The proposed approach will aid the Hadoop Framework to resolve issues caused by multiple small files not limited to reduced disk read/write times, memory load, function calls, etc. Moreover, this paper also aims to resolve small file management-related issues [8]. This article is in the following structure: Section 2 explains the Hadoops background and issues related to small files.…”

Section: Introductionmentioning

confidence: 99%

Enhanced Best Fit Algorithm for Merging Small Files

Ali¹,

Mirza²,

Ishak³

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

In the Big Data era, numerous sources and environments generate massive amounts of data. This enormous amount of data necessitates specialized advanced tools and procedures that effectively evaluate the information and anticipate decisions for future changes. Hadoop is used to process this kind of data. It is known to handle vast volumes of data more efficiently than tiny amounts, which results in inefficiency in the framework. This study proposes a novel solution to the problem by applying the Enhanced Best Fit Merging algorithm (EBFM) that merges files depending on predefined parameters (type and size). Implementing this algorithm will ensure that the maximum amount of the block size and the generated file size will be in the same range. Its primary goal is to dynamically merge files with the stated criteria based on the file type to guarantee the efficacy and efficiency of the established system. This procedure takes place before the files are available for the Hadoop framework. Additionally, the files generated by the system are named with specific keywords to ensure there is no data loss (file overwrite). The proposed approach guarantees the generation of the fewest possible large files, which reduces the input/output memory burden and corresponds to the Hadoop framework's effectiveness. The findings show that the proposed technique enhances the framework's performance by approximately 64% while comparing all other potential performance-impairing variables. The proposed approach is implementable in any environment that uses the Hadoop framework, not limited to smart cities, real-time data analysis, etc.

show abstract

A New Merging Numerous Small Files Approach for Hadoop Distributed File System

Cited by 2 publications

References 15 publications

The scheduling techniques in the Hadoop and Spark of smart cities environment: a systematic review

The scheduling techniques in the Hadoop and Spark of smart cities environment: a systematic review

Enhanced Best Fit Algorithm for Merging Small Files

Contact Info

Product

Resources

About