Performance Improvement of PrePost Algorithm Based on Hadoop for Big Data

Rochd, Yassir; Hafidi, Imad

doi:10.22266/ijies2018.1031.21

Cited by 6 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because the named node has a single failure problem, when the named node fails, the backup named node can assume the job of the named node [17].…”

Section: Hadoop Big Data Processingmentioning

confidence: 99%

Performance optimization of computing task scheduling based on the Hadoop big data platform

Hei

2022

Neural Comput & Applic

View full text Add to dashboard Cite

Hadoop, a distributed computing framework that can efficiently process large-scale datasets, has been used by an increasing number of organizations as the basic computing framework to build cloud computing platforms. Improving its execution efficiency is a hot research direction in the industry, and the scheduling problem is a key factor affecting the execution efficiency of Hadoop. It is very important to identify its shortcomings and improve them. This paper examines and analyses the optimization of computing task scheduling performance based on the Hadoop big data platform. This paper first analyses Hadoop big data processing. Hadoop has high scalability. Computing nodes can be added at any time, and they can participate in cluster work through simple configuration. The paper discusses the improvement in the Hadoop resource scheduling algorithm. The task scheduling algorithm in the Hadoop-based data task localization proposed in this paper is compared with the default algorithm used in the Hadoop task scheduling algorithm. The former shows better local data in all four jobs, there are more data localization tasks, and the expected goal is achieved. The effectiveness of the algorithm is verified, and the performance is improved by 30%.

show abstract

“…Because the named node has a single failure problem, when the named node fails, the backup named node can assume the job of the named node [17].…”

Section: Hadoop Big Data Processingmentioning

confidence: 99%

Performance optimization of computing task scheduling based on the Hadoop big data platform

Hei

2022

Neural Comput & Applic

View full text Add to dashboard Cite

show abstract

“…PrePost algorithm (9) based on the concept of N-lists is used for association rule mining which presents a data structure named N-list, for storing the information related to Association rule mining. PrePost scans the database twice to construct a tree which generates the N-list of frequent 1-itemsets.…”

Section: Background Studymentioning

confidence: 99%

An efficient FP-Growth based association rule mining algorithm using Hadoop MapReduce

Senthilkumar¹,

Prasad²

2020

IJST

View full text Add to dashboard Cite

Objectives: To achieve improved performance of FP-Growth based Association Rule Mining algorithm for massive data by effective utilization of storage, execution capability and improved partition technique within the Hadoop MapReduce framework. Methodology: The proposed methodology has four main phases: In the first phase, the item sets for finding the frequent pattern are encoded and thus minimizes the expensive operation for large data set. In the second phase, improved hash partitioning reduces the network overhead and improves the communication speed within the MapReduce phase for each item set. The effective usage of network bandwidth and storage is obtained by the impact of compression in the third phase. The use of combiner in final phase for frequent item set mining minimizes the overhead of reduce phase by finding the pattern in each partition and minimizes the overall execution time of the FP-Growth algorithm. Findings: FP-Growth based association rule mining algorithm is designed for parallel execution on distributed cluster of servers. Changes to the MapReduce implementation of FP-Growth with the impact of encoding. Improved hash partitioning, compression and configuration results in a significant performance gain with better improvement in execution time. Novelty/Improvements: According to the experimental results, the changes in storage and processing level within the MapReduce framework improves the overall performance of the parallel frequent item set mining in Hadoop cluster.

show abstract

“…In 2018, we presented, HPrePostPlus algorithm [26], a better version of PrePost, based on Hadoop, which uses a HashMap to traverse efficiently through the PPC tree and enhance the N-list creation process. The HPrePostPlus algorithm is very powerful and surpasses the state-of-the-art algorithms, such as PrePost [6], MRPrePost [23], PFP [18], and negFIN [27].…”

Section: Related Workmentioning

confidence: 99%

“…In this section, the DisPrePost algorithm has been compared to two advanced algorithms, HPrePostPlus [26] and the well-known HFIM [29]. DisPrePost is the first implementation of the PrePost algorithm in the Spark framework, HPrePostPlus is a recent implementation of the Hadoop-based PrePost parallel algorithm [26] with good results, and HFIM is a typical implementation of the Sparkbased Apriori parallel algorithm [29] with good performance. We evaluated speed performance by analyzing runtime and scalability.…”

Section: Performance Evaluationmentioning

confidence: 99%

An Efficient Distributed Frequent Itemset Mining Algorithm Based on Spark for Big Data

Rochd¹,

Hafidi²

2019

IJIES

Self Cite

View full text Add to dashboard Cite

Frequent item exploration is a fundamental element in many data mining problems aimed at finding interesting models in the data. Recently, the PrePost algorithm, a new algorithm for extraction frequent element sets based on the idea of N-lists, which in most cases surpasses other current state-of-the-art algorithms, has been introduced. The PrePost algorithm's performance deteriorates when it comes to handling big data. Nevertheless, the current existing PrePost algorithms in place implemented with the MapReduce model are not sufficiently powerful for iterative computation. To reduce IO overhead and take advantage of cluster memory, this article offers an enhanced version of PrePost, the Distributed PrePost (DisPrePost), a parallel algorithm built on the Spark framework, which incorporates the concept of resilient distributed datasets and performs in-memory processing to optimize the execution time of operation, that also utilises a HashMap to further refine the N-list creation process. Experience has shown that the DisPrePost algorithm is more efficient and scalable than the two advanced state-of-the-art methods HPrePostPlus and the well-known algorithm HFIM.

show abstract

Performance Improvement of PrePost Algorithm Based on Hadoop for Big Data

Cited by 6 publications

References 18 publications

Performance optimization of computing task scheduling based on the Hadoop big data platform

Performance optimization of computing task scheduling based on the Hadoop big data platform

An efficient FP-Growth based association rule mining algorithm using Hadoop MapReduce

An Efficient Distributed Frequent Itemset Mining Algorithm Based on Spark for Big Data

Contact Info

Product

Resources

About