An improved parallel FP-growth algorithm based on Spark and its application

Miao, Yuhang; Lin, Jinxing; Xu, Nuo

doi:10.23919/chicc.2019.8866373

Cited by 8 publications

(4 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it suffers from a preprocessing overhead. Furthermore, [17] proposed a parallel improved FP-growth algorithm called the IFPS. The IFPS aims to compress large-scale datasets using a matrix to reduce traffic between nodes and optimize the performance.…”

Section: Related Workmentioning

confidence: 99%

Towards Enhancing the Performance of Parallel FP-Growth on Spark

2022

View full text Add to dashboard Cite

Frequent itemset mining (FIM) is a crucial tool for identifying hidden patterns in information. FP-Growth is an FIM algorithm used to find associations. When the data size increases, the execution of FIM algorithms on a single machine suffers from computational problems, such as memory and time consumption. For these reasons, parallel and distributed processing on platforms such as Spark is essential. The parallel frequent pattern (PFP) is the implementation of FP-Growth in Spark. The main problem with PFP is that it does not consider the load balancing between cluster units. This research proposes an enhanced balanced parallel frequent pattern ''EBPFP'' algorithm to enhance and balance the PFP. The proposed algorithm (EBPFP) proposes two ideas. First, a strategy for load balancing between groups is proposed to ensure that the items are evenly divided between the nodes, and the cluster resources are used more effectively. Second, the improved conditional pattern base (ICPB) method aims to remove infrequent items from the conditional pattern base before constructing local FP-Trees. The experimental results show that the proposed EBPFP algorithm outperforms PFP, and the difference in running time between EBPFP and PFP was 21.56% and 39.72%, respectively.INDEX TERMS Big data, data mining, association rule analysis, frequent pattern growth algorithm, spark, load balancing.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards Enhancing the Performance of Parallel FP-Growth on Spark

2022

View full text Add to dashboard Cite

show abstract

“…The communication cost causes the problem of reduced efficiency. A parallel FPgrowth algorithm called IFPS [22] based on Spark was improved by matrix technology. In this algorithm, a dataset was compressed into an information matrix to reduce memory consumption.…”

Section: B Distributed Algorithms For Discovery Of Frequent Patternsmentioning

confidence: 99%

A Distributed Method for Fast Mining Frequent Patterns From Big Data

et al. 2021

View full text Add to dashboard Cite

In recent years, knowledge discovery in databases provides a powerful capability to discover meaningful and useful information. For numerous real-life applications, frequent pattern mining and association rule mining have been extensively studied. In traditional mining algorithms, data are centralized and memory-resident. As a result of the large amount of data, bandwidth limitation, and energy limitations when applying these methods to distributed databases, especially in this era of big data, the performance is not effective enough. Hence, data mining on distributed environments has emerged as an important research area. To improve the performance, we propose a set of algorithms based on FP growth that discover FPs that are capable of providing fast and scalable service in distributed computing environments and a brief data structure to store items and counts to minimize the data for transmission on the network. To ensure completeness and execution capability, DistEclat and BigFIM were considered for the experiment comparison. Experiments show that the proposed method has superior cost-effectiveness for processing massive datasets and good capabilities under various experiment conditions. The proposed method on average required only 33% of the execution time and 45% of the transmission cost of DistEclat. Compared to BigFIM, The proposed method on average required 23.3% of the execution time and 14.2% of the transmission cost of BigFIM.

show abstract

“…According to (Chen et al, 2021;Miao et al, 2019;Sunarti et al, 2021) Data Mining is also called KDD (knowledge discovery in database). KDD is an activity that covers data collection, the use of data to find relationships or patterns in large data sets.…”

Section: Introductionmentioning

confidence: 99%

Application of the FP-Growth Method to Determine Drug Sales Patterns

Anwar¹,

Ambiyar

Fadhilah

2023

SinkrOn

View full text Add to dashboard Cite

Pharmacies are shops that sell and mix medicines based on doctors’ prescriptions and trade medical goods. Apart from being a business actor, the pharmacy also plays a role in providing health services that are easily accessible to the public. The problem that often occurs in pharmacies selling drugs is that they are less than optimal in service to consumers. The habit of consumers buying more than one type of drug makes pharmacy staff slow in providing the drug due to the inaccurate layout of the drug. The FP-Growth method in Data Mining is a method that can provide a solution in determining drug sales patterns at pharmacies. The FP-Growth method is a method used in determining the data set that occurs most frequently together (Frequent Itemset). The research objective was to determine drug sales patterns based on drug sales transaction data so that drug layouts could be determined. This research was conducted at the Pharmacy at the Pratama Sehati Husada Clinic in Medan with drug sales transaction data from November 2021 to December 2021. The results of applying the FP-Growth method with a value of ≥ 15% as a minimum support and a value of ≥ 15% as a minimum confidence, a pattern of drug sales in pairs. The application of Data Mining with the FP-Growth method has been implemented at the Pharmacy at the Pratama Sehati Husada Clinic with the desired goals with the final result in the form of a report on the results of determining drug sales patterns.

show abstract

An improved parallel FP-growth algorithm based on Spark and its application

Cited by 8 publications

References 12 publications

Towards Enhancing the Performance of Parallel FP-Growth on Spark

Towards Enhancing the Performance of Parallel FP-Growth on Spark

A Distributed Method for Fast Mining Frequent Patterns From Big Data

Application of the FP-Growth Method to Determine Drug Sales Patterns

Contact Info

Product

Resources

About