Towards Enhancing the Performance of Parallel FP-Growth on Spark

Abstract

Frequent itemset mining (FIM) is a crucial tool for identifying hidden patterns in information. FP-Growth is an FIM algorithm used to find associations. When the data size increases, the execution of FIM algorithms on a single machine suffers from computational problems, such as memory and time consumption. For these reasons, parallel and distributed processing on platforms such as Spark is essential. The parallel frequent pattern (PFP) is the implementation of FP-Growth in Spark. The main problem with PFP is that it does not consider the load balancing between cluster units. This research proposes an enhanced balanced parallel frequent pattern ''EBPFP'' algorithm to enhance and balance the PFP. The proposed algorithm (EBPFP) proposes two ideas. First, a strategy for load balancing between groups is proposed to ensure that the items are evenly divided between the nodes, and the cluster resources are used more effectively. Second, the improved conditional pattern base (ICPB) method aims to remove infrequent items from the conditional pattern base before constructing local FP-Trees. The experimental results show that the proposed EBPFP algorithm outperforms PFP, and the difference in running time between EBPFP and PFP was 21.56% and 39.72%, respectively.INDEX TERMS Big data, data mining, association rule analysis, frequent pattern growth algorithm, spark, load balancing.

Public exposure to broadband electromagnetic fields and its association with population density and building density: The case study of Beijing

“…An association rule can be expressed as , which denotes that Y is much more likely to occur whenever X occurs. The classic algorithms for association rule mining include Apriori [30] and FP-growth [31] , but they cannot be directly used to analyze the data in this study because the algorithms deal with binary data (0 and 1). Therefore, firstly, the original data of EMF, population density and building density are classified by two-step clustering method; then the classified data are dualized to generate a transaction matrix that can be processed by the association rule mining algorithm.…”

Multi-keyword Searching and Top-K Encrypted Retrieval of Cloud Data

“…Little files for the most part allude to those file sizes, which are under 64 MB. As per a concentrate in 2007 at the National Energy Research Scientific Computing Center, 43% of the north of 13 million files on a common parallel file system are under 64 KB and close to 100% are under 64 MB (Petascale Data Storage Institute (2007)), and more logical applications consist of an enormous number of little files are portrayed in (Carns et al [7], Essam et.al [8]). In any case, notwithstanding monstrous little file datasets, the constructed FP-tree in Parallel FP-Growth (PFP) algorithm can't squeeze into the memory, which frequently creates some issues, for example, memory overflow and enormous communication above.…”