A data structure perspective to the RDD-based Apriori algorithm on Spark

Abstract

During the recent years, a number of efficient and scalable frequent itemset mining algorithms for big data analytics have been proposed by many researchers. Initially, MapReduce-based frequent itemset mining algorithms on Hadoop cluster were proposed. Although, Hadoop has been developed as a cluster computing system for handling and processing big data, but the performance of Hadoop does not meet the expectation for the iterative algorithms of data mining, due to its high I/O, and writing and then reading intermediate results in the disk. Consequently, Spark has been developed as another cluster computing infrastructure which is much faster than Hadoop due to its in-memory computation. It is highly suitable for iterative algorithms and supports batch, interactive, iterative, and stream processing of data. Many frequent itemset mining algorithms have been re-designed on the Spark, and most of them are Aprioribased. All these Spark-based Apriori algorithms use Hash Tree as the underlying data structure. This paper investigates the efficiency of various data structures for the Spark-based Apriori. Although, the data structure perspective has been investigated previously, but for MapReduce-based Apriori, and it must be re-investigated in the distributed computing environment of Spark. The considered underlying data structures are Hash Tree, Trie, and Hash Table Trie. The experimental results on the benchmark datasets show that the performance of Spark-based Apriori with Trie and Hash Table Trie are almost similar but both perform many times better than Hash Tree in the distributed computing environment of Spark.

Right-Hand Side Expanding Algorithm for Maximal Frequent Itemset Mining

“…Faced with high-dimensional mass big data, the exact algorithm itself is almost of no practicability due to the temporal complexity and explosion of storage space. However, some calculation platforms that can realize the temporal and spatial decomposition of data mining tasks have emerged in order to process big data, so the exact algorithm becomes feasible [12,[27][28][29]. The advantage of these calculation platforms lies in the fact that a big data analysis becomes feasible due to computer clusters, among which Spark reaches the highest rate at present.…”

Research on Apriori algorithm based on compression processing and hash table

“…Of course, the efficiency of association analysis of Apriori algorithm has been greatly improved after improvement. There is also an iconic association rule algorithm, which is called FP-Growth algorithm [3] . Compared with the original association rule algorithm, FP growth algorithm has greatly shortened the association analysis time of the algorithm.…”

Analysis of frequent itemset generation based on trie data structure in Apriori algorithm

“…From another perspective, it is common to use multi-thread and computing power with multicore architecture [12] in supporting data processing. This raises the suspicion that multi-thread in [13] and [14] can produce better performance also in getting this frequent itemset, as well as a challenge on how to determine the best performance process architecture that is applied to which subprocesses as threads and how big is the increase for the best multi-thread architecture in a single server environment, where an experiment in multi-node server environment was proposed by [15], [16].…”