FiDoop: Parallel Mining of Frequent Itemsets Using MapReduce

Abstract

Existing parallel mining algorithms for frequent itemsets lack a mechanism that enables automatic parallelization, load balancing, data distribution, and fault tolerance on large clusters. As a solution to this problem, we design a parallel frequent itemsets mining algorithm called FiDoop using the MapReduce programming model. To achieve compressed storage and avoid building conditional pattern bases, FiDoop incorporates the frequent items ultrametric tree, rather than conventional FP trees. In FiDoop, three MapReduce jobs are implemented to complete the mining task. In the crucial third MapReduce job, the mappers independently decompose itemsets, the reducers perform combination operations by constructing small ultrametric trees, and the actual mining of these trees separately. We implement FiDoop on our in-house Hadoop cluster. We show that FiDoop on the cluster is sensitive to data distribution and dimensions, because itemsets with different lengths have different decomposition and construction costs. To improve FiDoop's performance, we develop a workload balance metric to measure load balance across the cluster's computing nodes. We develop FiDoop-HD, an extension of FiDoop, to speed up the mining performance for high-dimensional data analysis. Extensive experiments using real-world celestial spectral data demonstrate that our proposed solution is efficient and scalable.Index Terms-Frequent itemsets, frequent items ultrametric tree (FIU-tree), Hadoop cluster, load balance, MapReduce.

An Improved Technique Of Extracting Frequent Itemsets From Massive Data Using MapReduce

“…A balanced parallel FP-growth algorithm used with MapReduce". Frequent itemset mining is a very essential part [12]in association rules and numerous other data mining applications. But when as dataset becomes larger step by step, mining algorithms can not to deal with such excessive databases.…”

“…Its output is obtained from reducer, in the form of frequent one itemset. This candidate 1 itemset shows the completion of the first phase of MapReduce [12]. Second phase accepts the input from first phase in the form of frequent 1-itemsets.…”

“…To accelerate the procedure of FIM is complex and indispensable, because FIM consumes vast significant portion of time to do high computation and input/output intensity. One of the solution to this issue is to use a new parallel frequent itemsets mining algorithm with MapReduce called as FIMMA (Frequent Itemsets Mining with Modified Apriori) [12]. In this modern era datasets are very large so only sequential FIM algorithms are not able to calculate large database and they failed to analyze data correctly and also they suffer from low performance.…”

“…FIMMA takes less time compared with the traditional Apriori. The working of mappers and reducers is done concurrently to optimize the speed, well balancing the load across various clusters [12]. …”

“…In association rule mining, to discover Frequent Itemset , characterizes the two similar itemsets in which first itemsets has similar itemsets of another. These rules are helpful for finding interesting relationships in the datasets and gives insight to the procedure that generated the data [12]. Now a days there are various information creates from different sources like IT enterprises, administrations, advancements and information.…”