Adaptive-Miner: an efficient distributed association rule mining algorithm on Spark

Abstract

Extraction of valuable data from extensive datasets is a standout amongst the most vital exploration issues. Association rule mining is one of the highly used methods for this purpose. Finding possible associations between items in large transaction based datasets (finding frequent itemsets) is most crucial part of the association rule mining task. Many single-machine based association rule mining algorithms exist but the massive amount of data available these days is above the capacity of a single machine based algorithm. Therefore, to meet the demands of this ever-growing enormous data, there is a need for distributed association rule mining algorithm which can run on multiple machines. For these types of parallel/distributed applications, MapReduce is one of the best fault-tolerant frameworks. Hadoop is one of the most popular open-source software frameworks with MapReduce based approach for distributed storage and processing of large datasets using standalone clusters built from commodity hardware. But heavy disk I/O operation at each iteration of a highly iterative algorithm like Apriori makes Hadoop inefficient. A number of MapReduce based platforms are being developed for parallel computing in recent years. Among them, a platform, namely, Spark have attracted a lot of attention because of its inbuilt support to distributed computations. Therefore, we implemented a distributed association rule mining algorithm on Spark named as Adaptive-Miner which uses adaptive approach for finding frequent patterns with higher accuracy and efficiency. Adaptive-Miner uses an adaptive strategy based on the partial processing of datasets. Adaptive-Miner makes execution plans before every iteration and goes with the best suitable plan to minimize time and space complexity. Adpative-Miner is a dynamic association rule mining algorithm which change its approach based on the nature of dataset. Therefore, it is different and better than state-of-the-art static association rule mining algorithms. We conduct in-depth experiments to gain insight into the effectiveness, efficiency, and scalability of the Adaptive-Miner algorithm on Spark.

RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework

“…R-Apriori is similar to YAFIM with an additional phase that reduces the computation to generate 2-itemsets. Adaptive-Miner [8] is an improvement over the R-Apriori, which dynamically selects a conventional or reduced approach of candidate generation, based on the number of frequent itemsets in recent iteration. DFIMA (Distributed Frequent Itemset Mining Algorithm) [9] is also an Apriori-based algorithm on Spark.…”

Collaborative filtering and association rule mining‐based market basket recommendation on spark

“…Tu and He proposed a parallel algorithm for mining association rules based on FP‐tree. Rathee et al presented a parallel Apriori algorithm and implemented a distributed association rule mining algorithm on Spark . On the other hand, to better decrease the runtime of ARM, bio‐inspired approaches such as Genetic Algorithms (GA) and particle swarm optimization have been applied to find approximate solutions for ARM.…”

“…Rathee et al presented a parallel Apriori algorithm 18 and implemented a distributed association rule mining algorithm on Spark. 19 On the other hand, to better decrease the runtime of ARM, bio-inspired approaches such as Genetic Algorithms (GA) and particle swarm optimization have been applied to find approximate solutions for ARM. For example, Djenouri et al 20 proposed an efficient GA-based parallel algorithm that runs on clusters of GPUs.…”

An Efficient Distributed Frequent Itemset Mining Algorithm Based on Spark for Big Data

“…Rathee continued improving his idea. In 2018, he developed a new algorithm named Adaptive-Miner [32] which is one of the best state-of-the-art frequent itemset mining algorithms. It uses an adaptive method for extracting frequent itemsets with higher accuracy and efficiency.…”

“…It reduces time and space by selecting the best plan. R-Apriori and Adaptive-Miner use bloom filters, which are faster than hash trees that make these algorithms more efficient with respect to time and space [31,32].…”