Association rule mining algorithm based on Spark for pesticide transaction data analyses

Abstract

With the development of smart agriculture, the accumulation of data in the field of pesticide regulation has a certain scale. The pesticide transaction data collected by the Pesticide National Data Center alone produces more than 10 million records daily. However, due to the backward technical means, the existing pesticide supervision data lack deep mining and usage. The Apriori algorithm is one of the classic algorithms in association rule mining, but it needs to traverse the transaction database multiple times, which will cause an extra IO burden. Spark is an emerging big data parallel computing framework with advantages such as memory computing and flexible distributed data sets. Compared with the Hadoop MapReduce computing framework, IO performance was greatly improved. Therefore, this paper proposed an improved Apriori algorithm based on Spark framework, ICAMA. The MapReduce process was used to support the candidate set and then to generate the candidate set. After experimental comparison, when the data volume exceeds 250 Mb, the performance of Spark-based Apriori algorithm was 20% higher than that of the traditional Hadoop-based Apriori algorithm, and with the increase of data volume, the performance improvement was more obvious.

Research on supermarket replenishment volume prediction based on LSTM and PSO algorithm

“…The Apriori algorithm is a classical algorithm for data mining and association rules to mine to find frequent association relations between items in the dataset.The Apriori algorithm has the advantages of finding potential correlation and flexibility as well as unsupervised learning, which can be used to mine potential correlation in datasets and discover association rules. [9] This paper first scans the dataset to determine the frequency of sales volume and finds the set of frequent items based on the minimum support threshold finding that the items exceed the threshold in the dataset.Candidates sets were generated using the frequent set and the dataset was scanned again to determine the frequency of each candidate set.The above procedure is finally repeated until a new set of frequent items cannot be generated.It is helpful for us to use frequent item sets to generate association rules,which can help understand the correlation between items and can be used for prediction. [10] 2.2.…”

Multi-threaded approach in generating frequent itemset of Apriori algorithm based on trie data structure

“…The hash tree [5]- [7] has been implemented in the distributed computing environment of Spark. Another used Hadoop-map reduce framework [8], [9], and graph computing techniques for frequent itemsets mining have been published by Zhang et al [10].…”

Optimization of Teaching Management System Based on Association Rules Algorithm

“…Ramu et al [17] proposed an association rule mining method based on cloud computing by improving the internal framework of the platform to meet the needs of association rule mining. Bai et al [18] proposed an association rule algorithm based on the MapReduce computing framework. e algorithm uses Map and Reduce to solve the candidate itemset and reduce statistics.…”