Association Rule Mining on Big Data Sets

Abstract

An accurate, complete, and rapid establishment of customer needs and existence of product recommendations are crucial points in terms of increasing customer satisfaction level in various different sectors such as the banking sector. Due to the significant increase in the number of transactions and customers, analyzing costs regarding time and consumption of memory becomes higher. In order to increase the performance of the product recommendation, we discuss an approach, a sample data creation process, to association rule mining. Thus instead of processing whole population, processing on a sample that represents the population is used to decrease time of analysis and consumption of memory. In this regard, sample composing methods, sample size determination techniques, the tests which measure the similarity between sample and population, and association rules (ARs) derived from the sample were examined. The mutual buying behavior of the customers was found using a well-known association rule mining algorithm. Techniques were compared according to the criteria of complete rule derivation and time consumption.

Scalable recommendation systems based on finding similar items and sequences

“…14 Many different MapReduce methodologies that leverage Hadoop platforms and HDFS are used efficiently to parallelize frequent-item extraction methods. 14,41,43,44 In MapReduce, however, there is a disk I/O bottleneck problem. The RDD architecture was used by Spark to solve this problem.…”

“…Unlike these studies, we focus on machine learning workflows for frequent itemsets mining. There exists some studies that focus on using association rule mining 41,44 and sequential pattern mining libraries [56][57][58] of big data processing frameworks for different purposes. Here, we only focus on utilizing association rule mining and sequential pattern mining algorithms for recommendation systems.…”

“…First, to handle these datasets, MapReduce proposes cluster‐based methods, which are complex and may have synchronization issues 14 . Many different MapReduce methodologies that leverage Hadoop platforms and HDFS are used efficiently to parallelize frequent‐item extraction methods 14,41,43,44 . In MapReduce, however, there is a disk I/O bottleneck problem.…”

Makine Öğrenmesi Yöntemlerine Dayalı Veri Yönetim Sistemi

“…Geleneksel istatistik yöntemlerine baktığımızda en çok kullanılan yöntemlerin satır bazında silme, ortalama alma veya manuel olarak doldurma gibi işlemler olduğunu görürüz [3]. Bu yöntemler arasında en çok kullanılan satır bazlı silme işlemini ele alırsak verinin özelliğine, verilerin birbirleri ile olan ilişkisine ve veri setindeki eksik verilerin toplam veri sayısına oranına bakarak silme işlemi yapıldığında varyans artar ve rassal olarak hatalı veya yanlı sonuçlar üretir [4]. Elde edilen bu yanlı sonuçlar ise araştırmacılar ve şirketler için zaman ve mali kayıplara yol açar.…”