Multipass Algorithms for Mining Association Rules in Text Databases

Abstract

In this paper, we propose two new algorithms for mining association rules between words in text databases. The characteristics of text databases are quite different from those of retail transaction databases, and existing mining algorithms cannot handle text databases efficiently because of the large number of itemsets (i.e., words) that need to be counted. Two well-known mining algorithms, Apriori algorithm and Direct Hashing and Pruning (DHP) algorithm, are evaluated in the context of mining text databases, and are compared with the new proposed algorithms named Multipass-Apriori (M-Apriori) and Multipass-DHP (M-DHP). It has been shown that the proposed algorithms have better performance for large text databases.

Parallel mining of association rules from text databases

“…Each processor can use the global hash table to determine if a candidate might have a sufficient support count, and also to determine which processors should be polled to obtain the support count of the candidate. This approach is unlikely to be efficient for text databases as the underlying DHP hash table was shown to be ineffective in mining text databases [20].…”

“…In addition, the whole set of candidates needs to fit in the memory of each processor for efficient counting, whereas the number of candidates is usually very large for text databases. The Multipass approach proposed in [20] can be used to control the number of candidates to be processed at the same time.…”

“…The classic implementation of this approach is the Apriori algorithm [2] which we discuss below. There have been three basic directions for improving Apriori: reducing the number of candidate itemsets using hashing [20,21,26]; reducing the size of the database through transaction trimming and pruning [20-22, 24, 26]; and reducing the number of passes on the database through partitioning [28], sampling [30], and by finding the maximal frequent itemsets [1,5,7,17,34]. The second approach is to count the itemsets without generating the candidates by transforming the database into a tree data structure [4,19].…”

Rough Association Rule Mining in Text Documents for Acquiring Web User Information Needs

“…Data mining has been used in Web text mining, which refers to the process of searching through unstructured data on the Web and deriving meaning from it [6] [8]. One of main purposes of text mining is association discovery [2].…”

Automatic Pattern-Taxonomy Extraction for Web Mining

“…Association mining has been used in Web text mining, which refers to the process of searching through unstructured data on the Web and deriving meanings from it [8] [11]. The main purposes of text mining include association discovery, trends discovery, and event discovery [5].…”