Using background knowledge to rank itemsets

Abstract

Assessing the quality of discovered results is an important open problem in data mining. Such assessment is particularly vital when mining itemsets, since commonly many of the discovered patterns can be easily explained by background knowledge. The simplest approach to screen uninteresting patterns is to compare the observed frequency against the independence model. Since the parameters for the independence model are the column margins, we can view such screening as a way of using the column margins as background knowledge. In this paper we study techniques for more flexible approaches for infusing background knowledge. Namely, we show that we can efficiently use additional knowledge such as row margins, lazarus counts, and bounds of ones. We demonstrate that these statistics describe forms of data that occur in practice and have been studied in data mining. To infuse the information efficiently we use a maximum entropy approach. In its general setting, solving a maximum entropy model is infeasible, but we demonstrate that for our setting it can be solved in polynomial time. Experiments show that more sophisticated models fit the data better and that using more information improves the frequency prediction of itemsets.Responsible editors:

Efficient Discovery of the Most Interesting Associations

“…Approaches that take account of background knowledge provide an important and closely related field of research [Jaroszewicz et al 2009;Tatti and Mampaey 2010;De Bie 2011].…”

Spatio-temporal data classification through multidimensional sequential patterns: Application to crop mapping in complex landscape

“…Indeed, most of pattern-based classification techniques focus on the sequential behavior and omit to take contextual and external knowledge into account. Though, recently, a new trend in the data mining field tries to incorporate expert knowledge in the process to improve the result quality [21,22,23]. Our approach clearly comes within this scope and we experimentally show that adding as much as available non-sequential information would lead to increase the classification performances.…”

Summarization of emergency news articles driven by relevance feedback

“…However, the number of mined frequent itemsets is typically very large, because it contains a lot of redundant or potentially irrelevant patterns. To generate a more compact set of frequent itemsets representing most significant yet non-redundant knowledge hidden in the analyzed data many research efforts have been made (e.g., [28], [29], [20], [30]). Given a minimum support threshold minsup and a maximum itemset model size K, we extract the top-K most interesting and non-redundant itemsets according to the entropy-based heuristics proposed in [20].…”