Power-law relationship and self-similarity in the itemset support distribution: analysis and applications

Chuang, Kun-Ta; Huang, Jiun-Long; Chen, Ming-Syan

doi:10.1007/s00778-007-0054-1

Cited by 15 publications

(20 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We can expect the distribution of feature vectors, the mixture of Zipf distributions, to be Zipfian. This has been confirmed for word n-grams [12] and itemset support distribution [6]. We can thus expect that a small set of partial feature vectors will commonly appear in the task.…”

Section: Construction Of Feature Sequence Triesupporting

confidence: 53%

Efficient Classification with Conjunctive Features

Yoshinaga

2012

Journal of Information Processing

View full text Add to dashboard Cite

This paper proposes a method that speeds up a classifier trained with many conjunctive features: combinations of (primitive) features. The key idea is to precompute as partial results the weights of primitive feature vectors that represent fundamental classification problems and appear frequently in the target task. A prefix tree (trie) compactly stores the primitive feature vectors with their weights, and it enables the classifier to find for a given feature vector its longest prefix feature vector whose weight has already been computed. Experimental results on base phrase chunking and dependency parsing demonstrated that our method speeded up the svm and llm classifiers by a factor of 1.8 to 10.6.

show abstract

Section: Construction Of Feature Sequence Triesupporting

confidence: 53%

Efficient Classification with Conjunctive Features

Yoshinaga

2012

Journal of Information Processing

View full text Add to dashboard Cite

show abstract

“…e.g., 'query optimizer' [21], 'selfjoin size estimation' [27], or 'spatial join selectivity' [13]. In relation to frequent pattern mining, Chuang et al observed that pattern frequency-count distribution follows a Power-law distribution [9]. We exploit this finding in efficient estimation of the SSJoin size.…”

Section: Related Workmentioning

confidence: 98%

“…Power-laws have been observed in a wide range of manmade and natural worlds [30,9]. They have been successfully used in the literature.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, several algorithms have been proposed to estimate the number of frequent patterns [4,9,22]. Accurate and efficient estimation of the number of frequent patterns is a key factor in our estimation of SSJoin size.…”

Section: Related Workmentioning

confidence: 99%

“…Accurate and efficient estimation of the number of frequent patterns is a key factor in our estimation of SSJoin size. Although the proposed technique is primarily based on the observation by Chuang et al [9], it can benefit from these new developments as well.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Power-law based estimation of set similarity join size

Lee

Shim

2009

Proc. VLDB Endow.

View full text Add to dashboard Cite

We propose a novel technique for estimating the size of set similarity join. The proposed technique relies on a succinct representation of sets using Min-Hash signatures. We exploit frequent patterns in the signatures for the Set Similarity Join (SSJoin) size estimation by counting their support. However, there are overlaps among the counts of signature patterns and we need to use the set Inclusion-Exclusion (IE) principle. We develop a novel lattice-based counting method for efficiently evaluating the IE principle. The proposed counting technique is linear in the lattice size. To make the mining process very light-weight, we exploit a recently discovered Power-law relationship of pattern count and frequency. Extensive experimental evaluations show the proposed technique is capable of accurate and efficient estimation.

show abstract

Efficient Discovery of Association Rules and Frequent Itemsets through Sampling with Tight Performance Guarantees

Riondato

Upfal

2012

Machine Learning and Knowledge Discovery in Databases

View full text Add to dashboard Cite

Abstract. The tasks of extracting (top-K) Frequent Itemsets (FI's) and Association Rules (AR's) are fundamental primitives in data mining and database applications. Exact algorithms for these problems exist and are widely used, but their running time is hindered by the need of scanning the entire dataset, possibly multiple times. High quality approximations of FI's and AR's are sufficient for most practical uses, and a number of recent works explored the application of sampling for fast discovery of approximate solutions to the problems. However, these works do not provide satisfactory performance guarantees on the quality of the approximation, due to the difficulty of bounding the probability of under-or oversampling any one of an unknown number of frequent itemsets. In this work we circumvent this issue by applying the statistical concept of Vapnik-Chervonenkis (VC) dimension to develop a novel technique for providing tight bounds on the sample size that guarantees approximation within user-specified parameters. Our technique applies both to absolute and to relative approximations of (top-K) FI's and AR's. The resulting sample size is linearly dependent on the VC-dimension of a range space associated with the dataset to be mined. The main theoretical contribution of this work is a characterization of the VC-dimension of this range space and a proof that it is upper bounded by an easy-to-compute characteristic quantity of the dataset which we call d-index, namely the maximum integer d such that the dataset contains at least d transactions of length at least d. We show that this bound is strict for a large class of datasets. The resulting sample size for an absolute (resp. relative) (ε, δ)-approximation of the collection of FI's is O(which is a significant improvement over previous known results. We present an extensive experimental evaluation of our technique on real and artificial datasets, demonstrating the practicality of our methods, and showing that they achieve even higher quality approximations than what is guaranteed by the analysis.

show abstract

Power-law relationship and self-similarity in the itemset support distribution: analysis and applications

Cited by 15 publications

References 25 publications

Efficient Classification with Conjunctive Features

Efficient Classification with Conjunctive Features

Power-law based estimation of set similarity join size

Efficient Discovery of Association Rules and Frequent Itemsets through Sampling with Tight Performance Guarantees

Contact Info

Product

Resources

About