Using association rules to mine for strong approximate dependencies

Sánchez, Daniel; Serrano, José María González; Blanco, Ignacio J.; Martín-Bautista, María J.; Vila, M. A.

doi:10.1007/s10618-008-0092-3

Cited by 24 publications

(13 citation statements)

References 27 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Another approach to harness the complexity is to use heuristics to prune potentially uninteresting AFD candidates [40]. Because this can cause the loss of interesting results, Pyro instead discovers all approximate dependencies for some given error threshold and leaves filtering or ranking of the dependencies to use-case specific post-processing.…”

Section: Related Workmentioning

confidence: 99%

“…Apart from dependency discovery, several other problems, such as frequent itemset mining [3,14], belong in this category and can be tackled with the same algorithmic foundations [32]. For instance, AFD discovery can be modeled as a frequent itemset mining problem; however, such adaptations require additional tailoring to be practically usable [40].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Efficient discovery of approximate dependencies

Kruse

Naumann

2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

Functional dependencies (FDs) and unique column combinations (UCCs) form a valuable ingredient for many data management tasks, such as data cleaning, schema recovery, and query optimization. Because these dependencies are unknown in most scenarios, their automatic discovery has been well researched. However, existing methods mostly discover only exact dependencies, i.e., those without violations. Realworld dependencies, in contrast, are frequently approximate due to data exceptions, ambiguities, or data errors. This relaxation to approximate dependencies renders their discovery an even harder task than the already challenging exact dependency discovery. To this end, we propose the novel and highly efficient algorithm Pyro to discover both approximate FDs and approximate UCCs. Pyro combines a separate-and-conquer search strategy with sampling-based guidance that quickly detects dependency candidates and verifies them. In our broad experimental evaluation, Pyro outperforms existing discovery algorithms by a factor of up to 33, scales to larger datasets, and at the same time requires the least main memory.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Efficient discovery of approximate dependencies

Kruse

Naumann

2018

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

“…Approximate dependencies were later defined to capture dependencies with exceptions. The "quality" of an approximated dependency is evaluated by measures such as support and confidence, whose definitions derive from that of association rules [10].…”

Section: Functional Approximate and Temporal Dependenciesmentioning

confidence: 99%

Data stream partitioning re-optimization based on runtime dependency mining

Viel

Ueda

2014

2014 IEEE 30th International Conference on Data Engineering Workshops

View full text Add to dashboard Cite

In distributed data stream processing, a program made of multiple queries can be parallelized by partitioning input streams according to the values of specific attributes, or partitioning keys. Applying different partitioning keys to different queries requires re-partitioning intermediary streams, causing extra communication and reduced throughput. Repartitionings can be avoided by detecting dependencies between the partitioning keys applicable to each query. Existing partitioning optimization methods analyze query syntax at compile-time to detect inter-key dependencies and avoid repartitionings. This paper extends those compile-time methods by adding a runtime re-optimization step based on the mining of temporal approximate dependencies (TADs) between partitioning keys. A TAD is defined in this paper as a type of dependency that can be approximately valid over a moving time window. Our evaluation, based on a simulation of the Linear Road Benchmark, showed a 94.5% reduction of the extra communication cost.

show abstract

“…Like the RBFNN, the connection weight vector (1) i w of the first hidden layer may be beforehand randomly selected from the total training sample vectors or determined by some adaptive methods [10] (such as k-means clustering, Kohonen training method , LBG, RPCL [11], etc.). Thus the output i y corresponding to the i-th hidden node can be expressed as…”

Section: The Neural Network Trainingmentioning

confidence: 99%

An Efficient Neuro-Fuzzy-Genetic Data Mining Framework Based on Computational Intelligence

Zhang

2009

2009 Ninth International Conference on Hybrid Intelligent Systems

View full text Add to dashboard Cite

In this paper, we combine computational intelligence tools: neural network, fuzzy logic, and genetic algorithm to develop a data mining framework (DMFBCI), which discovers patterns and represents them in understandable forms. In the DMFBCI, input data are preprocessed by fuzzification or one-of-m coding, then, Principal Component Analysis (PCA) is applied to reduce the dimensions of the preprocessed input variables in finding combinations of variables. The reduced dimensions of input variables are then used to train a radial basis probabilistic neural network (RBPNN) to classify the dataset according to the classes considered. A rule extraction technique is then applied in order to extract explicit knowledge from the trained neural networks and represent it in the form of fuzzy if-then rules. In the final stage, a genetic algorithm is used as a rule-pruning module to eliminate those weak rules that are still in the rule bases. Comparison with some known neural network classifier, the architecture we proposed has fast learning speed, and it is characterized by the incorporation of the possibility information into the consequents of classification rules in human understandable forms. The experiments show that the DMFBCI is more efficient and more robust than traditional method such as decision tree approaches such as CART, C4.5.

show abstract

Using association rules to mine for strong approximate dependencies

Cited by 24 publications

References 27 publications

Efficient discovery of approximate dependencies

Efficient discovery of approximate dependencies

Data stream partitioning re-optimization based on runtime dependency mining

An Efficient Neuro-Fuzzy-Genetic Data Mining Framework Based on Computational Intelligence

Contact Info

Product

Resources

About