Mining functional dependencies from data

Abstract

In this paper, we propose an efficient rule discovery algorithm, called FD_Mine, for mining functional dependencies from data. By exploiting Armstrong's Axioms for functional dependencies, we identify equivalences among attributes, which can be used to reduce both the size of the dataset and the number of functional dependencies to be checked. We first describe four effective pruning rules that reduce the size of the search space. In particular, the number of functional dependencies to be checked is reduced by skipping the search for FDs that are logically implied by already discovered FDs. Then, we present the FD_Mine algorithm, which incorporates the four pruning rules into the mining process. We prove the correctness of FD_Mine, that is, we show that the pruning does not lead to the loss of useful information. We report the results of a series of experiments. These experiments show that the proposed algorithm is effective on 15 UCI datasets and synthetic data.

Profiling relational data: a survey

“…A functional dependency states that values in one set of columns functionally determine the value of another column. Again, much research has been performed to automatically detect Fds [75,139]. Section 5 surveys dependency discovery algorithms in detail.…”

“…TANE [75], FUN [110], and FD_Mine [139] are three algorithms that follow this strategy, with FUN and FD_Mine introducing additional pruning rules beyond TANE's based on the properties of Fds. They start with sets of single columns in the LHS and work their way up the powerset lattice in a level-wise manner.…”

“…For instance, if we find that A → B and B → A, then we can prune all LHS column sets including B, because A and B are equivalent [139]. Another pruning strategy is to ignore columns sets that have the same number of distinct values as their subsets [110].…”

Data profiling revisited

“…Another form of dependency, which is also relevant for data quality, is the functional dependency (FD). Again, much research has been performed to automatically detect FDs [26,45].…”

Ontology driven search of compound IDs

“…Yet they have been proved not to scale properly for big loads of data, which is the most common scenario in DW systems, as pointed out, for example, in (C. Monash, 2008;Golfarelli and Rizzi, 2009). For this reason, this is still a research topic giving rise to new proposals, like (Sismanis, Brown, Haas and Reinwald, 2006;Yao and Hamilton, 2008;Yeh, Li and Chu, 2008). Furthermore, note that these approaches exclusively work over instances and they cannot easily tolerate erroneous data (that may generate fake IDs that do not really hold or overlook real ones).…”