Discovering Reliable Approximate Functional Dependencies

Abstract

Given a database and a target attribute of interest, how can we tell whether there exists a functional, or approximately functional dependence of the target on any set of other attributes in the data? How can we reliably, without bias to sample size or dimensionality, measure the strength of such a dependence? And, how can we efficiently discover the optimal or $\alpha$-approximate top-$k$ dependencies? These are exactly the questions we answer in this paper. As we want to be agnostic on the form of the dependence, we adopt an information-theoretic approach, and construct a reliable, bias correcting score that can be efficiently computed. Moreover, we give an effective optimistic estimator of this score, by which for the first time we can mine the approximate functional dependencies from data with guarantees of optimality. Empirical evaluation shows that the derived score achieves a good bias for variance trade-off, can be used within an efficient discovery algorithm, and indeed discovers meaningful dependencies. Most important, it remains reliable in the face of data sparsity.Comment: Accepted: In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), August 13-17, 2017, Halifax, NS, Canad

Discovering Reliable Dependencies from Data: Hardness and Improved Algorithms

“…Moreover, bounding functions can be incorporated as an early termination criterion. For the reliable fraction of information in particular, there is potential to prune many of the higher levels of the search space as it favors solutions that are small in cardinality [14]. The algorithm is presented in Algorithm 2.…”

“…Computing this function is of course equivalent to the original optimization problem and hence NP-hard. We can, however, 14 3 15 22 25 23 24 4 10 26 2 20 9 31 5 27 7 19 28 21 12 29 11 17 35 1 8 30 32 6 33 34 13 Figure 3: Evaluatingf spc for branch-and-bound optimization. Relative nodes explored difference (left) and relative runtime difference (right) between methods OPUS spc and OPUS mon .…”

“…Data are sorted in increasing α values. 14 3 15 22 25 23 24 4 10 26 2 20 9 31 5 27 7 19 28 21 12 29 11 17 35 1 8 30 32 6 33 34 13…”

Telling Cause from Effect Using MDL-Based Local and Global Regression

“…As there is no known combination of physical properties that fully describes δ E , we mine the top 10 features that each explain as much of δ E as possible [15], and therewith obtain 10 cause effect pairs where we set δ E as X and one of the mined features as Y . After consulting with domain experts, we assume Y → X as ground truth for all pairs.…”

Mining Approximate Primary Functional Dependency on Web Tables

“…Wang et al give countingbased algorithms for deriving AFDs and their probabilities [3]. Mandros et al propose to search AFDs by adopting an information theoretic approach [4]. Above approaches only consider AFDs with single attribute on the left-hand side, which are not applicable for tables with more than one entity column.…”