Divide & conquer-based inclusion dependency discovery

Abstract

The discovery of all inclusion dependencies (INDs) in a dataset is an important part of any data profiling effort. Apart from the detection of foreign key relationships, INDs can help to perform data integration, query optimization, integrity checking, or schema (re-)design. However, the detection of INDs gets harder as datasets become larger in terms of number of tuples as well as attributes. To this end, we propose Binder, an IND detection system that is capable of detecting both unary and n-ary INDs. It is based on a divide & conquer approach, which allows to handle very large datasets-an important property on the face of the ever increasing size of today's data. In contrast to most related works, we do not rely on existing database functionality nor assume that inspected datasets fit into main memory. This renders Binder an efficient and scalable competitor. Our exhaustive experimental evaluation shows the high superiority of Binder over the state-of-the-art in both unary (Spider) and n-ary (Mind) IND discovery. Binder is up to 26x faster than Spider and more than 2500x faster than Mind.

GB-KMV: An Augmented KMV Sketch for Approximate Containment Similarity Search

“…In order to compute the above optimization problem, we try to extract the roots of the first derivative function of Equation 33( i.e., f (r, α 1 , α 2 , b)) with respect to r. However, the derivative function is a polynomial function with degree of r larger than four. According to Abel's impossibility theorem [39], there is no algebraic solution, thus we try to give the numerical solution.…”

“…In a dataset, the discovery of all inclusion dependencies is a crucial part of data profiling efforts. It has many applications such as foreign-key detection and data integration(e.g., [22], [31], [8], [33], [30]).…”

Profiling relational data: a survey

“…Additionally, hybrid algorithms have been proposed in [87,102] that combine bottom-up and top-down traversal for additional pruning. The Binder algorithm uses divide and conquer principles to handle larger datasets than related work [114]. In the divide step, it splits the input dataset horizontally into partitions and vertically into buckets with the goal to fit each partition into main memory.…”

AutoMode: Relational Learning with Less Black Magic

“…If they are not available in the schema, one can extract them from the database content. AutoMode uses the Binder algorithm [9] to discover INDs from the database, shown by the Exact IND discovery box in Figure 1, and generates all unary INDs implied by them.…”