Fast Identification of Relational Constraint Violations

Chandel, A.; Koudas, Nick; Pu, Ken Q.; Srivastava, Divesh

doi:10.1109/icde.2007.367923

Cited by 4 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has also been previous work on detecting violations of hard constraints such as regular and conditional functional dependencies [51], [22]. The techniques in these papers are relevant to our work since they can be used to identify valid clusterings that satisfy all hard constraints.…”

Section: Related Workmentioning

confidence: 98%

Large-Scale Deduplication with Constraints Using Dedupalog

Arasu

Ré

Suciu

2009

2009 IEEE 25th International Conference on Data Engineering

135

141

View full text Add to dashboard Cite

We present a declarative framework for collective deduplication of entity references in the presence of constraints. Constraints occur naturally in many data cleaning domains and can improve the quality of deduplication. An example of a constraint is "each paper has a unique publication venue"; if two paper references are duplicates, then their associated conference references must be duplicates as well. Our framework supports collective deduplication, meaning that we can dedupe both paper references and conference references collectively in the example above. Our framework is based on a simple declarative Datalogstyle language with precise semantics. Most previous work on deduplication either ignore constraints or use them in an ad-hoc domain-specific manner. We also present efficient algorithms to support the framework. Our algorithms have precise theoretical guarantees for a large subclass of our framework. We show, using a prototype implementation, that our algorithms scale to very large datasets. We provide thorough experimental results over real-world data demonstrating the utility of our framework for high-quality and scalable deduplication.

show abstract

Section: Related Workmentioning

confidence: 98%

Large-Scale Deduplication with Constraints Using Dedupalog

Arasu

Ré

Suciu

2009

2009 IEEE 25th International Conference on Data Engineering

135

141

View full text Add to dashboard Cite

show abstract

“…Numerous methods have already been proposed on CFD-based data cleansing [2][3][4][5][6][7][8]. Prior to CFD, there had been data cleansing with FDs [9][10][11][12], and Association Rules (ARs) [13], but here we focus on the methods that have been applied to CFD.…”

Section: Related Workmentioning

confidence: 99%

Inferring the cause of errors for a scalable, accurate, and complete constraint-based data cleansing

Hoshino

Nakayama

Ito

et al. 2015

Vietnam J Comput Sci

View full text Add to dashboard Cite

In real-world dirty data, errors are often not randomly distributed. Rather, they tend to occur only under certain conditions, such as when the transaction is handled by a certain operator, or the weather is rainy. Leveraging such common conditions, or "cause conditions", the proposed data-cleansing algorithm resolves multi-tuple conflicts with high speed, achieves higher completeness, and runs with high accuracy in realistic settings. We first present complexity analyses of the problem, pointing out two subproblems that are NP-complete. We then introduce, for each subproblem, heuristics that work in sub-polynomial time. We also raise the issue that some previous studies overlook the notion of repaircompleteness, which means, having less number of unsolved conflicts in the resulting repairs. The proposed method is capable of obtaining a complete repair if we are allowed to preprocess the input set of constraints. The algorithms are tested with three sets of data and rules. The experiments show that, compared to the state-of-the-art methods for conditional functional dependencies-based and FD-based data cleansing, the proposed algorithm scales better with respect to the data size, is the only method that outputs complete repairs, and is more accurate especially when the error distribution is skewed.

show abstract