Conditional Functional Dependencies for Data Cleaning

Bohannon, Philip; Fan, Wenfei; Geerts, Floris; Jia, Xibei; Kementsietsidis, Anastasios

doi:10.1109/icde.2007.367920

Cited by 292 publications

(255 citation statements)

References 12 publications

Supporting

Mentioning

247

Contrasting

Unclassified

Order By: Relevance

“…The most common dependencies in the relational database model are functional dependencies (FD's), which have been widely studied in the field of database theory ( [16]). The reason of this success may be twofold: on the one hand, their semantics is very simple and intuitive, on the other hand, they have been proven to be very versatile, since they can be used for database design, database validation, and, also, data cleaning [9]. They play a key role to explain the normalization of a database scheme in the relational database model.…”

Section: Introductionmentioning

confidence: 99%

A Formal Context for Acyclic Join Dependencies

Baixeries

2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

A Formal Context for Acyclic Join Dependencies

Baixeries

2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…6 In all the experiments, we set the window size to 2 M and the sliding step to 1 M tuples respectively, regardless which windowing strategy we use. If not otherwise specified, we use Bleach windowing as the default strategy.…”

Section: Discussionmentioning

confidence: 99%

“…Many approaches [6,20,4,9,10,19,16] tackle the problem of detecting and repairing dirty data based on predefined data quality rules. [9] proposes a way to combine multiple rules together and to perform data cleaning work holistically.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Bleach: A Distributed Stream Data Cleaning System

Tian

Michiardi

Vukolić

2017

2017 IEEE International Congress on Big Data (BigData Congress)

View full text Add to dashboard Cite

In this paper we address the problem of rule-based stream data cleaning, which sets stringent requirements on latency, rule dynamics and ability to cope with the unbounded nature of data streams.We design a system, called Bleach, which achieves realtime violation detection and data repair on a dirty data stream. Bleach relies on efficient, compact and distributed data structures to maintain the necessary state to repair data, using an incremental version of the equivalence class algorithm. Additionally, it supports rule dynamics and uses a "cumulative" sliding window operation to improve cleaning accuracy.We evaluate a prototype of Bleach using a TPC-DS derived dirty data stream and observe its high throughput, low latency and high cleaning accuracy, even with rule dynamics. Experimental results indicate superior performance of Bleach compared to a baseline system built on the microbatch streaming paradigm.

show abstract

“…Data dependencies have already shown their importance in various data-oriented practice [8], such as optimizing query evaluation [9], capturing data inconsistency [10], removing data duplicates [7], etc. It is promising to study data dependencies for the heterogeneous data in dataspaces as well.…”

Section: Introductionmentioning

confidence: 99%

On data dependencies in dataspaces

Song

Chen

2011

2011 IEEE 27th International Conference on Data Engineering

View full text Add to dashboard Cite

Abstract-To study data dependencies over heterogeneous data in dataspaces, we define a general dependency form, namely comparable dependencies (CDs), which specifies constraints on comparable attributes. It covers the semantics of a broad class of dependencies in databases, including functional dependencies (FDs), metric functional dependencies (MFDs), and matching dependencies (MDs). As we illustrated, comparable dependencies are useful in real practice of dataspaces, e.g., semantic query optimization. Due to the heterogeneous data in dataspaces, the first question, known as the validation problem, is to determine whether a dependency (almost) holds in a data instance. Unfortunately, as we proved, the validation problem with certain error or confidence guarantee is generally hard. In fact, the confidence validation problem is also NP-hard to approximate to within any constant factor. Nevertheless, we develop several approaches for efficient approximation computation, including greedy and randomized approaches with an approximation bound on the maximum number of violations that an object may introduce. Finally, through an extensive experimental evaluation on real data, we verify the superiority of our methods.

show abstract

Conditional Functional Dependencies for Data Cleaning

Abstract: We propose a class of constraints, referred to as conditional functional dependencies (CFDs), and study their applications in data cleaning. In contrast to traditional functional dependencies (FDs)

Cited by 292 publications

References 12 publications

A Formal Context for Acyclic Join Dependencies

A Formal Context for Acyclic Join Dependencies

Bleach: A Distributed Stream Data Cleaning System

On data dependencies in dataspaces

Contact Info

Product

Resources

About