Towards dependable data repairing with fixing rules

Wang, Jiannan; Tang, Nan

doi:10.1145/2588555.2610494

Cited by 92 publications

(29 citation statements)

References 23 publications

(48 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Rule-based detection algorithms [1,6,15,16,25,33] that can be embedded into frameworks, such as Nadeef [8,23], where a rule can vary from a simple "not null" constraint to multi-attribute functional dependencies (FDs) to user-defined functions. Using this class of tools, a user can specify a collection of rules that clean data will obey, and the tool will find any violations.…”

Section: The Current Statementioning

confidence: 99%

Detecting data errors

Abedjan¹,

et al. 2016

Self Cite

View full text Add to dashboard Cite

Data cleaning has played a critical role in ensuring data quality for enterprise applications. Naturally, there has been extensive research in this area, and many data cleaning algorithms have been translated into tools to detect and to possibly repair certain classes of errors such as outliers, duplicates, missing values, and violations of integrity constraints. Since different types of errors may coexist in the same data set, we often need to run more than one kind of tool. In this paper, we investigate two pragmatic questions: (1) are these tools robust enough to capture most errors in real-world data sets? and (2) what is the best strategy to holistically run multiple tools to optimize the detection effort? To answer these two questions, we obtained multiple data cleaning tools that utilize a variety of error detection techniques. We also collected five real-world data sets, for which we could obtain both the raw data and the ground truth on existing errors. In this paper, we report our experimental findings on the errors detected by the tools we tested. First, we show that the coverage of each tool is well below 100%. Second, we show that the order in which multiple tools are run makes a big difference. Hence, we propose a holistic multi-tool strategy that orders the invocations of the available tools to maximize their benefit, while minimizing human effort in verifying results. Third, since this holistic approach still does not lead to acceptable error coverage, we discuss two simple strategies that have the potential to improve the situation, namely domain specific tools and data enrichment. We close this paper by reasoning about the errors that are not detectable by any of the tools we tested. * Work done while doing an internship at QCRI.

show abstract

Section: The Current Statementioning

confidence: 99%

Detecting data errors

Abedjan¹,

et al. 2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…Master data management systems are being developed by IBM, SAP, Microsoft and Oracle. In fact, master data has been used to clean relational data using editing rules [9], fixing rules [21] or Sherlock rules [14]. New matching rules across RDF data and relational master data (or trusted knowledge bases) need to be defined, such that similar techniques can be applied to clean RDF data.…”

Section: Some Possible Directionsmentioning

confidence: 99%

Big RDF data cleaning

Tang

2015

2015 31st IEEE International Conference on Data Engineering Workshops

Self Cite

View full text Add to dashboard Cite

Without a shadow of a doubt, data cleaning has played an important part in the history of data management and data analytics. Possessing high quality data has been proven to be crucial for businesses to do data driven decision making, especially within the information age and the era of big data. Resource Description Framework (RDF) is a standard model for data interchange on the semantic web. However, it is known that RDF data is dirty, since many of them are automatically extracted from the web. In this paper, we will first revisit data quality problems appeared in RDF data. Although many efforts have been put to clean RDF data, unfortunately, most of them are based on laborious manual evaluation. We will also describe possible solutions that shed lights on (semi-)automatically cleaning (big) RDF data.

show abstract

“…The NADEEF [15] system enables the users to declare a set of such rules via an API and uses advanced algorithms to more efficiently process those rules. The Fixing Rules proposed in [16] allow the users to provide evidence patterns, instead of deterministic rules, to avoid erroneous fixes. Here, an erroneous fix is one that "corrects" values in the data to satisfy the rules, but ends up with the whole tuple being semantically incorrect.…”

Section: Data Cleaning Based On Data Integritymentioning

confidence: 99%