Towards certain fixes with editing rules and master data

Abstract

A variety of integrity constraints have been studied for data cleaning. While these constraints can detect the presence of errors, they fall short of guiding us to correct the errors. Indeed, data repairing based on these constraints may not find certain fixes that are absolutely correct, and worse, may introduce new errors when repairing the data. We propose a method for finding certain fixes, based on master data, a notion of certain regions, and a class of editing rules. A certain region is a set of attributes that are assured correct by the users. Given a certain region and master data, editing rules tell us what attributes to fix and how to update them. We show how the method can be used in data monitoring and enrichment. We develop techniques for reasoning about editing rules, to decide whether they lead to a unique fix and whether they are able to fix all the attributes in a tuple, relative to master data and a certain region. We also provide an algorithm to identify minimal certain regions, such that a certain fix is warranted by editing rules and master data as long as one of the regions is correct. We experimentally verify the effectiveness and scalability of the algorithm.

Matching dependencies: semantics and query answering

“…Edit rules for master data is another area of data cleaning research in which a dynamic semantics has been proposed [19]. In contrast to duplicate resolution, which prescribes matchings of values without specifying the update value, edit rules require that a value be updated to a specific value contained in master data.…”

CrowdAidRepair: A Crowd-Aided Interactive Data Repairing Method

“…The third category of solutions are external source based repairing approaches, which leverage the information in reference master data set [5] or user's interaction data such as GuidedRepair [13] and NADEEF [3] for better data cleaning performance. However, the required external information is not always available and thus the methods can not be applied in general scenarios.…”

Improving Data Cleansing Accuracy - A Model-based Approach

“…events 01 and 02) can occur arbitrarily many times in a career before the event 04 is met (where the inconsistency is detected), therefore there can not be a value of k higher enough to surely catch the inconsistency described; (3) the higher the value of k, the more the set of possible sequence (variations) to be checked; (4) Functional Dependencies do not provide any hint on how to fix inconsistencies, as discussed in (Fan et al, 2010).…”

“…Nevertheless, the usefulness of formal systems in databases has been motivated in (Vardi, 1987) by observing that FDs are only a fragment of the firstorder logic used in formal methods while (Fan et al, 2010) observed that, even though FDs allow one to detect the presence of errors, they have a limited usefulness since they fall short of guiding one in correcting the errors.…”