Interactive and Deterministic Data Cleaning

He, Jun Kit; Veltri, Enzo; Santoro, Donatello; Li, Guoliang; Mecca, Giansalvatore; Papotti, Paolo; Tang, Nan

doi:10.1145/2882903.2915242

Cited by 63 publications

(43 citation statements)

References 43 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite all these efforts, detecting data errors with high accuracy is far from automatic [3,18], and almost all practical tools (heavily) involve users to properly tune the parameters and provide feedback. PFDs shine some light on automatically detecting data errors by carefully examining the correlation of partial values across different attributes.…”

Section: Related Workmentioning

confidence: 99%

Pattern functional dependencies for data cleaning

Qahtan

Tang²,

Ouzzani³

et al. 2020

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Patterns (or regex-based expressions) are widely used to constrain the format of a domain (or a column), e.g., a Year column should contain only four digits, and thus a value like "1980-" might be a typo. Moreover, integrity constraints (ICs) defined over multiple columns, such as (conditional) functional dependencies and denial constraints, e.g., a ZIP code uniquely determines a city in the UK, have been widely used in data cleaning. However, a promising, but not yet explored, direction is to combine regex-and IC-based theories to capture data dependencies involving partial attribute values. For example, in an employee ID such as"F-9-107", "F" is sufficient to determine the finance department. Inspired by the above observation, we propose a novel class of ICs, called pattern functional dependencies (PFDs), to model fine-grained data dependencies gleaned from partial attribute values. These dependencies cannot be modeled using traditional ICs, such as (conditional) functional dependencies, which work on entire attribute values. We also present a set of axioms for the inference of PFDs, analogous to Armstrong's axioms for FDs, and study the complexity of consistency and implication analysis of PFDs. Moreover, we devise an effective algorithm to automatically discover PFDs even in the presence of errors in the data. Our extensive experiments on 15 real-world datasets show that our approach can effectively discover valid and useful PFDs over dirty data, which can then be used to detect data errors that are hard to capture by other types of ICs.

show abstract

Section: Related Workmentioning

confidence: 99%

Pattern functional dependencies for data cleaning

Qahtan

Tang²,

Ouzzani³

et al. 2020

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In fact, standardizing the vocabulary of pathologies and pathology indicators is crucial in the early stages of data preparation. To this end we used a consolidated suite of data-cleaning tools [8][9][10].…”

Section: The Greg ML Ecosystemmentioning

confidence: 99%

Greg, ML – Machine Learning for Healthcare at a Scale

Lapadula¹,

Mecca

Santoro

et al. 2020

Health Technol.

Self Cite

View full text Add to dashboard Cite

This paper introduces the Greg, ML platform, a machine-learning engine and toolset conceived to generate automatic diagnostic suggestions based on patient profiles. Greg, ML departs from many other experiences in machine learning for healthcare in the fact that it was designed to handle a large number of different diagnoses, in the order of the hundreds. We discuss the architecture that stands at the core of Greg, ML, designed to handle the complex challenges posed by this ambitious goal, and confirm its effectiveness with experimental results based on the working prototype we have developed. Finally, we discuss challenges and opportunities related to the use of this kind of tools in medicine, and some important lessons learned while developing the tool. In this respect, we underline that Greg, ML should be conceived primarily as a support for expert doctors in their diagnostic decisions, and can hardly replace humans in their judgment.

show abstract

“…Technically speaking, there are similarities between the algorithms provided in this paper and the classical powerset lattice and combinatorial set enumeration problems [11], such as data cube modeling [26], frequent item-sets and association rule mining [12], data profiling [27], recommendation systems [28], and data cleaning [29]. While such work, and the algorithms such as apriori, traverse over the powerset lattice, our problem is modeled as the traversal over the pattern graph which has a different structure (and properties) compared to a powerset lattice.…”

Section: Related Workmentioning

confidence: 99%

Assessing and Remedying Coverage for a Given Dataset

Asudeh¹,

Jin²,

Jagadish³

2019

2019 IEEE 35th International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Data analysis impacts virtually every aspect of our society today. Often, this analysis is performed on an existing dataset, possibly collected through a process that the data scientists had limited control over. The existing data analyzed may not include the complete universe, but it is expected to cover the diversity of items in the universe. Lack of adequate coverage in the dataset can result in undesirable outcomes such as biased decisions and algorithmic racism, as well as creating vulnerabilities such as opening up room for adversarial attacks.In this paper, we assess the coverage of a given dataset over multiple categorical attributes. We first provide efficient techniques for traversing the combinatorial explosion of value combinations to identify any regions of attribute space not adequately covered by the data. Then, we determine the least amount of additional data that must be obtained to resolve this lack of adequate coverage. We confirm the value of our proposal through both theoretical analyses and comprehensive experiments on real data. c Ai = Π ∀Aj ∈Ai c j to show the number of value combinations for A i .Definition 1 (Pattern). A pattern P is a vector of size d, in which P [i] is either X (meaning that its value is unspecified) or is a value of attribute A i . We name the elements with value X as non-deterministic and the others as deterministic.An item t matches a pattern P (written as M (t, P ) = ), if for all i for which P [i] is deterministic, t[i] is equal to P [i]. Formally:

show abstract

Interactive and Deterministic Data Cleaning

Cited by 63 publications

References 43 publications

Pattern functional dependencies for data cleaning

Pattern functional dependencies for data cleaning

Greg, ML – Machine Learning for Healthcare at a Scale

Assessing and Remedying Coverage for a Given Dataset

Contact Info

Product

Resources

About