Pattern functional dependencies for data cleaning

Abstract

Patterns (or regex-based expressions) are widely used to constrain the format of a domain (or a column), e.g., a Year column should contain only four digits, and thus a value like "1980-" might be a typo. Moreover, integrity constraints (ICs) defined over multiple columns, such as (conditional) functional dependencies and denial constraints, e.g., a ZIP code uniquely determines a city in the UK, have been widely used in data cleaning. However, a promising, but not yet explored, direction is to combine regex-and IC-based theories to capture data dependencies involving partial attribute values. For example, in an employee ID such as"F-9-107", "F" is sufficient to determine the finance department. Inspired by the above observation, we propose a novel class of ICs, called pattern functional dependencies (PFDs), to model fine-grained data dependencies gleaned from partial attribute values. These dependencies cannot be modeled using traditional ICs, such as (conditional) functional dependencies, which work on entire attribute values. We also present a set of axioms for the inference of PFDs, analogous to Armstrong's axioms for FDs, and study the complexity of consistency and implication analysis of PFDs. Moreover, we devise an effective algorithm to automatically discover PFDs even in the presence of errors in the data. Our extensive experiments on 15 real-world datasets show that our approach can effectively discover valid and useful PFDs over dirty data, which can then be used to detect data errors that are hard to capture by other types of ICs.

TabReformer: Unsupervised Representation Learning for Erroneous Data Detection

“…Alternatively, pattern mining approaches attempt to discover the syntactic and semantic characterizations of the data. One technique for pattern discovery is inducing functional dependencies from the data [30][31][32]. Functional dependencies are considered a special form of denial constraints [30] and are commonly used to specify business rules.…”

“…Existing research [31] has studied repeated patterns in the data and formalize them into functional dependencies to suggest better repair solutions. Another study [32] focuses on deriving such dependencies with the presence of erroneous data; the method [32] introduces a new class of integrity constraints that can infer dependencies between data attributes even if a portion of the attributes violates these dependencies.…”

RPT

“…(i) Only examine the data at hand. There are integrity constraints (FDs [6], its extensions CFDs [26] and PFDs [54], denial constraints [16], and rule-based methods [33,66]), and probabilistic based methods (e.g., HoloClean [57]). They need enough signals or data redundancy from D. Supervised ML based methods (e.g., GDR [70], SCAREd [69], Raha [49] and Baran [48]) learn only from the data at hand, which cannot be generalized to other datasets.…”

RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation

“…This type of techniques cleans 𝐷 by only examining 𝐷, where predefined domain knowledge is often coded as rules. There are integrity constraints (FDs [10], its extensions conditional FDs [26] and pattern FDs [53], and denial constraints [16]), and probabilistic based methods (e.g., HoloClean [55]). They need enough signals or data redundancy from 𝐷.…”