Approximate Denial Constraints

Livshits, Ester; Heidari, Alireza; Ilyas, Ihab F.; Kimelfeld, Benny

doi:10.48550/arxiv.2005.08540

Cited by 2 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In those cases, we ignore this feature and include other features. The next information that might be available is a set of data rules in form of denial constraints [21]. Before, we discuss how we capture this information, lets formally define a denial constraint.…”

Section: Database-level Featuresmentioning

confidence: 99%

Record fusion: A learning approach

Heidari,

Michalopoulos,

Kushagra

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Record fusion is the task of aggregating multiple records that correspond to the same real-world entity in a database. We can view record fusion as a machine learning problem where the goal is to predict the "correct" value for each attribute for each entity. Given a database, we use a combination of attribute-level, recordlevel, and database-level signals to construct a feature vector for each cell (or (row, col)) of that database. We use this feature vector alongwith the ground-truth information to learn a classifier for each of the attributes of the database. Our learning algorithm uses a novel stagewise additive model. At each stage, we construct a new feature vector by combining a part of the original feature vector with features computed by the predictions from the previous stage. We then learn a softmax classifier over the new feature space. This greedy stagewise approach can be viewed as a deep model where at each stage, we are adding more complicated non-linear transformations of the original feature vector. We show that our approach fuses records with an average precision of ∼98% when source information of records is available, and ∼94% without source information across a diverse array of real-world datasets. We compare our approach to a comprehensive collection of data fusion and entity consolidation methods considered in the literature. We show that our approach can achieve an average precision improvement of ∼ 20%/ ∼ 45% with/without source information respectively.Preprint. Under review.

show abstract

Section: Database-level Featuresmentioning

confidence: 99%

Record fusion: A learning approach

Heidari,

Michalopoulos,

Kushagra

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, consider the problem of estimating the mean and/or variance of some column of a table. The presence of duplicates can lead to inaccurate estimates [2]. Another example is unsupervised learning tasks, such as k-means clustering, where the presence of duplicates can perturb the computed centres and drastically change the clustering output.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we investigate certain properties of the data under which it is possible to construct efficient (both statistical and computational) procedures that can sample uniformly at random form the set of entities E. We consider three categories of datasets/methods: (1) in Section 3, we consider datasets that are 'balanced' (Defn. 2) and show how that can help us estimate the frequencies of all the entities from a 'small' sample; (2) in Section 4, we consider datasets that can be successfully partitioned into hashing blocks, and we show how access to such blocks can help us estimate the frequencies and then sample uniformly from the set of entities E; and (3) in section 5, we consider the case when the dataset is generated by a mixture of k-spherical Gaussian distributions. For all the three cases, we provide mathematical bounds to prove correctness of our approach.…”

Section: Introductionmentioning

confidence: 99%

On sampling from data with duplicate records

Heidari¹,

Kushagra²,

Ilyas³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Data deduplication is the task of detecting records in a database that correspond to the same real-world entity. Our goal is to develop a procedure that samples uniformly from the set of entities present in the database in the presence of duplicates. We accomplish this by a two-stage process. In the first step, we estimate the frequencies of all the entities in the database. In the second step, we use rejection sampling to obtain a (approximately) uniform sample from the set of entities. However, efficiently estimating the frequency of all the entities is a non-trivial task and not attainable in the general case. Hence, we consider various natural properties of the data under which such frequency estimation (and consequently uniform sampling) is possible. Under each of those assumptions, we provide sampling algorithms and give proofs of the complexity (both statistical and computational) of our approach. We complement our study by conducting extensive experiments on both real and synthetic datasets.Preprint. Under review.

show abstract

Approximate Denial Constraints

Cited by 2 publications

References 34 publications

Record fusion: A learning approach

Record fusion: A learning approach

On sampling from data with duplicate records

Contact Info

Product

Resources

About