Large-Scale Deduplication with Constraints Using Dedupalog

Abstract

We present a declarative framework for collective deduplication of entity references in the presence of constraints. Constraints occur naturally in many data cleaning domains and can improve the quality of deduplication. An example of a constraint is "each paper has a unique publication venue"; if two paper references are duplicates, then their associated conference references must be duplicates as well. Our framework supports collective deduplication, meaning that we can dedupe both paper references and conference references collectively in the example above. Our framework is based on a simple declarative Datalogstyle language with precise semantics. Most previous work on deduplication either ignore constraints or use them in an ad-hoc domain-specific manner. We also present efficient algorithms to support the framework. Our algorithms have precise theoretical guarantees for a large subclass of our framework. We show, using a prototype implementation, that our algorithms scale to very large datasets. We provide thorough experimental results over real-world data demonstrating the utility of our framework for high-quality and scalable deduplication.

A Holistic Solution for Duplicate Entity Identification in Deep Web Data Integration

“…Since only one threshold is learned in the previous works, we use the mean of as the threshold. We compare DWDEI with two recent related works [35] and [36]. From the experimental results shown in Table 6 and the performance comparison on F-measure shown in Figure 7, two conclusions can be made.…”

Feature-aligned segmentation using correlation clustering

“…This prevents the result from degenerating to a single cluster (as such a cut uses no negative edges) or |V | clusters (as such a cut includes all positive edges). CC has found uses in problems involving unknown and possibly large number of clusters, such as entity deduplication [37], community detection in social networks [38], gene clustering [39], and image segmentation [10,[40][41][42][43].…”

Defining Key Semantics for the RDF Datasets: Experiments and Evaluations

“…non composite S-keys). Other approaches aim to enrich the ontology and/or use these S-keys to generate identity links between pairs of instances that can be propagated to other pairs of instances ( [19,1]). Such approaches, are called collective or global approaches of data linking.…”