D-Swoosh: A Family of Algorithms for Generic, Distributed Entity Resolution

We consider the Entity Resolution (ER) problem (also known as deduplication, or merge-purge), in which records determined to represent the same real-world entity are successively located and merged. We formalize the generic ER problem, treating the functions for comparing and merging records as black-boxes, which permits expressive and extensible ER solutions. We identify four important properties that, if satisfied by the match and merge functions, enable much more efficient ER algorithms. We develop three efficient ER algorithms: G-Swoosh for the case where the four properties do not hold, and R-Swoosh and F-Swoosh that exploit the 4 properties. F-Swoosh in addition assumes knowledge of the "features" (e.g., attributes) used by the match function. We experimentally evaluate the algorithms using comparison shopping data from Yahoo! Shopping and hotel information data from Yahoo! Travel. We also show that R-Swoosh (and F-Swoosh) can be used even when the four match and merge properties do not hold, if an "approximate" result is acceptable.

show abstract

“…Though our server had multiple processors, we did not exploit parallelism. This is a topic we are addressing in a separate paper [7].…”

Section: Experimental Settingmentioning

confidence: 96%

“…It is also possible to distribute the ER computations across multiple processors, in order to handle larger data sets. In [7] we study various strategies for distributing the work done by R-Swoosh.…”

Section: Scalabilitymentioning

confidence: 99%

Swoosh: a generic approach to entity resolution

et al. 2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, for every tuple pair, computing the satisfied set of predicates is independent of each other. In our implementation we use the Grid Scheme strategy, a standard approach to scale in entity resolution [4]. We partition the data into B blocks, and define each task as a comparison of tuples from two blocks.…”

Section: Proofmentioning

confidence: 99%

Discovering denial constraints

Chu

Ilyas²,

Papotti³

2013

Proc. VLDB Endow.

183

117

View full text Add to dashboard Cite

Integrity constraints (ICs) provide a valuable tool for enforcing correct application semantics. However, designing ICs requires experts and time. Proposals for automatic discovery have been made for some formalisms, such as functional dependencies and their extension conditional functional dependencies. Unfortunately, these dependencies cannot express many common business rules. For example, an American citizen cannot have lower salary and higher tax rate than another citizen in the same state. In this paper, we tackle the challenges of discovering dependencies in a more expressive integrity constraint language, namely Denial Constraints (DCs). DCs are expressive enough to overcome the limits of previous languages and, at the same time, have enough structure to allow efficient discovery and application in several scenarios. We lay out theoretical and practical foundations for DCs, including a set of sound inference rules and a linear algorithm for implication testing. We then develop an efficient instance-driven DC discovery algorithm and propose a novel scoring function to rank DCs for user validation. Using real-world and synthetic datasets, we experimentally evaluate scalability and effectiveness of our solution.

show abstract

“…The model and the algorithms are extended in [32] for handling approximate results as records with confidences. [1] adapts the algorithms to a distributed environment. Our generic ER algorithms for relational databases were published in [35].…”

Section: Related Workmentioning

confidence: 99%

“…If we use multiple features, the unified feature-based match function is defined as a disjunction. For some feature set f 1 The third aspect of a feature is indexability. To ensure the efficient computation of feature-based matches, we expect features and their match functions to be indexable.…”

Section: Featuresmentioning

confidence: 99%

Flexible and Efficient Distributed Resolution of Large Entities

Molnár

Benczúr

Sidló

2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Entity resolution (ER) is a computationally hard problem of data integration scenarios, where database records have to be grouped according to the real-world entities they belong to. In practice these entities may consist of only a few records from different data sources with typos or historical data. In other cases they may contain significantly more records, especially when we search for entities on a higher level of a concept hierarchy than records.In this paper we give theoretical foundation of a variety of practically important match functions. We show that under these formulations, ER with large entities can be solved efficiently with algorithms based on MapReduce, a distributed computing paradigm. Our algorithm can efficiently incorporate probabilistic and similarity-based record match, enabling flexible match function definition. We demonstrate the usability of our model and algorithm in a real-world insurance ER scenario, where we identify household groups of client records.

show abstract

D-Swoosh: A Family of Algorithms for Generic, Distributed Entity Resolution

Cited by 26 publications

References 22 publications

Swoosh: a generic approach to entity resolution

Swoosh: a generic approach to entity resolution

Discovering denial constraints

Flexible and Efficient Distributed Resolution of Large Entities

Contact Info

Product

Resources

About