The merge/purge problem for large databases

Abstract

Many commercial organizations routinely gather large numbers of databases for various marketing and business analysis functions. The task is to correlate information from different databases by identifying distinct individuals that appear in a number of different databases typically in an inconsistent and often incorrect fashion. The problem we study here is the task of merging data from multiple sources in as efficient manner as possible, while maximizing the accuracy of the result. We call this the merge/purge problem. In this paper we detail the sorted neighborhood method that is used by some to solve merge/purge and present experimental results that demonstrates this approach may work well in practice but at great expense. An alternative method based upon clustering is also presented with a comparative evaluation to the sorted neighborhood method. We show a means of improving the accuracy of the results based upon a multi-pass approach that succeeds by computing the Transitive Closure over the results of independent runs considering alternative primary key attributes in each pass.

Mining for Information Discovery on the Web: Overview and Illustrative Research

“…Extended-Manual manually specifies the matching rules (e.g., "if similarity(name1,name2) ≥ 0.8 but position=student then the two tuples do not match"). Thus, in a sense this method extends the manual method described in [46], which would exploit only shared attributes such as "name1" and "name2". Extended-AR is similar to Extended-Manual, but uses the association rule classification method of [63] to guide the process of generating rules.…”

“…Object matching is often used to consolidate information about entities and to remove duplicates when merging multiple information sources. As such, it plays an important role in many information processing contexts, including information integration, data warehousing, information extraction, and text join in databases (e.g., [95,21,72,98,7,57,1,90,42,46]). …”

“…The performance of matching algorithms have typically been evaluated with matching accuracy and runtime efficiency [46,1]. As the first step, in this section we shall focus on improving matching accuracy.…”

Classification of Contradiction Patterns

“…Identification of tuples representing the same individual is accomplished by the unique object identifier ID. The problem of assigning these object identifiers is not considered within this paper, i.e., we assume a preceding duplicate detection step (see for example Hernandez and Stolfo (1995)). Note that we are only interested in finding update operations that introduce conflicts between the overlapping parts of databases.…”

Privacy Preserving Group Linkage

“…Record linkage (RL), also known as the merge-purge [12] or object identity [24] problem, is one of the key tasks in data cleaning [10] and integration [9]. Its goal is to identify related records that are associated with the same entity from multiple databases.…”