A Practical and Effective Sampling Selection Strategy for Large Scale Deduplication

Abstract

The data deduplication task has attracted a considerable amount of attention from the research community in order to provide effective and efficient solutions. The information provided by the user to tune the deduplication process is usually represented by a set of manually labeled pairs. In very large datasets, producing this kind of labeled set is a daunting task since it requires an expert to select and label a large number of informative pairs. In this article, we propose a two-stage sampling selection strategy (T3S) that selects a reduced set of pairs to tune the deduplication process in large datasets. T3S selects the most representative pairs by following two stages. In the first stage, we propose a strategy to produce balanced subsets of candidate pairs for labeling. In the second stage, an active selection is incrementally invoked to remove the redundant pairs in the subsets created in the first stage in order to produce an even smaller and more informative training set. This training set is effectively used both to identify where the most ambiguous pairs lie and to configure the classification approaches. Our evaluation shows that T3S is able to reduce the labeling effort substantially while achieving a competitive or superior matching quality when compared with state-of-the-art deduplication methods in large datasets.

Redblock: a tool for online deduplication on large datasets

“…A traditional method of producing data blocks is defining one of the attributes as the blocking criterion or key [4]. Only records that have the same blocking key will be inserted into the same block, reducing substantially the number of comparisons.…”

“…However, such a method results in a quadratic cost of processing which would be non-practical in the case of a large dataset. In this context, blocking appears as an alternative to reduce the search space by only processing the records that have some indication of representing a duplicate [4].…”

“…Thus, it is important for the blocking process that represents the largest processing slice [4], to be efficient enough to not result in processing delays.…”

“…This initial experimentation illustrates that Redblock is able to promote a deduplication technique with high efficiency, finding all duplicate pairs as well as the classification method 4 Table 7 states the efficiency of Redblock spouts and bolts regarding time consumed to be executed. More specifically, it shows the average time that a tuple spends to be executed by each step.…”

Place Deduplication with Embeddings

“…Instead, we may consider direct matching of place names, which can be done using deep neural networks [3,15,19,30,40,45,49]. However, these models are known to be more suitable for longer sentences with more complex structures, rather than the short simple place names, and they require large amounts of training data, which is extremely hard to acquire for place deduplication [7,9,31]. Finally, by considering additional place attributes like location, ad-hoc models have been developed based on heuristic feature engineering [8,10,22], but their flexibility is limited to take more different place attributes, such as address and category.…”