Multi-pass sorted neighborhood blocking with MapReduce

Abstract

Cloud infrastructures enable the efficient parallel execution of data-intensive tasks such as entity resolution on large datasets. We investigate challenges and possible solutions of using the MapReduce programming model for parallel entity resolution using Sorting Neighborhood blocking (SN). We propose and evaluate two efficient MapReducebased implementations for single-and multi-pass SN that either use multiple MapReduce jobs or apply a tailored data replication. We also propose an automatic data partitioning approach for multi-pass SN to achieve load balancing. Our evaluation based on real-world datasets shows the high efficiency and effectiveness of the proposed approaches.

Privacy‐preserving record linkage

“…Using the most promising approaches from PPRL blocking, massive parallel or GPU implementations should be easy by comparison, since the results of blocks are in many applications independent. Although, parallel implementations of sorted neighborhood (Kolb et al 2012) and canopy clustering (https://mahout.apache.org) have been published, practical parallel implementations of PPRL blocking for secure environments are currently not readily available.…”

Parallel Entity Resolution with Dedoop

“…It, thus, only compares entities within a window of a predetermined size w. A MR-based implementation of SN must ensure that reduce tasks can evaluate the entities in sort order and that the windows of neighboring entities are available despite the need to distribute entities among different reduce tasks. Dedoop uses the RepSN algorithm of [14] for this purpose. The map function determines the blocking key for each input entity and applies a specific range partitioning function to redistribute entities among reduce tasks so that the sort order according to the blocking key is pre- served.…”

“…All approaches utilze the BDM information determined by the analysis job. We only sketch the main ideas of the approaches in the following and refer to [14,13] for more detailed descriptions. Figure 4 illustrates the BlockSplit load balancing approach for the example from Figure 1.…”

“…We use the CiteseerX dataset 3 from [24] containing 1, 385, 532 publication records. To test Dedoop's load balancing, we apply both Sorted Neighborhood (SN) in combination with the RepSN algorithm [14] as well as Standard Blocking (SB) in combination with the PairRange algorithm [13]. For both strategies, the first three characters of the publication title form an entity's blocking key (single-pass blocking).…”

“…Previously, we have already described several of the MRbased approaches incorporated into Dedoop, in particular the MR-based Sorted Neighborhood blocking approach [14], the load balancing approaches [13], the support of learningbased matching [10], and the approaches for avoiding redundant entity comparisons [12]. Dedoop makes these approaches usable in a unified easy-to-use system that includes additional blocking and matching approaches.…”