Set similarity joins on mapreduce

Abstract

Set similarity joins, which compute pairs of similar sets, constitute an important operator primitive in a variety of applications, including applications that must process large amounts of data. To handle these data volumes, several distributed set similarity join algorithms have been proposed. Unfortunately, little is known about the relative performance, strengths and weaknesses of these techniques. Previous comparisons are limited to a small subset of relevant algorithms, and the large differences in the various test setups make it hard to draw overall conclusions. In this paper we survey ten recent, distributed set similarity join algorithms, all based on the MapReduce paradigm. We empirically compare the algorithms in a uniform test environment on twelve datasets that expose different characteristics and represent a broad range of applications. Our experiments yield a surprising result: All algorithms in our test fail to scale for at least one dataset and are sensitive to long sets, frequent set elements, low similarity thresholds, or a combination thereof. Interestingly, some algorithms even fail to handle the small datasets that can easily be processed in a non-distributed setting. Our analytic investigation of the algorithms pinpoints the reasons for the poor performance and targeted experiments confirm our analytic findings. Based on our investigation, we suggest directions for future research in the area.

Blocking and Filtering Techniques for Entity Resolution

“…These parallel algorithms for set similarity joins are compared in a recent experimental study [46]. VernicaJoin achieved the best performance in most experiments, followed by MRGroupJoin, FS-Join and MGJoin.…”

“…Still, as presented in Section 5, relatively few of the existing similarity join techniques are designed for these cases. Moreover, as shown in the experimental survey of [46], scalability remains an open challenge for string and set similarity joins. Finally, there is a need for extensible, open-source ER tools that incorporate the majority of established Blocking and Filtering methods and apply seamlessly to structured, semi-structured and unstructured data [52].…”

“…Recent surveys on string and set similarity joins also exist but have a much more limited scope. They focus exclusively on either centralized [60,84,167] or distributed approaches [46], with the purpose of experimental comparison, and without covering approximate techniques or methods that allow for more relaxed matching criteria. Also, none of these surveys considers similarity joins in the broader context of ER.…”

Customizable and scalable fuzzy join for big data

“…For the scalability experiments, we generate datasets with scale factors: 1×, 2×, 3×, 4× and 5× the number of rows in the original dataset. We follow the methodology similar to that used in Vernica et al [39] and Fier et al [19], which preserves the original set of distinct tokens, their distribution and record lengths; but increases the number of records by replacing a token with a neighboring token in the sorted token frequency order.…”

“…Therefore, scale-out approaches including [18,39,8,33,27,31,13,16,17,30] have been developed on MapReduce [15] engines such as Hadoop. A recent experimental study by Fier et al [19] compared several scale-out techniques. Based on these results, the stateof-the-art scale-out fuzzy join techniques are lacking in two fundamental ways.…”