Leveraging set relations in exact set similarity join

Abstract

Exact set similarity join, which finds all the similar set pairs from two collections of sets, is a fundamental problem with a wide range of applications. The existing solutions for set similarity join follow a filtering-verification framework, which generates a list of candidate pairs through scanning indexes in the filtering phase, and reports those similar pairs in the verification phase. Though much research has been conducted on this problem, set relations, which we find out is quite effective on improving the algorithm efficiency through computational cost sharing, have never been studied. Therefore, in this paper, instead of considering each set individually, we explore the set relations in different levels to reduce the overall computational costs. First, it has been shown that most of the computational time is spent on the filtering phase, which can be quadratic to the number of sets in the worst case for the existing solutions. Thus we explore index-level set relations to reduce the filtering cost to be linear to the size of the input while keeping the same filtering power. We achieve this by grouping related sets into blocks in the index and skipping useless index probes in joins. Second, we explore answer-level set relations to further improve the algorithm based on the intuition that if two sets are similar, their answers may have a large overlap. We derive an algorithm which incrementally generates the answer of one set from an already computed answer of another similar set rather than compute the answer from scratch to reduce the computational cost. Finally, we conduct extensive performance studies using 21 real datasets with various data properties from a wide range of domains. The experimental results demonstrate that our algorithm outperforms all the existing algorithms across all datasets and can achieve more than an order of magnitude speedup against the stateof-the-art algorithms.

Blocking and Filtering Techniques for Entity Resolution

“…The discrepancy between the two studies was attributed to the more efficient process that performed the verification step in [84]; reducing the cost of verification means that employing a complex filter to reduce the number of candidate pairs may not pay off. Another interesting finding is that leveraging set relations can improve the performance of the filtering algorithms; SKJ was found to consistently outperform PPJoin, PPJoin+, AdaptJoin and PTJ in [159].…”

Data structure set-trie for storing and querying sets: Theoretical and empirical analysis

“…Firstly, given the collection of objects and a query object, the similarity search retrieves all objects similar to a query object [20][21][22]. Secondly, given two sets of objects, the exact set similarity join finds all pairs of similar objects [23][24][25]. The operations are used for data cleaning, information integration, entity detection, near duplicate detection and personalized recomendation.…”

“…Sets are used in Internet applications for the representation of the properties of objects, sparse vector data, text files, itemsets, tags and the neighbours in graphs. Two types of queries attracted research attention: the similarity search [20,22] and the set similarity join [23][24][25]41] We present each of the above stated related areas in more detail in the following Sections 2.1-2.5.…”

Josie

“…Instead of using a fixed-length prefix as done by Bayardo et al, Wang et al [29] designed an adaptive prefix filter framework. Wang et al [30] further improved Xiao et al 's work by leveraging the relation between sets.…”