Incremental all pairs similarity search for varying similarity thresholds

Abstract

All Pairs Similarity Search (AP SS) is a ubiquitous problem in many data mining applications and involves finding all pairs of records with similarity scores above a specified threshold. In this paper, we introduce the problem of Incremental All Pairs Similarity Search (IAP SS), where AP SS is performed multiple times over the same dataset by varying the similarity threshold. To the best of our knowledge, this is the first work that addresses the IAP SS problem. All existing solutions for AP SS perform redundant computations by invoking AP SS independently for each threshold value. In contrast, our solution to the IAP SS problem avoids redundant computations by storing the history of previous AP SS invocations and using index splitting. While offering obvious benefits, the computation and I/O intensive nature of the IAP SS solution raises two key research challenges: (1) to develop efficient I/O techniques to manage computation history and (2) to efficiently identify and prune redundant computations. We address these challenges through the proposed (a) history binning technique that clusters record pairs based on similarity values and performs I/O during the similarity computation, and (b) splitting of inverted index that maps each dimension to a list of records that have a non-zero projection along that dimension. As a result, we evaluate the effectiveness of our techniques by demonstrating speed-ups in the order of 2X to over 10 5 X over the state-of-the-art AP SS algorithm for four real-world large-scale datasets.

Scaling up top-K cosine similarity search

“…The above studies focus on finding binary or non-binary pairs with some specific similarity measures above some given thresholds. Recently, Awekar et al [4] studied the problem of searching candidate pairs incrementally for varying similarity thresholds. Xiao et al [35] studied the top-K set similarity joins problem for near duplicate detection, which enumerated all the "necessary" similarity thresholds in the decreasing order until the top-K set had been found.…”

“…In other words, in the initial stage, we push P [1,2] and P [2,3] (P [i, j] is the pair of item [i] and item [j], given i≤j) into the top-2 list, and compute their cosine values. Then, in the updating stage, we traverse along the diagonals (denoted by the dash-dotted line) in the sorted item-matrix to check in sequence whether P [3,4] , P [4,5] , P [5,6] , P [4,6] , P [3,5] ,…, P [1,6] can enter the top-2 list, as shown in Fig. 1.…”

“…For example, for the sorted item-matrix in Fig. 1, if P [3,4] cannot enter the top-2 list for upper(cos(P [3,4] )) ≤ minCos, then all the pairs in the upper right corner of P [3,4] will also fail to enter the list, as shown by the shadowed area in Fig. 1.…”

“…The vector of "stage 2" shows the updated values. Next, suppose P [4,7] is the third pair with upper(cos(P [4,7] )) ≤ minCos, the boundary vector will be further updated to the one of "stage 3" accordingly. Now, given the asymptotic boundary vector above, we have the following criterion to decide whether an item pair should be pruned or not.…”

“…6. end For example, in the above case, after the traversal of the third diagonal, since the only one not pruned item pair P [4,7] has cosine upper bound less than minCos, we can safely stop our searching and return the current top-2 list as the final result. And the final boundary vector, i.e., the one of "stage 3", is indicated by the shaded areas of Fig.…”