Efficient and effective KNN sequence search with approximate n-grams

Abstract

In this paper, we address the problem of finding k-nearest neighbors (KNN) in sequence databases using the edit distance. Unlike most existing works using short and exact ngram matchings together with a filter-and-refine framework for KNN sequence search, our new approach allows us to use longer but approximate n-gram matchings as a basis of KN-N candidates pruning. Based on this new idea, we devise a pipeline framework over a two-level index for searching KNN in the sequence database. By coupling this framework together with several efficient filtering strategies, i.e. the frequency queue and the well-known Combined Algorithm (CA), our proposal brings various enticing advantages over existing works, including 1) huge reduction on false positive candidates to avoid large overheads on candidate verifications; 2) progressive result update and early termination; and 3) good extensibility to parallel computation. We conduct extensive experiments on three real datasets to verify the superiority of the proposed framework.

A Generic Inverted Index Framework for Similarity Search on the GPU

“…As we can see from Figure 1, query Q 1 to retrieve the tuples with conditions [1,2]), (B, [1,1]), (C, [2,3]…”

“…Finally the output of the match-count model is the sum of the integers M C(Q, O) = ri∈Q C(r i , O). For example, in Figure 1, for Q 1 and O 1 we have C((A, [1,2]), O 1 ) = 1, C((B, [1,1]), O 1 ) = 0 and C((C, [2,3]), O 1 ) = 0, then we have M C(Q 1 , O 1 ) = 1 + 0 + 0 = 1.…”

“…2. How to organize data structures as inverted indexes has been extensively investigated by previous literature [2], [10], [11] and it is beyond the scope of this paper. 1) Transformed by LSH.…”

“…Initially we have AT = 1, ZA = [0, 0, 0], BC = {O 1 : 0, O 2 : 0, O 3 : 0} and HT = ∅. For easy explanation, we assume the postings lists matched by Q 1 are scanned with the order of (A, [1,2]), (B, [1,1]) and (C, [2,3]). (On the GPU they are processed with multiple blocks in parallel with random order.…”

“…For example, image matching is often done by extracting hundreds of high dimensional SIFT (scale-invariant feature transform) features and matching them against SIFT features in the database. Parallelization for similarity search is required for high performance on modern hardware architectures [1], [2], [3], [4].…”