Norm Adjusted Proximity Graph for Fast Inner Product Retrieval

Tan, Shulong; Xu, Zhaozhuo; Zhao, Weijie; Fei, Hongliang; Li, Ping

doi:10.1145/3447548.3467412

Cited by 15 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sourmash or skani calculated ANI) are also not metric (58)(59)(60). To solve this "metric" problem, a norm adjusted proximity graph (NAPG) was proposed based on inner product and it shows improvements in terms of both speed and recall using non-metric distances (61). This could be another direction for further improving the speed and recall of GSearch.…”

Section: Discussionmentioning

confidence: 99%

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

Zhao

Pierre-both

Rodriguez-R

et al. 2022

Preprint

View full text Add to dashboard Cite

Genome search and/or classification is a key step in microbiome studies and has become more challenging due to the increasing number of available (reference) genomes in recent years and the fact that traditional methods do not scale well with larger databases. By combining a kmer hashing-based genomic distance metric (Probminhash) with a graph based nearest neighbor search (NNS) algorithm (called Hierarchical Navigable Small World Graphs), we developed a new program, GSearch, that is at least ten times faster than alternative tools for the same purposes while maintaining high accuracy. GSearch can identify/classify eight thousand query genomes against all available microbial and viral genomic species within several minutes on a personal laptop, using only ∼6GB of memory. Further, GSearch can scale well with millions of database genomes based on a database splitting strategy. Therefore, GSearch solves a major bottleneck in current and future microbiome studies that require genome search and/or classification.

show abstract

Section: Discussionmentioning

confidence: 99%

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

Zhao

Pierre-both

Rodriguez-R

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The challenge is then to design such an approximation algorithm that returns a search result with high accuracy in time sub-linear to n. This is not trivial, and existing approximate MIPS algorithms cannot overcome this challenge. This is because state-of-the-art approximate MIPS algorithms have no theoretical time [13], [29], [36], [38] or have O(n log n) > O(n) time [17], [28].…”

Section: B Challengementioning

confidence: 99%

“…Proximity graph algorithms [23], [29], [38] are based on a greedy algorithm. In the pre-processing (offline) phase, these algorithms build a proximity graph, where each vertex of this proximity graph corresponds to a vector in X.…”

Section: B Approximate Mipsmentioning

confidence: 99%

Categorical Diversity-Aware Inner Product Search

Hirata

Amagata

Fujita³

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The problem of maximum inner product search (MIPS) is one of the most important components in machine learning systems. However, this problem does not care about diversity, although result diversification can improve user satisfaction. This paper hence considers a new problem, namely the categorical diversity-aware IPS problem, in which users can select preferable categories. Exactly solving this problem needs O(n) time, where n is the number of vectors, and is not efficient for large n. We hence propose an approximation algorithm that has a probabilistic success guarantee and runs in sub-linear time to n. We conduct extensive experiments on real datasets, and the results demonstrate the superior performance of our algorithm to that of a baseline using an existing MIPS technique.

show abstract

“…This naturally gives rise to the Maximum Inner Product Search (MIPS) problem, which finds the vector in a set of n item vectors P ⊂ R d that has the largest inner product with a query (user) vector q ∈ R d , i.e., p * = arg max p∈P ⟨p, q⟩, as well as its extension kMIPS that finds k (k > 1) vectors with the largest inner products for recommending items to users. Due to its prominence in recommender systems, the kMIPS problem has attracted significant research interests, and numerous methods have been proposed to improve the search performance (Ram and Gray 2012;Koenigstein, Ram, and Shavitt 2012;Keivani, Sinha, and Ram 2018;Teflioudi and Gemulla 2017;Li et al 2017;Abuzaid et al 2019;Shrivastava and Li 2014;Neyshabur and Srebro 2015;Shrivastava and Li 2015;Huang et al 2018;Yan et al 2018;Ballard et al 2015;Yu et al 2017;Ding, Yu, and Hsieh 2019;Lorenzen and Pham 2020;Pham 2021;Shen et al 2015;Guo et al 2016;Dai et al 2020;Xiang et al 2021;Morozov and Babenko 2018;Tan et al 2019;Zhou et al 2019;Liu et al 2020;Tan et al 2021).…”

Section: Introductionmentioning

confidence: 99%

SAH: Shifting-Aware Asymmetric Hashing for Reverse k Maximum Inner Product Search

Huang

Wang

Tung

2023

AAAI

View full text Add to dashboard Cite

This paper investigates a new yet challenging problem called Reverse k-Maximum Inner Product Search (RkMIPS). Given a query (item) vector, a set of item vectors, and a set of user vectors, the problem of RkMIPS aims to find a set of user vectors whose inner products with the query vector are one of the k largest among the query and item vectors. We propose the first subquadratic-time algorithm, i.e., Shifting-aware Asymmetric Hashing (SAH), to tackle the RkMIPS problem. To speed up the Maximum Inner Product Search (MIPS) on item vectors, we design a shifting-invariant asymmetric transformation and develop a novel sublinear-time Shifting-Aware Asymmetric Locality Sensitive Hashing (SA-ALSH) scheme. Furthermore, we devise a new blocking strategy based on the Cone-Tree to effectively prune user vectors (in a batch). We prove that SAH achieves a theoretical guarantee for solving the RMIPS problem. Experimental results on five real-world datasets show that SAH runs 4~8x faster than the state-of-the-art methods for RkMIPS while achieving F1-scores of over 90%. The code is available at https://github.com/HuangQiang/SAH.

show abstract

Norm Adjusted Proximity Graph for Fast Inner Product Retrieval

Cited by 15 publications

References 40 publications

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

GSearch: Ultra-Fast and Scalable Microbial Genome Search by Combining K-mer Hashing with Hierarchical Navigable Small World Graphs

Categorical Diversity-Aware Inner Product Search

SAH: Shifting-Aware Asymmetric Hashing for Reverse k Maximum Inner Product Search

Contact Info

Product

Resources

About