DotHash: Estimating Set Similarity Metrics for Link Prediction and Document Deduplication

Nunes, Igor; Heddes, Mike; Vergés, Pere; Abraham, Dilip; Veidenbaum, Alexander V.; Nicolau, Alexandru; Givargis, Tony

doi:10.1145/3580305.3599314

Cited by 2 publications

(5 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we present HyperGen: a genome sketching tool based on hyperdimensional computing (HDC) (Nunes et al ., 2023; Kanerva, 2009) that improves accuracy, runtime performance, and memory efficiency for large-scale genomic analysis. HyperGen inherits the advantages of both FracMinHash-based sketching (Hera et al ., 2023) and DotHash (Nunes et al ., 2023). HyperGen first samples the k -mer set using FracMinHash.…”

Section: Discussionmentioning

confidence: 99%

“…A recent work (Nunes et al ., 2023) demonstrates that the speed and memory efficiency of Jaccard similarity approximation can be improved by using the DotHash based on Random Indexing (Sahlgren, 2005). The key step to compute Jaccard similarity in Eq.…”

Section: Methodsmentioning

confidence: 99%

“…Each set is expressed as an aggregation vector a ∈ ℝ D such that where the aggregation vector sums all the elements’ vectors generated by the mapping function ϕ ( x ). One necessary constraint for function ϕ ( x ) is: the generated vectors should satisfy the quasi-orthogonal properties: DotHash (Nunes et al ., 2023) uses a pseudo random number generator (RNG) as the mapping function ϕ ( x ) because the RNG can generate uniform and quasi-orthogonal vectors in an efficient manner.…”

Section: Methodsmentioning

confidence: 99%

“…We take advantage of recent DotHash (Nunes et al ., 2023), which shows superior space and computation efficiency for the Jaccard similarity estimation. DotHash leverages the HDC-based random indexing (Sahlgren, 2005; Kanerva et al ., 2000) and is originally designed for fast set intersection estimation.…”

Section: Introductionmentioning

confidence: 99%

“…In this work, we propose HyperGen, a novel tool for efficient genome sketching and ANI estimation. HyperGen exploits the emerging HDC (similar to DotHash (Nunes et al ., 2023)) to boost genomic ANI calculation. Specifically, we optimize DotHash’s efficiency by converting the sketch generation process into a low bit-width integer domain.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

HyperGen: Compact and Efficient Genome Sketching using Hyperdimensional Vectors

Xu,

Hsu,

Moshiri

et al. 2024

Preprint

View full text Add to dashboard Cite

Motivation: Genomic distance estimation is a critical workload since exact computation for whole-genome similarity metrics such as Average Nucleotide Identity (ANI) incurs exhibitive runtime overhead. Genome sketching is a fast and memory-efficient solution to estimate ANI similarity by distilling representative kmers from the original sequences. In this work, we present HyperGen that improves accuracy, runtime performance, and memory efficiency for large-scale ANI estimation. Unlike existing genome sketching algorithms that convert large genome files into discrete k-mer hashes, HyperGen leverages the emerging hyperdimensional computing (HDC) to encode genomes into quasi-orthogonal vectors (Hypervector, HV) in high-dimensional space. HV is compact and can preserve more information, allowing for accurate ANI estimation while reducing required sketch sizes. In particular, the HV sketch representation in HyperGen allows efficient ANI estimation using vector multiplication, which naturally benefits from highly optimized general matrix multiply (GEMM) routines. As a result, HyperGen enables the efficient sketching and ANI estimation for massive genome collections. Results: We evaluate HyperGen's sketching and database search performance using several genome datasets at various scales. HyperGen is able to achieve comparable or superior ANI estimation error and linearity compared to other sketch-based counterparts. Compared to other sketch-based baselines, HyperGen achieves comparable sketching speed and up to 4.3x faster in database search. Meanwhile, HyperGen's sketch size is on average 1.8-2.7x smaller.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%