Efficient kNN Join over Dynamic High-Dimensional Data

Ukey, Nimish; Yang, Zhengyi; Zhang, Guangjian; Liu, Boge; Li, Binghao; Zhang, Wenjie

doi:10.1007/978-3-031-15512-3_5

Cited by 7 publications

(21 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is not any standard dimension for high dimensionality. For example, work with 12D [ 117 ], 30D [ 112 ], 32D [ 109 ] and 500D [ 129 ] datasets is regarded as a high-dimensional dataset. Therefore, considering the scenario in mind, we divide the dimensionality into three different ranges.…”

Section: Classification Of Knn Queriesmentioning

confidence: 99%

“…Data Partitioning algorithms work better for creating indexes because of their adaptability to data distributions. For example, the performance of techniques such as

-tree [ 74 ],

-tree [ 74 , 118 ], BP [ 114 ], kNNJoin

[ 76 ], HDR-tree [ 83 ], EkNNJ [ 129 ], etc., is greatly improved with a data-based partitioning strategy. This makes retrieval much faster in real-world environments.…”

Section: Classification Of Knn Queriesmentioning

confidence: 99%

“…However, as per the analysis, it has been observed that fundamental operations like batch updates and efficient deletion operations are not supported by the current existing kNN Join algorithms. Therefore, the authors of [ 129 ] present a novel approach for kNN Join over high-dimensional datasets where they focused on batch operations, lazy updates and optimised deletions. The results of an experiment show that the proposed approaches outperform existing techniques such as HDR-tree and naive RkNN.…”

Section: Knn Join Approachesmentioning

confidence: 99%

“…In ref. [ 129 ], authors address the problem of existing kNN Join techniques, i.e., the lack of support for deletions and batch updates, with the help of new efficient deletion, lazy update and batch update algorithms.…”

Section: Comparative Studymentioning

confidence: 99%

See 3 more Smart Citations

Survey on Exact kNN Queries over High-Dimensional Data Space

Ukey

Yang

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

k nearest neighbours (kNN) queries are fundamental in many applications, ranging from data mining, recommendation system and Internet of Things, to Industry 4.0 framework applications. In mining, specifically, it can be used for the classification of human activities, iterative closest point registration and pattern recognition and has also been helpful for intrusion detection systems and fault detection. Due to the importance of kNN queries, many algorithms have been proposed in the literature, for both static and dynamic data. In this paper, we focus on exact kNN queries and present a comprehensive survey of exact kNN queries. In particular, we study two fundamental types of exact kNN queries: the kNN Search queries and the kNN Join queries. Our survey focuses on exact approaches over high-dimensional data space, which covers 20 kNN Search methods and 9 kNN Join methods. To the best of our knowledge, this is the first work of a comprehensive survey of exact kNN queries over high-dimensional datasets. We specifically categorise the algorithms based on indexing strategies, data and space partitioning strategies, clustering techniques and the computing paradigm. We provide useful insights for the evolution of approaches based on the various categorisation factors, as well as the possibility of further expansion. Lastly, we discuss some open challenges and future research directions.

show abstract

Section: Classification Of Knn Queriesmentioning

confidence: 99%

“…Data Partitioning algorithms work better for creating indexes because of their adaptability to data distributions. For example, the performance of techniques such as

-tree [ 74 ],

-tree [ 74 , 118 ], BP [ 114 ], kNNJoin

[ 76 ], HDR-tree [ 83 ], EkNNJ [ 129 ], etc., is greatly improved with a data-based partitioning strategy. This makes retrieval much faster in real-world environments.…”

Section: Classification Of Knn Queriesmentioning

confidence: 99%

Section: Knn Join Approachesmentioning

confidence: 99%

Section: Comparative Studymentioning

confidence: 99%

See 2 more Smart Citations

Survey on Exact kNN Queries over High-Dimensional Data Space

Ukey

Yang

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…Considering the fast growth of high-velocity streaming data, these approaches significantly limit the performance of dynamic kNN join on large datasets. To address these issues, we came up with lazy updates, batch updates, and optimised deletions in our previous work [16]. We design a lazy update mechanism.…”

Section: Introductionmentioning

confidence: 99%

Efficient Continuous kNN Join over Dynamic High-dimensional Data

Ukey

Zhang

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

Given a user dataset U and an object dataset I, a kNN join query in high-dimensional space returns the k nearest neighbors of each object in dataset U from the object dataset I. The kNN join is a basic and necessary operation in many applications, such as databases, data mining, computer vision, multi-media, machine learning, recommendation systems, and many more. In the real world, datasets frequently update dynamically as objects are added or removed. In this paper, we propose novel methods of continuous kNN join over dynamic high-dimensional data. We firstly propose the HDR+ Tree which supports more efficient insertion, deletion, and batch update. Further observed that the existing methods rely on globally correlated datasets for effective dimen-sionality reduction, we then propose the HDR Forest. It clusters the dataset and constructs multiple HDR Trees to capture local correlations among the data. As a result, our HDR Forest is able to process non-globally correlated dataset efficiently. Two novel optimisations are applied to the proposed HDR Forest, including the precomputation of the PCA states of data items and pruning-based kNN recomputation during item deletion. For the completeness of the work, we also present the proof of computing distances in reduced dimensions of PCA in HDR Tree. Extensive experiments on real-world datasets show that the proposed methods and optimisations outperform the baseline algorithms of naive RkNN join and HDR Tree.

show abstract