An incremental clustering method based on the boundary profile

Abstract

Many important applications continuously generate data, such as financial transaction administration, satellite monitoring, network flow monitoring, and web information processing. The data mining results are always evolving with the newly generated data. Obviously, for the clustering task, it is better to incrementally update the new clustering results based on the old data rather than to recluster all of the data from scratch. The incremental clustering approach is an essential way to solve the problem of clustering with growing Big Data. This paper proposes a boundary-profile-based incremental clustering (BPIC) method to find arbitrarily shaped clusters with dynamically growing datasets. This method represents the existing clustering results with a collection of boundary profiles and discards the inner points of clusters rather than keep all data. It greatly saves both time and space storage costs. To identify the boundary profile, this paper presents a boundary-vector-based boundary point detection (BV-BPD) algorithm that summarizes the structure of the existing clusters. The BPIC method processes each new point in an online fashion and updates the clustering results in a batch mode. When a new point arrives, the BPIC method either immediately labels it or temporarily puts it into a bucket according to the relationship between the new data and the boundary profiles. A bucket is employed to distinguish the noise from the potential seeds of new clusters and alleviate the effects of data order. When the bucket is full, the BPIC method will cluster the data within it and update the clustering results. Thus, the BPIC method is insensitive to noise and the order of new data, which is critical for the robustness of the incremental clustering process. In the experiments, the performance of the boundary point detection algorithm BV-BPD is compared with the state-of-the-art method. The results show that the BV-BPD is better than the state-of-the-art method. Additionally, the performance of BPIC and other two incremental clustering methods are investigated in terms of clustering quality, time and space efficiency. The experimental results indicate that the BPIC method is able to get a qualified clustering result on a large dataset with higher time and space efficiency.

The unmapped chemical complexity of our diet

“…Indeed, the closer two ingredients are on the phylogenetic tree 46 , the more similar is their expected metabolic pathway structure and biochemical composition. Machine learning is ideally suited to combine the known chemical composition of chosen food ingredients over different taxonomical branches with the list of orthologous enzymes in sequenced organisms; the missing chemical information can then be elucidated by learning the appropriate distance metric 47,48 between organisms and clustering correlated groups of pathways and biochemicals 49 . Such efforts, taking full advantage of existing knowledge, could offer experimentally verifiable predictions about the missing chemicals and their concentration.…”

An incremental clustering method for anomaly detection in flight data

“…Several papers discussed the challenges in developing incremental clustering methods (Ackerman & Dasgupta, 2014). Meanwhile, many incremental or online clustering algorithms have been developed for stream data (Bao, Wang, Yang, & Wu, 2018;Gupta & Grossman, 2004;Z. Li, Lee, Li, & Han, 2010;Lin, Vlachos, Keogh, & Gunopulos, 2004).…”