Anomaly detection is a hard data analysis process that requires constant creation and improvement of data analysis algorithms. Using traditional clustering algorithms to analyse data streams is impossible due to processing power and memory issues. To solve this, the traditional clustering algorithm complexity needed to be reduced, which led to the creation of sequential clustering algorithms. The usual approach is two-phase clustering, which uses online phase to relax data details and complexity, and offline phase to cluster concepts created in the online phase. Detecting anomalies in a data stream is usually solved in the online phase, as it requires unreduced data. Contrarily, producing good macro-clustering is done in the offline phase, which is the reason why two-phase clustering algorithms have difficulty being equally good in anomaly detection and macro-clustering. In this paper, we propose a statistical hierarchical clustering algorithm equally suitable for both detecting anomalies and macro-clustering. The proposed algorithm is single-phased and uses statistical inference on the input data stream, resulting in statistical distributions that are constantly updated. This makes the classification adaptable, allowing agglomeration of outliers into clusters, tracking population evolution, and to be used without knowing the expected number of clusters and outliers. The proposed algorithm was tested against typical clustering algorithms, including two-phase algorithms suitable for data stream analysis. A number of typical test cases were selected, to show the universality and qualities of the proposed clustering algorithm.

No abstract

No abstract

Today organizations capture and store an abundant amount of data from their interaction with clients, internal information systems, technical systems and sensors. Data captured this way comprises many useful insights that can be discovered by various analytical procedures and methods. Discovering regular and irregular data sequences in the captured data can reveal processes performed by the organization, which can be then assessed, measured and optimized, to achieve overall better performance, lower costs, resolve congestions, find potentially fraudulent activities and similar. Besides process discovery, capturing data sequences can give additional behavioral and tendency insights for various observations in the organization, such as sales dynamic, customer behaviour and similar. The issue is that most of the captured data intertwine multiple processes, customers, cases, products in a single data log or data stream. In this article, we propose an evolving tokenized transducer (ETT), capable of learning data sequences from a multi-contextual data log or stream. The proposed ETT is a semi-supervised relational learning method that can be used as a classifier on an unknown data log or stream, revealing previously learned data sequences. The proposed ETT was tested on multiple synthetic and real-life cases and datasets, such as dataset of retail sales sequences, hospital process log involving septic patient treatment and BPI challenge 2019 dataset. Test results are successful, revealing ETT as a prominent process discovery method.

No abstract