Exploration and Visualization Approach for Outlier Detection on Log Files

Abstract

Abstract. We propose a novel clustering-based outlier detection approach for data streams. To deal with the data streams, we propose splitting the data into several windows. In each window, the data is divided into subspaces. First, a clustering algorithm is applied on one subspace. Based on the existing relations between the different subspaces, the obtained clusters can represent partitions on another subspace. Then the same clustering algorithm is applied on each partition separately in this second subspace. The process can be iterated on n subspaces. We perform tests on firewall logs data sets, we choose to test our approach with two subspaces and to visualize the results with neighborhood graphs in each window. A comparison is provided between the obtained results and the MCOD algorithm results. We can identify visually the outliers events and observe the evolution of the stream.

Large dataset summarization with automatic parameter optimization and parallel processing for local outlier detection

“…Regardless of the exact metric used for detection, local outliers are invariably contextual in the sense that their outlier-ness is defined based on their neighboring data, and different outliers may feature varying neighbors for defining their outlier-ness. This is evidenced by the viewpoint presented by other works [6][7][8][9] that outliers are important patterns with strong contextual implication. These neighboring data of the outliers are called their reference data.…”

“…5,[23][24][25][26][27][28] There has also been some research work on interactive outlier detection, which introduces user-friendly interactive and visualization features to assist outlier detection. 6,[29][30][31][32][33] A comprehensive survey of recent outlier detection techniques has been conducted in other works. [34][35][36] It is very important to point out that our proposed method is fundamentally different from all the existing outlier detection in terms of its basic paradigm.…”

“…Because of the close relationships between data clusters and outliers, clustering analysis can also be performed to assist the detection of outliers by defining outliers as data that do not lie in or located far apart from any clusters . There has also been some research work on interactive outlier detection, which introduces user‐friendly interactive and visualization features to assist outlier detection . A comprehensive survey of recent outlier detection techniques has been conducted in other works …”