Anomaly Detection from Network Logs Using Diffusion Maps

Abstract

Abstract. The goal of this study is to detect anomalous queries from network logs using a dimensionality reduction framework. The fequencies of 2-grams in queries are extracted to a feature matrix. Dimensionality reduction is done by applying diffusion maps. The method is adaptive and thus does not need training before analysis. We tested the method with data that includes normal and intrusive traffic to a web server. This approach finds all intrusions in the dataset.

An Efficient Network Log Anomaly Detection System Using Random Projection Dimensionality Reduction

“…The authors have already been involved in developing several network anomaly detection systems [17], [18], [19]. These papers mainly focus on diffusion map (DM) methodology for dimensionality reduction.…”

Combining conjunctive rule extraction with diffusion maps for network intrusion detection

“…A distributed environment has been proposed where intelligent agents analyze the network connections using data mining with association rule mining [13]. Moreover, in our previous work we have researched intrusion detection using dimensionality reduction and clustering to find anomalies from network traffic [14], [15].…”

“…Our study uses 2-gram features generated from the network logs. This approach produces a rather sparse feature matrix [14]. The rule extraction algorithm works with symbolic conjunctive rules.…”

Growing Hierarchical Self-organizing Maps and Statistical Distribution Models for Online Detection of Web Attacks

“…Let us call this dataset "A". This case has been presented in an earlier publication [33]. The log files contain mostly normal traffic, but they also include anomalities and actual intrusions.…”