Language models for detection of unknown attacks in network traffic

Abstract

In this paper we propose a method for network intrusion detection based on language models. Our method proceeds by extracting language features such as n-grams and words from connection payloads and applying unsupervised anomaly detection without prior learning phase or presence of labeled data. The essential part of this procedure is linear-time computation of similarity measures between language models of connection payloads. Particular patterns in these models decisive for discrimination of attacks and normal data can be traced back to attack semantics and utilized for automatic generation of attack signatures. Results of experiments conducted on two datasets of network traffic demonstrate the importance of higher-order n-grams and variable-length language models for detection of unknown network attacks. An implementation of our system achieved detection accuracy of over 80% with no false positives on instances of recent remote-to-local attacks in HTTP, FTP and SMTP traffic

Tracking Users on the Internet with Behavioral Patterns: Evaluation of Its Practical Feasibility

“…They have also been used for traffic analysis problems with success [21]. This technique combines the hostnames of n adjacent events to obtain a composite hostname.…”

Auto-Sign: an automatic signature generator for high-speed malware filtering devices

“…Considering the global variety of development platforms and the mobility of threats facilitated by the Internet, ensuring the external validity of this study relies substantially on reaching a critical mass of CFL files which represents abundant development platforms. Furthermore, it often does not suffice for a signature to be available-deployed signatures must be managed, distributed and kept upto-date by security administrators [16].…”

NIDS based on payload word frequencies and anomaly of transitions

“…Based on the results from [24], and [26], and our own experiments, a set of 20 boundary symbols that provides the highest percentage of meaningful words for HTTP is defined. These 20 delimiters are:…”

“…The frequency analysis of multiple consecutive bytes, ngrams, in payload has been proposed in [25]. The use of payload words, seen as consecutive bytes separated by delimiters, to make a language model was considered in [26]. Authors of [27] take a different view and use HTTP GET request parameters and their values as the starting point for the model: the length, character positions, the structure, the values and existence of parameters are considered.…”