Anomalous Payload-Based Worm Detection and Signature Generation

Abstract. Network-level emulation has recently been proposed as a method for the accurate detection of previously unknown polymorphic code injection attacks. In this paper, we extend network-level emulation along two lines. First, we present an improved execution behavior heuristic that enables the detection of a certain class of non-self-contained polymorphic shellcodes that are currently missed by existing emulation-based approaches. Second, we present two generic algorithmic optimizations that improve the runtime performance of the detector. We have implemented a prototype of the proposed technique and evaluated it using off-the-shelf non-self-contained polymorphic shellcode engines and benign data. The detector achieves a modest processing throughput, which however is enough for decent runtime performance on actual deployments, while it has not produced any false positives. Finally, we report attack activity statistics from a seven-month deployment of our prototype in a production network, which demonstrate the effectiveness and practicality of our approach.

Section: Related Workmentioning

confidence: 99%

Emulation-Based Detection of Non-self-contained Polymorphic Shellcode

Polychronakis

Anagnostakis

Markatos

“…Anomaly detectors designed as NIDS include the Anagram [23] and PAYL [22,24] anomaly detection systems. Anagram stores the n-grams, a contiguous sequence of n characters of a given string, in a Bloom filter [2].…”

Section: Related Workmentioning

confidence: 99%

transAD: An Anomaly Detection Network Intrusion Sensor for the Web

Hiremagalore

Barbará

Fleck

et al. 2014

Abstract. Content-based Anomaly Detection (AD) techniques are regarded as a promising mechanism to detect 'zero-day' attacks. AD sensors have also been shown to perform better than signature-based systems in detecting polymorphic attacks. However, the False Positive Rates (FPRs) produced by current AD sensors have been a cause of concern. In this paper, we introduce and evaluate transAD, a system of network traffic inspection AD sensors that are based on Transductive Confidence Machines (TCM). Existing TCM-based implementations have very high FPRs when used as NIDS.Our approach leverages an unsupervised machine-learning algorithm to identify anomalous packets and thus, unlike most AD sensors, transAD does not require manually labeled data. Moreover, transAD uses an ensemble of TCM sensors to achieve better detection rates and lower FPRs than single sensor implementations. Therefore, transAD presents a hardened defense against poisoning attacks.We evaluated our prototype implementation using two real-world data sets collected from a public university's network. TransAD processed approximately 1.1 million packets containing real attacks. To compute the ground truth, we manually labeled 18,500 alerts. In the course of scanning millions of packets, our sensor's low FPR would significantly reduce the number of false alerts that need to be inspected by an operator, while maintaining a high detection rate.

“…We first look at detectors that examine the contents of network traffic, including AutoGraph [13], EarlyBird [14], PAYL [15], Anagram [16], and LESG [17]. Each of these detection mechanisms share a similar limitation that leads us to exclude them from our comparison: they are unable to monitor encrypted traffic.…”

Section: Detector Selectionmentioning

confidence: 99%

Behavior-Based Worm Detectors Compared

Stafford¹,

Zhang²

2010

Abstract. Many worm detectors have been proposed and are being deployed, but the literature does not clearly indicate which one is the best. New worms such as IKEE.B (also known as the iPhone worm) continue to present new challenges to worm detection, further raising the question of how effective our worm defenses are. In this paper, we identify six behavior-based worm detection algorithms as being potentially capable of detecting worms such as IKEE.B, and then measure their performance across a variety of environments and worm scanning behaviors, using common parameters and metrics. We show that the underlying network trace used to evaluate worm detectors significantly impacts their measured performance. An environment containing substantial gaming and file sharing traffic can cause the detectors to perform poorly. No single detector stands out as suitable for all situations. For instance, connection failure monitoring is the most effective algorithm in many environments, but it fails badly at detecting topologically aware worms.