Emulation-Based Detection of Non-self-contained Polymorphic Shellcode

Abstract

Abstract. Network-level emulation has recently been proposed as a method for the accurate detection of previously unknown polymorphic code injection attacks. In this paper, we extend network-level emulation along two lines. First, we present an improved execution behavior heuristic that enables the detection of a certain class of non-self-contained polymorphic shellcodes that are currently missed by existing emulation-based approaches. Second, we present two generic algorithmic optimizations that improve the runtime performance of the detector. We have implemented a prototype of the proposed technique and evaluated it using off-the-shelf non-self-contained polymorphic shellcode engines and benign data. The detector achieves a modest processing throughput, which however is enough for decent runtime performance on actual deployments, while it has not produced any false positives. Finally, we report attack activity statistics from a seven-month deployment of our prototype in a production network, which demonstrate the effectiveness and practicality of our approach.

Efficient Shellcode Detection on Commodity Hardware

“…In other words, our system tries each byte of the network stream as a potential entry point of shellcode. Similar to the dynamic approaches [1], [6], [9], we refer to a complete execution from each offset of the stream as an execution chain. Compared with the dynamic approaches that require intercepting and tracking each instruction, our approach allows instruction sequences being executed directly on the CPU, and only trap a small number of instructions selectively by utilizing hardware paging mechanisms.…”

“…When we utilize 2 CPU cores for two different detection processes, our system can achieve 73.8 Mbps network throughput without packet loss. Compared with the emulation-based methods [1], [6], [9], the network throughput is greatly improved.…”

“…In the past few years, many detection approaches [1], [3], [4], [6], [9] have been proposed. Basically, these methods can be divided into two categories: static analysis [3], [4] and dynamic analysis [1], [6], [9].…”

“…Basically, these methods can be divided into two categories: static analysis [3], [4] and dynamic analysis [1], [6], [9]. The core idea of static analysis is to disassemble the network stream and then analyze the code-level patterns that could be signatures obtained from existing shellcode.…”

Defending Browsers against Drive-by Downloads: Mitigating Heap-Spraying Code Injection Attacks

“…The class of non-self-contained shellcode, however, contains code that reaches its goal without showing such behavior. In [25], the authors extend their detection techniques to also identify this class of attacks. While network-traffic-based techniques are useful, they typically cannot be used to detect drive-by downloads.…”