Reverse Engineering Self-Modifying Code: Unpacker Extraction

Abstract

Abstract-An important application of binary-level reverse engineering is in reconstructing the internal logic of computer malware. Most malware code is distributed in encrypted (or "packed") form; at runtime, an unpacker routine transforms this to the original executable form of the code, which is then executed. Most of the existing work on analysis of such programs focuses on detecting unpacking and extracting the unpacked code. However, this does not shed any light on the functionality of different portions of the code so obtained, and in particular does not distinguish between code that performs unpacking and code that does not; identifying such functionality can be helpful for reverse engineering the code. This paper describes a technique for identifying and extracting the unpacker code in a self-modifying program. Our algorithm uses offline analysis of a dynamic instruction trace both to identify the point(s) where unpacking occurs and to identify and extract the corresponding unpacker code.

SoK: Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers

“…First, Debray et al [13], [29] proposed a formalization of the semantics of self-unpacking code, and modeled the concept of execution phases. In their model, a phase involves all the executed instructions written by any of the previous phases.…”

Binary-code obfuscations in prevalent packer tools

“…An alternative static approach is to extract the portion of the bootstrap code that does the unpacking and use it to create an unpacker tool. A research prototype by Coogan et al makes strides towards automating this process, and Debray and Patel built an improved prototype that incorporates dynamic analysis to help better identify and extract the code that does the unpacking [Coogan et al 2009;Debray and Patel 2010]. Though promising, this approach has not yet been shown to work on a broad sample of real-world malware.…”

Overcoming Trial Version Software Cracking Using a Hybridized Self-Modifying Technique

“…Hence, the usage of software applications has become one of the corner stone of our lives. Obviously, all these applications rely on the correct functioning of software and hardware components [6]. According to Howard and LeBlanc [13], in the 1980s, application security was achieved through secure hardware, such as ATM terminals or set-top boxes.…”

“…As a result, threats such as piracy, reverse engineering, and tampering have emerged. These threats are exacerbated by poorly protected software [5][6][7]. Therefore, it is important to have a thorough threat analysis as well as software protection schemes.…”

“…These applications have been exposed to attacks over the years. Nevertheless, they still withstand the major problems caused by software threats such as reverse engineering, tampering, cracking or piracy [6] [8].…”