Metadata recovery from obfuscated programs using machine learning

Abstract

Obfuscation is a mechanism used to hinder reverse engineering of programs. To cope with the large number of obfuscated programs, especially malware, reverse engineers automate the process of deobfuscation i.e. extracting information from obfuscated programs. Deobfuscation techniques target specific obfuscation transformations, which requires reverse engineers to manually identify the transformations used by a program, in what is known as metadata recovery attack. In this paper, we present Oedipus, a Python framework that uses machine learning classifiers viz., decision trees and naive Bayes, to automate metadata recovery attacks against obfuscated programs. We evaluated Oedipus' performance using two datasets totaling 1960 unobfuscated C programs, which were used to generate 11.075 programs obfuscated using 30 configurations of 6 different obfuscation transformations. Our results empirically show the feasibility of using machine learning to implement the metadata recovery attacks with classification accuracies of 100% in some cases.

Fine-grained static detection of obfuscation transforms using ensemble-learning and semantic reasoning

“…One is the traditional k-folds cross-validation with scores in black colored font. e other is made with the functionality-based cross-validation approach in red colored font, used in Salem et al related work [53]. Besides, we use as a traditional single-model random-forest algorithm throughout all our studies.…”

“…dispatch-methods for control-ow a ening or code virtualization). is approach is previously known as metadata recovery a acks [53].…”

“…From the variety of obfuscation techniques, as well as deobfuscation methodologies, the ability to e ciently detect the so ware protections used is at a prime. To that end, the recent work of Salem et al [53] focuses on the detection of obfuscation transformations. eir goal is to facilitate the selection and application of adequate deobfuscation techniques.…”

“…Current limitations. Existing detection technique for code obfuscation [53] based on machine learning techniques comes with the following limitations:…”

“…As previously discussed in [53], metadata recovery a acks are usually manual tasks, therefore a potential bo leneck in the reverse engineering process. Our methodology, which could be plugged-in a disassembler framework, provides all applied transformation and construction and allows reverse-engineers to setup automated deobfuscation strategies.…”