Defeating Opaque Predicates Statically through Machine Learning and Binary Analysis

Abstract

We present a new approach that bridges binary analysis techniques with machine learning classification for the purpose of providing a static and generic evaluation technique for opaque predicates, regardless of their constructions. We use this technique as a static automated deobfuscation tool to remove the opaque predicates introduced by obfuscation mechanisms. According to our experimental results, our models have up to 98% accuracy at detecting and deobfuscating state-of-the-art opaque predicates patterns. By contrast, the leading edge deobfuscation methods based on symbolic execution show less accuracy mostly due to the SMT solvers constraints and the lack of scalability of dynamic symbolic analyses. Our approach underlines the efficiency of hybrid symbolic analysis and machine learning techniques for a static and generic deobfuscation methodology.

A Heuristic Approach to Detect Opaque Predicates that Disrupt Static Disassembly

“…For example, Backward-Bounded DSE [3] can detect 100% of the opaque predicates inserted by the OLLVM obfuscator [18]. However, dynamic symbolic execution-based approaches' detection accuracy decreases when encountering opaque predicates of varied constructions [37]. This is because their detection approaches are based on determining if a conditional branch contains an invariant expression.…”

“…Depending on how the opaque predicate is constructed, it can be non-trivial or even undecidable to identify the invariant expression. Tofighi-Shirazi et al [37] recently introduce a detection approach using machine learning based on a decision-tree classification model. Although it can detect opaque predicates of varied constructions, the performance of Tofighi-Shirazi's approach suffers when detecting an opaque predicate whose construction is not previously encountered in its training data.…”

Fine-grained static detection of obfuscation transforms using ensemble-learning and semantic reasoning

“…is language is part of the symbolic execution engine that we use for the implementation of our methodology as IDA Pro plug-in. Additionally, the normalized Miasm2 intermediate language has also been successful for the application of machine learning techniques in order to deobfuscate opaque predicates [61]. Listing 1 illustrates the symbolic state S of the rst basic-block of the function quick-sort, which is illustrated in Figure 1.…”

“…As seen in [61], semantic reasoning and machine learning provides promising results for deobfuscation methodology. e evaluations shown in this paper illustrate that our model does not depend on the code functionality.…”

“…Sun et al [59], and more recently Biondi et al [7], aim at detecting and identifying packers using machine learning techniques. To ghi-Shirazi et al [61] propose a deobfuscation methodology for invariant opaque predicates based on machine learning techniques.…”