Search-Based Local Black-Box Deobfuscation: Understand, Improve and Mitigate

Abstract

Code obfuscation aims at protecting Intellectual Property and other secrets embedded into software from being retrieved. Recent works leverage advances in artificial intelligence (AI) with the hope of getting blackbox deobfuscators completely immune to standard (whitebox) protection mechanisms. While promising, this new field of AI-based, and more specifically search-based blackbox deobfuscation, is still in its infancy. In this article we deepen the state of search-based blackbox deobfuscation in three key directions: understand the current state-of-the-art, improve over it and design dedicated protection mechanisms. In particular, we define a novel generic framework for search-based blackbox deobfuscation encompassing prior work and highlighting key components; we are the first to point out that the search space underlying code deobfuscation is too unstable for simulation-based methods (e.g., Monte Carlo Tree Search used in prior work) and advocate the use of robust methods such as S-metaheuristics; we propose the new optimized search-based blackbox deobfuscator Xyntia which significantly outperforms prior work in terms of success rate (especially with small time budget) while being completely immune to the most recent anti-analysis code obfuscation methods; and finally we propose two novel protections against search-based blackbox deobfuscation, allowing to counter Xyntia powerful attacks.

Technical Report: Hardening Code Obfuscation Against Automated Attacks

“…They treat code as a black box and attempt to reconstruct the original code based on the observable behavior, often represented in the form of input-output samples. Approaches such as SYNTIA [7] and XYNTIA [46] attempt to find an expression with equivalent behavior by relying on a stochastic algorithm traversing a large search space. Other approaches, e. g., QSYNTH [22], are based on enumerative synthesis: they compute large lookup tables of expressions which they use to simplify parts of an expression, reducing its overall complexity.…”

“…Our deobfuscation tooling is based on MIASM [10] (commit 65ab7b8), TRITON [56] (v. 0.8.1), and SYNTIA [7] (commit e26d9f5). Unfortunately, the code of other automated deobfuscation tools, such as QSYNTH [22], MBA-BLAST [43], or XYN-TIA [46], is not publicly available, preventing us from evaluating them. While NEUREDUCE [29] is available, we found that it crashed on every execution.…”

“…While conceptually simple, this approach proved effective against many analysis techniques, such as symbolic execution. As a consequence, a number of approaches towards deobfuscating MBAs were proposed, including pattern matching (SSPAM [27]), symbolic simplification using a Boolean expression solver (ARYBO [33]), program synthesis (SYNTIA [7], XYNTIA [46], QSYNTH [22]), machine learning (NEUREDUCE [29]), and algebraic simplification (MBA-BLAST [43]). While those techniques are effective against common MBAs, LOKI's generic approach to synthesize diverse MBAs produces expressions resilient against such attacks (cf.…”

“…Thwarting Program Synthesis. Program synthesis is one of the most powerful attack vectors [7,46]. Concurrent work [46] proposes a search-based program synthesis approach outperforming SYNTIA.…”

“…Usually, automated approaches to deobfuscate MBAs are based on symbolic simplification [6,26,27,33]; these techniques rely on certain assumptions about the expression's structure, making them unfit to simplify such expressions in the general case. Other approaches are based on program synthesis [7,22,46], which have been proven highly effective for most tasks. In general, such synthesis-based deobfuscation techniques remain unchallenged to date and are valuable methods for automated analysis of obfuscated code.…”