SMODIC: A Model Checker for Self-modifying Code

Abstract

In this paper, we present SMODIC, a model checker for selfmodifying binary codes. SMODIC uses Self Modifying Pushdown Systems (SM-PDS) to model self-modifying binary code. This allows to faithfully represent the program's stack as well as the self-modifying instructions of the program. SMODIC takes a selfmodifying binary code or a self modifying pushdown system as input. It can then perform reachability analysis and LTL/CTL modelchecking for these models. We successfully used SMODIC to modelcheck more than 900 self-modifying binary codes. In particular, we applied SMODIC for malware detection, since malwares usually use self-modifying instructions, and since malicious behaviors can be described by LTL or CTL formulas. In our experiments, SMODIC was able to detect 895 malwares and to prove that 200 benign programs were benign. SMODIC was also able to detect several malwares that well-known antiviruses such as Bit-Defender, Kinsoft, Avira, eScan, Kaspersky, Baidu, Avast, and Symantec failed to detect. SMODIC can be found in https://lipn.univ-paris13.fr/~touili/smodic CCS CONCEPTS• Theory of computation → Verification by model checking;• Security and privacy → Logic and verification.