LTL Model Checking of Self Modifying Code

Abstract

Self modifying code is code that can modify its own instructions during the execution of the program. It is extensively used by malware writers to obfuscate their malicious code. Thus, analysing self modifying code is nowadays a big challenge. In this paper, we consider the LTL model-checking problem of self modifying code. We model such programs using self-modifying pushdown systems (SM-PDS), an extension of pushdown systems that can modify its own set of transitions during execution. We reduce the LTL model-checking problem to the emptiness problem of self-modifying Büchi pushdown systems (SM-BPDS). We implemented our techniques in a tool that we successfully applied for the detection of several self-modifying malware. Our tool was also able to detect several malwares that well-known antiviruses such as Bit-Defender, Kinsoft, Avira, eScan, Kaspersky, Qihoo-360, Baidu, Avast, and Symantec failed to detect. arXiv:1909.12635v1 [cs.LO]

SMODIC: A Model Checker for Self-modifying Code

“…As for LTL and CTL modelchecking, they can be reduced to the emptiness problem for Selfmodifying (Alternating) Büchi Pushdown Systems. The basic idea is to make a product of the given SM-PDS and the given LTL or CTL formula to get a Self-modifying (Alternating) Büchi Pushdown system and then apply the algorithms of [43] to check the emptiness of the Self-modifying (Alternating) Büchi Pushdown system.…”

“…For example, we show in this section how LTL can be used to express the malicious behavior of registry key injecting. Other LTL and CTL formulas that describe other malicious behaviors can be found in [13,14,43].…”

“…For example, if we consider the sequence 𝑓 1 , 𝑓 2 , 𝑓 3 , then Reachability component checks whether the SM-PDS has a run that calls first 𝑓 1 , then 𝑓 2 , then 𝑓 3 . The LTL (CTL) component takes as input a SM-PDS and an LTL (CTL) formula, and applies the algorithms of [43] to check whether the SM-PDS satisfies the LTL (CTL) formula.…”

“…Then, it translates this CFG into a SM-PDS. It then implements the algorithms of [42,43] to perform reachability analysis and LTL/CTL model-checking for this model.…”