Proactive Detection of Computer Worms Using Model Checking

Abstract

Abstract-Although recent estimates are speaking of 200,000 different viruses, worms, and Trojan horses, the majority of them are variants of previously existing malware. As these variants mostly differ in their binary representation rather than their functionality, they can be recognized by analyzing the program behavior, even though they are not covered by the signature databases of current antivirus tools. Proactive malware detectors mitigate this risk by detection procedures which use a single signature to detect whole classes of functionally related malware without signature updates. It is evident that the quality of proactive detection procedures depends on their ability to analyze the semantics of the binary. In this paper, we propose the use of model checking-a well established software verification technique-for proactive malware detection. We describe a tool which extracts an annotated control flow graph from the binary and automatically verifies it against a formal malware specification. To this end, we introduce the new specification language CTPL, which balances the high expressive power needed for malware signatures with efficient model checking algorithms. Our experiments demonstrate that our technique indeed is able to recognize variants of existing malware with a low risk of false positives.

A Tour of the Computer Worm Detection Space

“…A number of techniques [12]- [16] have been presented for detecting polymorphic worms. Again, the technique used is highlighted, the worm characteristic leverage is indicated and the limitations of the technique are pointed out in the synthesis in Table 3.…”

Towards Static Analysis of Virtualization-Obfuscated Binaries

“…To evaluate the precision of the analysis, we count the number of distinct possible arguments that the analysis can detect for the final external call to printf, before invariably widening the values (since the Fibonacci sequence grows to infinity, an exhaustive presentation of all values is impossible). This metric is representative for an application in malware detection, for example, where possible arguments to external system calls are evaluated against a malware specification [7]. In fact, most reverse engineering applications will have similar precision requirements, since at the very least they require a precise set of possible VPC values for each location.…”

“…In earlier work, we have argued for applying static analysis to x86 binaries, both for verifying specifications of API contracts [5] and to detect malware [7]. Static analysis can aid reverse engineering by extracting data flow information (invariants) about the program behavior.…”

Pushdown Model Checking for Malware Detection

“…Towards this aim, we propose in this paper to use modelchecking for virus detection. Model-checking has already been used for virus detection in [6,20,9,11,16,15,17]. However, these works model the program as a finite state graph (automaton).…”

“…CTPL was introduced in [16,15,17]. It can be seen as an extension of CTL with variables and quantifiers.…”