Dynamic Analysis of Executables to Detect and Characterize Malware

Abstract

Malware detection and remediation is an on-going task for computer security and IT professionals. It is needed to ensure the integrity of systems that process sensitive information and control many aspects of everyday life. We examine the use of machine learning algorithms to detect malware using the system calls generated by executables-alleviating attempts at obfuscation as the behavior is monitored rather than the bytes of an executable. We examine several machine learning techniques for detecting malware including random forests, deep learning techniques, and liquid state machines. The experiments examine the effects of concept drift on each algorithm to understand how well the algorithms generalize to novel malware samples by testing them on data that was collected after the training data. The results suggest that each of the examined machine learning algorithms is a viable solution to detect malware-achieving between 90% and 95% class-averaged accuracy (CAA). In real-world scenarios, the performance evaluation on an operational network may not match the performance achieved in training. Namely, the CAA may be about the same, but the values for precision and recall over the malware can change significantly. We structure experiments to highlight these caveats and offer insights into expected performance in operational environments. In addition, we use the induced models to gain a better understanding about what differentiates the malware samples from the goodware, which can further be used as a forensics tool to understand what the malware (or goodware) was doing to provide directions for investigation and remediation.

Mind the Gap

“…They identify six key challenges: 1) ML is better for finding similarities rather than differences, 2) very high cost of classification errors, 3) a semantic gap between detection results and their operational interpretation, 4) enormous variability in what is "normal", 5) difficulties in sound evaluation of the results, and 6) operating in an adversarial setting. In the context of detecting malware, other work noted discrepancies particularly with respect to the precision of malware-indicating a large jump in false negatives when deployed in real-world settings stemming from the difference in the proportion of malware and the difficulty on modeling "normal" in executables [71].…”

Mind the Gap: On Bridging the Semantic Gap between Machine Learning and Information Security

“…They identify six key challenges: 1) ML is better for finding similarities rather than differences, 2) very high cost of classification errors, 3) a semantic gap between detection results and their operational interpretation, 4) enormous variability in what is "normal", 5) difficulties in sound evaluation of the results, and 6) operating in an adversarial setting. In the context of detecting malware, other work noted discrepancies particularly with respect to the precision of malware-indicating a large jump in false negatives when deployed in real-world settings stemming from the difference in the proportion of malware and the difficulty on modeling "normal" in executables [77].…”

An Ensemble-Based Malware Detection Model Using Minimum Feature Set

“…However, dynamic analysis approaches are also imperfect. It is reported in [3,[5][6][7][8][9] that smart malware can detect whether it runs on a virtual or real environment. Moreover, smart malware can modify their behavior by hiding their malicious code to avoid detection.…”

“…Unknown vulnerabilities allow an attack window time, that is the time between threat discovery to signature update. It is shown in [2][3][4][5][6] that malware authors use tools to pack and obfuscate malware to hide malware and to escape from detection. Consequently, signature-based approaches are inefficient to cope with malware authors tricks and therefore, they are unreliable.…”

“…However, non-signature based approaches can be vulnerable to false positives or false negatives results. Technically, it identifies normal benign files as malware and incapable to detect malware in the latter case [5][6][7]. However, it totally avoids the attack window time.…”