Evading Anti-Malware Engines With Deep Reinforcement Learning

Fang, Zhiyang; Wang, Junfeng; Li, Boya; Wu, Siqi; Zhou, Yingjie; Huang, Haiying

doi:10.1109/access.2019.2908033

Cited by 63 publications

(24 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This malware corpus is from [47] which is significantly large containing more than 222,700 Windows PE malware of over 500 different classes such as Trojan, Virus, Worm, Backdoor and so on. The samples are mainly downloaded from VirusTotal [48] and class tags are mostly manually labeled.…”

Section: Large Pe Malware Datasetmentioning

confidence: 99%

ConvProtoNet: Deep Prototype Induction towards Better Class Representation for Few-Shot Malware Classification

Tang

Wang

2020

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Traditional malware classification relies on known malware types and significantly large datasets labeled manually which limits its ability to recognize new malware classes. For unknown malware types or new variants of existing malware containing only a few samples each class, common classification methods often fail to work well due to severe overfitting. In this paper, we propose a new neural network structure called ConvProtoNet which employs few-shot learning to address the problem of scarce malware samples while prevent from overfitting. We design a convolutional induction module to replace the insufficient prototype reduction in most few-shot models and generates more appropriate class-level malware prototypes for classification. We also adopt meta-learning scheme to make classifier robust enough to adapt unseen malware classes without fine-tuning. Even in extreme conditions where only 5 samples in each class are provided, ConvProtoNet still achieves more than 70% accuracy on average and outperforms other traditional malware classification methods or existed few-shot models in experiments conducted on several datasets. Extra experiments across datasets illustrate that ConvProtoNet learns general knowledge of malware which is dataset-invariant and careful model analysis proves effectiveness of ConvProtoNet in few-shot malware classification.

show abstract

Section: Large Pe Malware Datasetmentioning

confidence: 99%

ConvProtoNet: Deep Prototype Induction towards Better Class Representation for Few-Shot Malware Classification

Tang

Wang

2020

Applied Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the case of malware detection [18], [19], [20], [21] and binary code analysis [22], [23] relevant studies have been actively conducted. However, anti-reversing techniques have not attracted special interest from researchers except for specific topics such as code virtualization [24], [25].…”

Section: Related Workmentioning

confidence: 99%

x64Unpack: Hybrid Emulation Unpacker for 64-bit Windows Environments and Detailed Analysis Results on VMProtect 3.4

et al. 2020

View full text Add to dashboard Cite

In spite of recent remarkable advances in binary code analysis, malware developers are still using complex anti-reversing techniques to make analysis difficult. To protect malware, they use packers, which are (commercial) tools that contain various anti-reverse engineering techniques such as code encryption, anti-debugging, and code virtualization. In this paper, we present x64Unpack: a hybrid emulation scheme that makes it easier to analyze packed executable files and automatically unpacks them in 64-bit Windows environments. The most distinguishable feature of x64Unpack compared to other dynamic analysis tools is that x64Unpack and the target program share virtual memory to support both instruction emulation and direct execution. Emulation runs slow but provides detailed information, whereas direct execution of the code chunk runs very fast and can handle complex cases regarding to operating systems or hardware devices. With x64Unpack, we can monitor major API (Application Programming Interface) function calls or conduct fine-grained analysis at the instruction-level. Furthermore, x64Unpack can detect anti-debugging code chunks, dump memory, and unpack the packed files. To verify the effectiveness of x64Unpack, experiments were conducted on the obfuscation tools: UPX 3.95, MPRESS 2.19, Themida 2.4.6, and VMProtect 3.4. Especially, VMProtect and Themida are considered as some of the most complex commercial packers in 64bit Windows environments. Experimental results show that x64Unpack correctly emulates the packed executable files and successfully produces the unpacked version. Based on this, we provide the detailed analysis results on the obfuscated executable file that was generated by VMProtect 3.4.

show abstract

“…In addition, some research [23], [24] creatively applied reinforcement learning methods to the field of information security. Inspired by the research [25], which trained a reinforcement learning model to design the architecture of CNN for the task of image classification, we previously proposed a DQEAF framework using reinforcement learning to evade anti-malware engines [26]. Considering that rarely researches has previously used reinforcement learning to select features for malware classification, a reinforcement learning-based model is proposed.…”

Section: B Reinforcement Learningmentioning

confidence: 99%

Feature Selection for Malware Detection Based on Reinforcement Learning

2019

Self Cite

View full text Add to dashboard Cite

Machine learning based malware detection has been proved great success in the past few years. Most of the conventional methods are based on supervised learning, which relies on static features with labels. While selecting static features requires both human expertise and labor. New selections, which fix features from a wide range, are handcrafted by careful manual experimentation or modified from existing methods. Despite their success, the static features are still hard to be determined. In this paper, a Deep Q-learning based Feature Selection Architecture (DQFSA) is introduced to cover the deficiencies of traditional methods. The proposed architecture automatically selects a small set of highly differentiated features for malware detection task without human intervention. DQFSA trains an agent through Q-learning to maximize the expected accuracy of the classifiers on a validation dataset by sequentially interacting with the features space. The agent, based on an -greedy exploration strategy and experience replay, explores a large but finite space of possible actions and iteratively discovers selections with improved performance on the learning task. Actions are a set of reasonable choices, which indicate whether a feature is chosen or not. Extensive experimental results indicate that the proposed DQFSA outperforms existing baseline approaches for feature selection on malware detection with minimum features, improves the generalization performance of the learning model and reduces human intervention. More specifically, the proposed architecture's underlying representation is robust enough for re-calibrating models to other domains of information security.

show abstract

Evading Anti-Malware Engines With Deep Reinforcement Learning

Cited by 63 publications

References 20 publications

ConvProtoNet: Deep Prototype Induction towards Better Class Representation for Few-Shot Malware Classification

ConvProtoNet: Deep Prototype Induction towards Better Class Representation for Few-Shot Malware Classification

x64Unpack: Hybrid Emulation Unpacker for 64-bit Windows Environments and Detailed Analysis Results on VMProtect 3.4

Feature Selection for Malware Detection Based on Reinforcement Learning

Contact Info

Product

Resources

About