An Insight into the Machine-Learning-Based Fileless Malware Detection

Abstract

In recent years, massive development in the malware industry changed the entire landscape for malware development. Therefore, cybercriminals became more sophisticated by advancing their development techniques from file-based to fileless malware. As file-based malware depends on files to spread itself, on the other hand, fileless malware does not require a traditional file system and uses benign processes to carry out its malicious intent. Therefore, it evades conventional detection techniques and remains stealthy. This paper briefly explains fileless malware, its life cycle, and its infection chain. Moreover, it proposes a detection technique based on feature analysis using machine learning for fileless malware detection. The virtual machine acquired the memory dumps upon executing the malicious and non-malicious samples. Then the necessary features are extracted using the Volatility memory forensics tool, which is then analyzed using machine learning classification algorithms. After that, the best algorithm is selected based on the k-fold cross-validation score. Experimental evaluation has shown that Random Forest outperforms other machine learning classifiers (Decision Tree, Support Vector Machine, Logistic Regression, K-Nearest Neighbor, XGBoost, and Gradient Boosting). It achieved an overall accuracy of 93.33% with a True Positive Rate (TPR) of 87.5% at zeroFalse Positive Rate (FPR) for fileless malware collected from five widely used datasets (VirusShare, AnyRun, PolySwarm, HatchingTriage, and JoESadbox).

A Review of State-of-the-Art Malware Attack Trends and Defense Mechanisms

“…• Fileless malware attacks computer memory or registries, leaving no files and making detection difficult [11].…”

“…Sanjay et al [155] proposed a technique that uses memory forensics-based analysis to detect fileless malware based on opcode sequences, whereas Tsai et al [10] utilized multi-label classifiers for de-obfuscating and profiling malicious PowerShell commands. Khalid et al [11] presented an overview of ML techniques for fileless malware detection, suggesting that combining deep-learning methods with large datasets can provide an effective solution. Borana et al [156] VOLUME 11,2023 proposed an assistive tool for detecting fileless malware, whereas Bozkir et al [128] combined memory forensics, manifold learning, and computer vision to detect malware.…”

“…Khalid et al [11] presented an overview of ML techniques for fileless malware detection, suggesting that combining deep-learning methods with large datasets can provide an effective solution. Borana et al [156] VOLUME 11,2023 proposed an assistive tool for detecting fileless malware, whereas Bozkir et al [128] combined memory forensics, manifold learning, and computer vision to detect malware.…”

“…For example, in 2021, ransomware was the top cyber threat, with attacks increasing by 140% in the third quarter [9]. Fileless attacks are another emerging trend in malware attacks that do not leave traces in the system; therefore, they are difficult to detect [10], [11], [12]. Advanced persistent threats (APTs) are highly complex, target-specific, persistent, and remain undetected until the target is compromised [13], [14].…”

“…The authors of [26], [27], [28], [29], [30], and [31] only studied ransomware detection and defense solutions, and no other malware types. Similarly, [10], [11], [13], [32], [33], [34], [35], and [36] studied APT and fileless malware detection and prevention. In addition, Wang et al and Liu et al examined deep learning-based detection against Android malware [37], [38], offering a thorough analysis but they did not fully explore practical implementation and malware evolution.…”