A Distance-Based Method for Building an Encrypted Malware Traffic Identification Framework

Abstract

The popularity of encryption method brings a great challenge to malware traffic identification. Traditional classes defined by expert experience are usually classified based on the host behaviors of malware, such as banking malware or ransomware, which are often irrelevant to its communication traffic behaviors. It leads to the fact that the boundaries of traffic feature dataset of different malware classes are fuzzy and make these traditional classes unhelpful for classification based on traffic features. Meanwhile, traditional machine learning-based encrypted malware traffic identification methods, such as using the multi-classification supervised learning model, are inefficient both in model training and detection, and its detection accuracy cannot meet the demand. In this paper, we propose a distance-based method, which utilizes unsupervised learning algorithm Gaussian mixture model (GMM) and ordering points to identify the clustering structure (OPTICS) to calculate the Distance between malwares and make use of the Distance to define the new malware class called FClass. Then, a set of models are trained by XGBoost algorithm to build an identification framework based on the FClass. The performance of the proposed method has been evaluated by comparing it with the other four methods. The results show that the proposed distance-based method is more efficient and accurate.

Experimental Evaluation of Malware Family Classification Methods from Sequential Information of TLS-Encrypted Traffic

“…For the random forest classifiers, their testing time seems to be correlated to a tunable parameter (i.e., the number of trees in the forest), each of which is hand-tuned to optimize the accuracy (800 for Markov chain-based random forest and 200 for TIG-based random forest). It implies that the raw input size (25) to construct the traffic interaction graph is optimized, as described in [11]. A similar tendency is observed in the SVM classifiers.…”

“…For malware family classification for TLS-encrypted traffic, [9] showed that the L1 multinomial logistic regression classifier with the enhanced flow features can classify TLS-encrypted malware traffic into 1 of 18 classes (i.e., malware families) with a total accuracy of 90.3%. In [25], an XGBoost [26]-based malware family classification framework was proposed with a distance-based clustering method to measure the similarity between malware families. Ref.…”

“…[27] proposed a multi-task hierarchical learning method, which is based on one-dimensional CNN and gives both mid-level class (e.g., malware family) and top-level class (e.g., either malware or benign) at the same time. In both [25,27], features were selected from the enhanced flow features.…”

“…To this end, MalClassifier [35] presents a malware family classification system from connection logs of a widely used real-time network monitoring framework called Bro (currently Zeek) [36]. Clearly, MalClassifier is efficient and can be operated in real time, but it is reported that a distance-based method [25] outperforms MalClassifer especially due to false positives due to the similarity of several malware families.…”

Rethinking the Weakness of Stream Ciphers and Its Application to Encrypted Malware Detection

“…In [24], Prasse et al proposed a Long Short-term Memory neural network which uses only observable features of HTTPS traffic (client and host IP addresses and ports, timestamps, data flow volume, and the unencrypted host domain name) for malware classification, claiming it outperforms random forest models in similar applications with a 64% detection rate and 70% precision. The authors of [25] proposed a distance-based, supervised learning solution which suffers from the pitfall of relying on collecting a large amount of traffic data and extracting the features before any analysis can be completed. An approach that analyzes persistent communications, instead of the presence of anomalies within the persistent communication itself, is offered in [26].…”