Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Drift

Abstract

Machine learning for malware classification shows encouraging results, but real deployments suffer from performance degradation as malware authors adapt their techniques to evade detection. This phenomenon, known as concept drift, occurs as new malware examples evolve and become less and less like the original training examples. One promising method to cope with concept drift is classification with rejection in which examples that are likely to be misclassified are instead quarantined until they can be expertly analyzed.We revisit TRANSCEND, a recently proposed framework for performing rejection based on conformal prediction theory. In particular, we provide a formal treatment of TRANSCEND, enabling us to refine conformal evaluation theory-its underlying statistical engine-and gain a better understanding of the theoretical reasons for its effectiveness. In the process, we develop two additional conformal evaluators that match or surpass the performance of the original while significantly decreasing the computational overhead. We evaluate our extension on a large dataset that removes sources of experimental bias present in the original evaluation.Finally, to aid practitioners, we determine the optimal operational settings for a TRANSCEND deployment and show how it can be applied to many popular learning algorithms. These insights support both old and new empirical findings, making TRANSCEND a sound and practical solution for the first time. To this end, we release our implementation of TRANSCEND as open source, to aid the adoption of rejection strategies by the security community.

Dos and Don'ts of Machine Learning in Computer Security

“…Many learning-based systems in security are evaluated solely in laboratory settings, overstating their practical impact. A common example are detection methods evaluated only in a closed-world setting with limited diversity and no consideration of non-stationarity [15,70]. For example, a large number of website fingerprinting attacks are evaluated only in closed-world settings spanning a limited time period [71].…”

“…However, public datasets need to be treated with caution. Firstly, data ages and becomes less relevant in the fast-moving security landscape, partially due to concept drift [15,70,85,101]. Secondly, the characteristics of the data are increasingly exposed and thereby lead to implicit data snooping (P3) [see 1,88].…”

“…For example, temporal and spatial relations of the data should be considered to account for the typical dynamics encountered in the wild [see 101]. Similarly, runtime and storage constraints should be analyzed under practical conditions [see 15,107,126]. Ideally, the proposed system should be deployed to uncover problems that are not observable in a lab-only environment, such as the diversity and complexity of real-world network traffic [see 115].…”

Early Detection of Abnormal Attacks in Software-Defined Networking Using Machine Learning Approaches

“…For example, Jordaney et al [54] proposed the Transced framework to identify concept drift to establish prediction indicators. Barbero et al [55] based on the former framework for performing rejection classification, has improved efficiency and reduced computing expenses. For an ML-based classifier to be highly sustainable, it is critical to understand the underlying features: the ability to distinguish benign applications from malware and extract the changing pattern of those features through evolutionary processes [56].…”