Empirical assessment of machine learning-based malware detectors for Android

Abstract

To address the issue of malware detection through large sets of applications, researchers have recently started to investigate the capabilities of machine-learning techniques for proposing effective approaches. So far, several promising results were recorded in the literature, many approaches being assessed with what we call in the lab validation scenarios. This paper revisits the purpose of malware detection to discuss whether such in the lab validation scenarios provide reliable indications on the performance of malware detectors in real-world settings, aka in the wild.To this end, we have devised several Machine Learning classifiers that rely on a set of features built from applications' CFGs. We use a sizeable dataset of over 50 000 Android applications collected from sources where state-of-the art approaches have selected their data. We show that, in the lab, our approach outperforms existing machine learning-based approaches. However, this high performance does not translate in high performance in the wild. The performance gap we observed-F-measures dropping from over 0.9 in the lab to below 0.1 in the wild -raises one important question: How do state-of-the-art approaches perform in the wild ?

Potential Component Leaks in Android Apps: An Investigation into a New Feature Set for Malware Detection

“…We found that the performance decreases (but still with Fmeasure above 0.86) with the ratio of malware in the set. Such a finding was already shown in Allix et al's large scale empirical study with a different feature set [2]. RQ3:PCLs constitute good features for discriminating malicious apps from benign apps in a Machine learning-based malware detection scheme.…”

“…The size of training sets and the parameters we use (e.g., malware/goodware ratio) take different values that appear to be unjustified since, as shown in [2], no survey has determined the appropriate values for malware detection. However, our results show the same trends of that shown in [2].…”

“…Allix et al [2] empirically investigated the assessment of machine learning-based malware detectors for Android apps to measure the impact of datasets size and goodware/malware ratio, and the importance of validation scenarios. Our work is related in that we also measure the impact of several parameters and we raise one more factor to take into account when evaluating a malware detection approach: One specific approach may perform well only on a subset of Android applications.…”

“…Machine learning techniques, by allowing to sift through large sets of apps to detect malicious apps, appear to be promising for large-scale malware detection and eventually to keep malicious apps from entering app markets [2]. Stateof-the-art machine learning approaches for Android malware detection mainly differ in the feature sets that are considered for training the classifiers.…”

On the Lack of Consensus in Anti-Virus Decisions: Metrics and Insights on Building Ground Truths of Android Malware

“…It is thus obvious that the performance of the detector is tightly dependent on the quality of the training dataset. Previous works have even shown that the accuracy of such detectors can be degraded by orders of magnitude if the training data is faulty [26]. Following these ndings, one can easily infer that it is also possible to articially improve the performance of malware detectors by selecting a ground truth that splits around malware corner cases.…”