Are Your Training Datasets Yet Relevant?

Abstract

In this paper, we consider the relevance of timeline in the construction of datasets, to highlight its impact on the performance of a machine learning-based malware detection scheme. Typically, we show that simply picking a random set of known malware to train a malware detector, as it is done in many assessment scenarios from the literature, yields significantly biased results. In the process of assessing the extent of this impact through various experiments, we were also able to confirm a number of intuitive assumptions about Android malware. For instance, we discuss the existence of Android malware lineages and how they could impact the performance of malware detection in the wild.

Self-protection of Android systems from inter-component communication attacks

“…These results are particularly notable since previous work has demonstrated that machine learning-based Android malware detection was unable to obtain an F1 score higher than 70% in a time-aware scenario [56]. In that work, dates newer in time resulted in lower F1 scores; however, RevealDroid actually improves to as high as 99%.…”

“…We grouped apps into two-year time periods, due to the fact that some years only have a few apps, mainly 2009 with 29 apps, and 2017 with 130 apps. Similar to [55], we consider the year of the last modified date of classes.dex in an app as the year from which it originates. We consider any transformed app as belonging to the same year as its original version, in order to determine the actual effect of obfuscation on product accuracy for each time period.…”

“…In a time-agnostic scenario, training and testing as part of machine learning is conducted without considering the age of apps in the dataset. This scenario has been utilized to evaluate an overwhelming majority of machine learning-based Android malware-detection approaches [56]. A time-aware scenario uses the modification date of apps to determine training and testing sets, which avoids training on apps from the future to test on apps from the past.…”

“…for precision, recall, and the F1 score; and the Total number of apps. To evaluate RevealDroid in a time-aware scenario, we followed the methodology described by Allix et al [56]. Specifically, we extracted the modification date of the classes.dex file in each app's APK file.…”

Untitled

“…In a time-agnostic scenario, training and testing as part of machine learning is conducted without considering the age of apps in the dataset. This scenario has been utilized to evaluate an overwhelming majority of machine-learning-based Android malware-detection approaches [13]. A time-aware scenario uses the modification date of apps to determine training and testing sets, which avoids training on apps from the future to test on apps from the past.…”