Maat

Abstract

The malware analysis and detection research community relies on the online platform VirusTotal to label Android apps based on the scan results of around 60 antiviral scanners. Unfortunately, there are no standards on how to best interpret the scan results acquired from VirusTotal, which leads to the utilization of different threshold-based labeling strategies (e.g., if 10 or more scanners deem an app malicious, it is considered malicious). While some of the utilized thresholds may be able to accurately approximate the ground truths of apps, the fact that VirusTotal changes the set and versions of the scanners it uses makes such thresholds unsustainable over time. We implemented a method, Maat , that tackles these issues of standardization and sustainability by automatically generating a Machine Learning ( ML )-based labeling scheme, which outperforms threshold-based labeling strategies. Using the VirusTotal scan reports of 53K Android apps that span 1 year, we evaluated the applicability of Maat ’s Machine Learning ( ML )-based labeling strategies by comparing their performance against threshold-based strategies. We found that such ML -based strategies (a) can accurately and consistently label apps based on their VirusTotal scan reports, and (b) contribute to training ML -based detection methods that are more effective at classifying out-of-sample apps than their threshold-based counterparts.

Sharing cyber threat intelligence: Does it really help?

“…To build a ground truth dataset of Indicator objects, we retrieve scanning reports from three services: VirusTotal, HybridAnalysis, and MetaDefender, which are widely used to validate threat indicators [28], [38], [37], [21], [55]. For each attribute type (i.e., malware hashes, domains, URLs, and IP addresses), VirusTotal and MetaDefender provide detection results from various anomaly detection engines, and HybridAnalysis provides a threat score with three status tags (i.e., malicious, suspicious, and no specific threat).…”

Analysis of Malicious Files Gathering via Honeypot Trap System and Benchmark of Anti-Virus Software

“…As a result, the platform user must choose how to interpret this information to determine if a file is malicious. There are no standard procedures for interpreting scan results from Virus Total to directly tag applications [32]. In this study, malicious files were detected using the signature-based detection method with Virus Total's assist, and the anti-virus programs' responses were evaluated.…”

Applying Machine Learning to use security oracles: a case study in virus and malware detection

“…First of all, no standards exist on how to interpret the scan results. Several researchers use threshold-based labeling strategies but these thresholds are inconsistent over time as the set and version of scanners used changes [8]. Salem et al propose a machine learning-based labeling schema that outperforms this threshold-based strategy [8].…”

“…Peng et al show that phishing detection is difficult for most vendors, that scanning results are not updated immediately and that third-party vendors often not permit to use all functionality or to access the most updated blacklists [10]. While the evaluation and benchmarking of VirusTotal is out of scope for this work, we refer to [8], [9], [11]- [13] for more details.…”

“…Perhaps we still miss some of these corner cases. Nonetheless, the construction of adversarial examples that meets the above complex constraints is not as straightforward as is the case for creating adversarial images with algorithms like Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD) using a toolbox like Clever-Hans 7 or the Adversarial Robustness Toolbox (ART) 8 . In our application area, there is no immediate notion of "Gradient" to guide the creation of adversarial examples against our ensemble model.…”