Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift

Barbero, Federico; Pendlebury, Feargus; Pierazzi, Fabio; Cavallaro, Lorenzo

doi:10.1109/sp46214.2022.9833659

Cited by 17 publications

(27 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Improved Active Learning Schemesmentioning

confidence: 99%

“…First, we adapt TRANSCENDENT [8] to active learning. TRANSCENDENT [8] was originally designed to support classification with rejection, so that the classifier can decline to make any prediction for samples that appear to have drifted. In particular, they construct two scores to recognize drifted samples and decline to classify any sample whose scores are too low.…”

Section: Improved Active Learning Schemesmentioning

confidence: 99%

“…The prediction confidence of a classifier is commonly used to detect OOD samples [23]. TRANSCEN-DENT [8,17] uses two metrics, credibility and confidence, both utilizing the nonconformity measure to reject test samples that may have drifted. The paper did not provide a way to use the two metrics for active learning.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Continuous Learning for Android Malware Detection

Chen¹,

Ding²,

Wagner³

2023

Preprint

View full text Add to dashboard Cite

Machine learning methods can detect Android malware with very high accuracy. However, these classifiers have an Achilles heel, concept drift: they rapidly become out of date and ineffective, due to the evolution of malware apps and benign apps. Our research finds that, after training an Android malware classifier on one year's worth of data, the F1 score quickly dropped from 0.99 to 0.76 after 6 months of deployment on new test samples.In this paper, we propose new methods to combat the concept drift problem of Android malware classifiers. Since machine learning technique needs to be continuously deployed, we use active learning: we select new samples for analysts to label, and then add the labeled samples to the training set to retrain the classifier. Our key idea is, similarity-based uncertainty is more robust against concept drift. Therefore, we combine contrastive learning with active learning. We propose a new hierarchical contrastive learning scheme, and a new sample selection technique to continuously train the Android malware classifier. Our evaluation shows that this leads to significant improvements, compared to previously published methods for active learning. Our approach reduces the false negative rate from 16% (for the best baseline) to 10%, while maintaining the same false positive rate (0.6%). Also, our approach maintains more consistent performance across a seven-year time period than past methods.

show abstract

Section: Improved Active Learning Schemesmentioning

confidence: 99%

Section: Improved Active Learning Schemesmentioning

confidence: 99%

See 1 more Smart Citation

Continuous Learning for Android Malware Detection

Chen¹,

Ding²,

Wagner³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…On "present" data, performances of the reference and test models appear highly comparable (AUC R = 0.9981 versus AUC T = 0.9984), but on future data, model performances are known to degrade-and will our estimates of them-as covariate shift, concept drift, and/or label drift mount [13][14][15]. Using Firenze on the unlabeled dataset, we investigate to what extent the change in model architecture improves performance by (i) increasing true malicious file identifications (true positives) by the model, without increasing false positives and (ii) improving identification of benign files without increasing false negatives.…”

Section: Evaluating Malware Detection Models Using Firenzementioning

confidence: 99%

“…Worse still, any effortfully-collected labeled data carries a high risk of becoming obsolete, since adversaries frequently change tactics and move targets. Taken together, these pecularities can induce severe concept drift, label drift, and covariate shift [13][14][15].…”

Section: Introductionmentioning

confidence: 99%

Firenze: Model Evaluation Using Weak Signals

Soman¹,

Torkamani²,

Morais³

et al. 2022

Preprint

View full text Add to dashboard Cite

Data labels in the security field are frequently noisy, limited, or biased towards a subset of the population. As a result, commonplace evaluation methods such as accuracy, precision and recall metrics, or analysis of performance curves computed from labeled datasets do not provide sufficient confidence in the real-world performance of a machine learning (ML) model. This has slowed the adoption of machine learning in the field. In the industry today, we rely on domain expertise and lengthy manual evaluation to build this confidence before shipping a new model for security applications. In this paper, we introduce Firenze, a novel framework for comparative evaluation of ML models' performance using domain expertise, encoded into scalable functions called markers. We show that markers computed and combined over select subsets of samples called regions of interest can provide a robust estimate of their real-world performances. Critically, we use statistical hypothesis testing to ensure that observed differences-and therefore conclusions emerging from our framework-are more prominent than that observable from the noise alone. Using simulations and two real-world datasets for malware and domain-name-service reputation detection, we illustrate our approach's effectiveness, limitations, and insights. Taken together, we propose Firenze as a resource for fast, interpretable, and collaborative model development and evaluation by mixed teams of researchers, domain experts, and business owners.Preprint. Under review.

show abstract

BenchMFC: A benchmark dataset for trustworthy malware family classification under concept drift

Jiang,

Li,

et al. 2024

Computers & Security

View full text Add to dashboard Cite

Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift

Cited by 17 publications

References 30 publications

Continuous Learning for Android Malware Detection

Continuous Learning for Android Malware Detection

Firenze: Model Evaluation Using Weak Signals

BenchMFC: A benchmark dataset for trustworthy malware family classification under concept drift

Contact Info

Product

Resources

About