Enabling Fair ML Evaluations for Security

Pendlebury, Feargus; Pierazzi, Fabio; Jordaney, Roberto; Kinder, Johannes; Cavallaro, Lorenzo

doi:10.1145/3243734.3278505

Cited by 33 publications

(62 citation statements)

References 13 publications

(19 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The wrapper classes provided by the Keras library allow the usage of neural network models developed with Keras in scikit-learn. Also, there is a KerasClassifier class in Keras that can be used as an Estimator in scikit-learn ( Pendlebury et al, 2018 ). The KerasClassifier takes the name of a function as an argument.…”

Section: Methodsmentioning

confidence: 99%

Toward fault tolerant modelling for SCADA based electricity distribution networks, machine learning approach

Masri

Al-Jabi

2021

PeerJ Computer Science

View full text Add to dashboard Cite

Maintaining electrical energy is a crucial issue, especially in developing countries with very limited possibilities and recourses. However, the increasing reliance on electrical appliances generates many challenges for operators to fix any fault optimally within minimum time. Even with numerous researches conducted in this area, very few were interested in minimizing the fault duration, especially in the developing countries with very limited resources. Since decision-making requires enough information within minimum time, the integration of information technology with the existing electrical grids is the most appropriate. In this paper, we propose precise and accurate load redistribution estimation models. While several modeling techniques exist, the proposed modeling techniques in this work are based on machine learning models: multiple linear regression, nonlinear regression, and classifier neural network models. The novelty of this work is it introduces a fault-tolerant approach that relies on machine learning and supervisory control and data acquisition system (SCADA). The purpose of this approach is to help electricity distribution companies to maintain power for the customers and to shorten the fault duration from many hours to the minimum possible time. The work was performed based on real data of smart grids split into zones of about 20 transformers. The models’ input data collected from the sensors allocated in the power grid, make the grid becomes able to redistribute the loads by sufficient strategies. To test and validate the models, two powerful modeling tools were used: MATLAB and Anaconda–Python. The results showed an accuracy of about 97% with a standard deviation of 2.3%. The load redistribution was also presented in details. With such eager results, they approve the validity of our model in minimizing the fault duration, by helping the system in taking ideal actions within the optimal time.

show abstract

Section: Methodsmentioning

confidence: 99%

Toward fault tolerant modelling for SCADA based electricity distribution networks, machine learning approach

Masri

Al-Jabi

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…In this example we focus on ML-based detection methods given their popularity within the academic community [8], [10], [15], [17], [18]. Researchers devise new techniques to extract features from Android apps and use those features to train ML models.…”

Section: A Motivating Examplementioning

confidence: 99%

“…On the one hand, researchers might dismiss promising detection approaches, because they underperform on a dataset that utilizes a labeling strategy that does not reflect the true nature of the apps in the dataset. On the other hand, developers of inadequate detection methods might get a false sense of confidence in the detection capabilities of their detection methods because they perform well, albeit using an inaccurate labeling strategy [15], [16].…”

Section: Introductionmentioning

confidence: 99%

Towards Accurate Labeling of Android Apps for Reliable Malware Detection

Salem

2021

Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy

View full text Add to dashboard Cite

In training their newly-developed malware detection methods, researchers rely on threshold-based labeling strategies that interpret the scan reports provided by online platforms, such as VirusTotal. The dynamicity of this platform renders those labeling strategies unsustainable over prolonged periods, which leads to inaccurate labels. Using inaccurately labeled apps to train and evaluate malware detection methods significantly undermines the reliability of their results, leading to either dismissing otherwise promising detection approaches or adopting intrinsically inadequate ones. The infeasibility of generating accurate labels via manual analysis and the lack of reliable alternatives force researchers to utilize VirusTotal to label apps. In the paper, we tackle this issue in two manners. Firstly, we reveal the aspects of VirusTotal's dynamicity and how they impact thresholdbased labeling strategies and provide actionable insights on how to use these labeling strategies given VirusTotal's dynamicity reliably. Secondly, we motivate the implementation of alternative platforms by (a) identifying VirusTotal limitations that such platforms should avoid, and (b) proposing an architecture of how such platforms can be constructed to mitigate VirusTotal's limitations.

show abstract

“…In particular, the API design of TESSERACT is heavily inspired by and fully compatible with SCIKIT-LEARN [39] and KERAS [10]; as a result, many of the conventions and workflows in TESSERACT will be familiar to users of these libraries. Here we present an overview of the library's core modules while further details of the design can be found in [40].…”

Section: A5 Tesseract Implementationmentioning

confidence: 99%

TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time

Pendlebury,

Pierazzi,

Jordaney

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

Is Android malware classification a solved problem? Published F 1 scores of up to 0.99 appear to leave very little room for improvement. In this paper, we argue that results are commonly inflated due to two pervasive sources of experimental bias: spatial bias caused by distributions of training and testing data that are not representative of a real-world deployment; and temporal bias caused by incorrect time splits of training and testing sets, leading to impossible configurations. We propose a set of space and time constraints for experiment design that eliminates both sources of bias. We introduce a new metric that summarizes the expected robustness of a classifier in a real-world setting, and we present an algorithm to tune its performance. Finally, we demonstrate how this allows us to evaluate mitigation strategies for time decay such as active learning. We have implemented our solutions in TESSERACT, an open source evaluation framework for comparing malware classifiers in a realistic setting. We used TESSERACT to evaluate three Android malware classifiers from the literature on a dataset of 129K applications spanning over three years. Our evaluation confirms that earlier published results are biased, while also revealing counter-intuitive performance and showing that appropriate tuning can lead to significant improvements.

show abstract

Enabling Fair ML Evaluations for Security

Cited by 33 publications

References 13 publications

Toward fault tolerant modelling for SCADA based electricity distribution networks, machine learning approach

Toward fault tolerant modelling for SCADA based electricity distribution networks, machine learning approach

Towards Accurate Labeling of Android Apps for Reliable Malware Detection

TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time

Contact Info

Product

Resources

About