Intriguing Properties of Adversarial ML Attacks in the Problem Space

Abstract

Recent research efforts on adversarial ML have investigated problem-space attacks, focusing on the generation of real evasive objects in domains where, unlike images, there is no clear inverse mapping to the feature space (e.g., software). However, the design, comparison, and real-world implications of problem-space attacks remain underexplored.This paper makes two major contributions. First, we propose a novel formalization for adversarial ML evasion attacks in the problem-space, which includes the definition of a comprehensive set of constraints on available transformations, preserved semantics, robustness to preprocessing, and plausibility. We shed light on the relationship between feature space and problem space, and we introduce the concept of side-effect features as the byproduct of the inverse feature-mapping problem. This enables us to define and prove necessary and sufficient conditions for the existence of problem-space attacks. We further demonstrate the expressive power of our formalization by using it to describe several attacks from related literature across different domains.Second, building on our formalization, we propose a novel problem-space attack on Android malware that overcomes past limitations. Experiments on a dataset with 170K Android apps from 2017 and 2018 show the practical feasibility of evading a state-of-the-art malware classifier along with its hardened version. Our results demonstrate that "adversarial-malware as a service" is a realistic threat, as we automatically generate thousands of realistic and inconspicuous adversarial applications at scale, where on average it takes only a few minutes to generate an adversarial app. Yet, out of the 1600+ papers on adversarial ML published in the past six years, roughly 40 focus on malware [15]-and many remain only in the feature space.Our formalization of problem-space attacks paves the way to more principled research in this domain. We responsibly release the code and dataset of our novel attack to other researchers, to encourage future work on defenses in the problem space.

Query-Efficient Black-Box Attack Against Sequence-Based Malware Classifiers

“…It is extremely challenging for the malware classifier to differentiate between malware that is trying to read a non-existent registry key as an added adversarial no-op and a benign application functionality, e.g., trying to find a registry key containing information from previous runs and creating it if it doesn't exist (for instance, during the first run of the application). This makes our problem-space attack robust to preprocessing [38].…”

“…This is due to the fact that adversarial training is less effective against random attacks like ours, because a different stochastic adversarial sequence is generated every time, making it challenging for the classifier to generalize from one adversarial sequence to another. More effective RNN defense methods, including domain specific methods, e.g., systems that measure CPU usage [35], contain irregular API call subsequences [27] (such as the no-op API calls used in this paper), or otherwise assess the plausibility of our attack [38], in order to detect adversarial examples, will be a part of our future work.…”

“…We develop an end-to-end attack [40,41] (a.k.a, problem-space attack [38]) by recrafting the malware behavior so it can evade detection by such machine learning malware classifiers while minimizing the amount of queries to the malware classifier (meaning that the attack is query-efficient).…”

Learn2Evade: Learning-Based Generative Model for Evading PDF Malware Classifiers

“…Depending on the different levels of knowledge held by an attacker, attack scenarios can be categorized into three different classes: white-box (i.e., perfect knowledge), gray-box (i.e., limited knowledge) and black-box (i.e., zero knowledge) [32], [33]. The less information available to an attacker, the closer the attack scenario is to black-box.…”

A Review on Android Malware: Attacks, Countermeasures and Challenges Ahead

“…Cylance's PROTECT anti-malware engine which deploys DL models has been recently evaded by adversarial attacks [38]. Thus, ML based malware detectors are under the constant threat of adversarial attacks [39][40][41][42][43][44][45]. This demands for the development of robust and secure learning models to be deployed in malware detectors which can withstand the adaptive adversarial attacks [46][47][48][49][50][51][52].…”