Malicious Attacks against Deep Reinforcement Learning Interpretations

Huai, Mengdi; Sun, Jianhui; Cai, Renqin; Yao, Liuyi; Zhang, Aidong

doi:10.1145/3394486.3403089

Cited by 23 publications

(15 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, we consider the top-k robustness, which requires that the set of concepts with the k highest importance scores remains invariant over small-norm adversarial perturbations. In practice, the top-k attack (Ghorbani, Abid, and Zou 2019;Slack et al 2020;Huai et al 2020b;Sarkar, Sarkar, and Balasubramanian 2020;Stergiou 2021) seeks to perturb the concept importance map by decreasing the relative importance of the k initially most important concepts. Let [D] and S x,k denote the index set of the concepts and the set of concepts that had the top k highest importance scores for sample x, respectively.…”

Section: Methodsmentioning

confidence: 99%

Towards Automating Model Explanations with Certified Robustness Guarantees

Huai

Liu

Miao³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Providing model explanations has gained significant popularity recently. In contrast with the traditional feature-level model explanations, concept-based explanations can provide explanations in the form of high-level human concepts. However, existing concept-based explanation methods implicitly follow a two-step procedure that involves human intervention. Specifically, they first need the human to be involved to define (or extract) the high-level concepts, and then manually compute the importance scores of these identified concepts in a post-hoc way. This laborious process requires significant human effort and resource expenditure due to manual work, which hinders their large-scale deployability. In practice, it is challenging to automatically generate the concept-based explanations without human intervention due to the subjectivity of defining the units of concept-based interpretability. In addition, due to its data-driven nature, the interpretability itself is also potentially susceptible to malicious manipulations. Hence, our goal in this paper is to free human from this tedious process, while ensuring that the generated explanations are provably robust to adversarial perturbations. We propose a novel concept-based interpretation method, which can not only automatically provide the prototype-based concept explanations but also provide certified robustness guarantees for the generated prototype-based explanations. We also conduct extensive experiments on real-world datasets to verify the desirable properties of the proposed method.

show abstract

Section: Methodsmentioning

confidence: 99%

Towards Automating Model Explanations with Certified Robustness Guarantees

Huai

Liu

Miao³

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…For the theoretical analysis, two standard victims with adversarial observations, i.e., tabular certainty equivalence learner in reinforcement learning and linear quadratic regulator in control, have been analyzed in a convex optimization problem on which global optimality, and the attack feasibility and attack cost have been provided [201]. In addition, the effectiveness of an universal adversarial attack against DRL interpretations (i.e., UADRLI) has been verified by the theoretical analysis [204], from which the attacker can add the crafted universal perturbation uniformly to the environment states in a maximum number of steps to incur minimal damage. In order to stealthily attack the DRL agents, the work in [205] has injected adversarial samples in a minimal set of critical moments while causing the most severe damage to the agent.…”

Section: • Model Inversion By Casting the Model Inversion Taskmentioning

confidence: 99%

“…If the adversary attack the DRL system with the capacity of accessing to the architecture, weight parameters of the policy and Q networks, and querying the network, we can call it as a white-box attack. Clearly, the attacker can formulate an optimization framework for the white-box setting [142], [204] and derive the optimal adversarial perturbation. Moreover, via the theoretical analysis of the attack feasibility and attack cost, the adversary can attack the DRL agent efficient and stealthily [143], [201].…”

Section: • Model Inversion By Casting the Model Inversion Taskmentioning

confidence: 99%

“…Specifically, the work [142] has showed existing adversarial example crafting techniques can be used to significantly degrade test-time performance of trained policies. However, in terms of defense mechanisms, the attacker may control time steps [205] or solve an optimization framework in a stealthy manner [204]. Another attack of this category aims at maliciously luring an agent to a designated state more than decrease the cumulative rewards [199].…”

Section: • Model Inversion By Casting the Model Inversion Taskmentioning

confidence: 99%

See 1 more Smart Citation

Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

Ma¹,

Li²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Motivated by the advancing computational capacity of distributed end-user equipments (UEs), as well as the increasing concerns about sharing private data, there has been considerable recent interest in machine learning (ML) and artificial intelligence (AI) that can be processed on on distributed UEs. Specifically, in this paradigm, parts of an ML process are outsourced to multiple distributed UEs, and then the processed ML information is aggregated on a certain level at a central server, which turns a centralized ML process into a distributed one, and brings about significant benefits. However, this new distributed ML paradigm raises new risks of privacy and security issues. In this paper, we provide a survey of the emerging security and privacy risks of distributed ML from a unique perspective of information exchange levels, which are defined according to the key steps of an ML process, i.e.: i) the level of preprocessed data, ii) the level of learning models, iii) the level of extracted knowledge and, iv) the level of intermediate results. We explore and analyze the potential of threats for each information exchange level based on an overview of the current state-of-the-art attack mechanisms, and then discuss the possible defense methods against such threats. Finally, we complete the survey by providing an outlook on the challenges and possible directions for future research in this critical area.

show abstract

“…In DRL, the policy is a neural network, and its framework contains an agent that interacts with the environment as it operates and makes a decision by rewarding the desired behavior and punishing the undesired one. The continual interaction of the DRL agent with the end-user exposes it to the number of adversarial attacks [4], [7]. The assumption of having a secure environment to interact is not satisfactory in vehicular applications where misbehavior from an attacker can be lifethreatening in case of accidents in vehicles.…”

Section: Introductionmentioning

confidence: 99%

Adversarial Attacks Against Deep Reinforcement Learning Framework in Internet of Vehicles

Talpur,

Gurusamy

2021

Preprint

View full text Add to dashboard Cite

Machine learning (ML) has made incredible impacts and transformations in a wide range of vehicular applications. As the use of ML in Internet of Vehicles (IoV) continues to advance, adversarial threats and their impact have become an important subject of research worth exploring. In this paper, we focus on Sybil-based adversarial threats against a deep reinforcement learning (DRL)-assisted IoV framework and more specifically, DRL-based dynamic service placement in IoV. We carry out an experimental study with real vehicle trajectories to analyze the impact on service delay and resource congestion under different attack scenarios for the DRL-based dynamic service placement application. We further investigate the impact of the proportion of Sybil-attacked vehicles in the network. The results demonstrate that the performance is significantly affected by Sybil-based data poisoning attacks when compared to adversary-free healthy network scenario.

show abstract

Malicious Attacks against Deep Reinforcement Learning Interpretations

Cited by 23 publications

References 5 publications

Towards Automating Model Explanations with Certified Robustness Guarantees

Towards Automating Model Explanations with Certified Robustness Guarantees

Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

Adversarial Attacks Against Deep Reinforcement Learning Framework in Internet of Vehicles

Contact Info

Product

Resources

About