The Sample Complexity of Teaching by Reinforcement on Q-Learning

Zhang, Xuezhou; Bharti, Shubham Kumar; Ma, Yuzhe; Singla, Adish; Zhu, Xiaojin

doi:10.1609/aaai.v35i12.17306

Cited by 12 publications

(34 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This section summarizes the aforementioned works and discusses how our study differs from these existing attacks. [14][15][16]. For example, Ma et al [14] poison rewards in the training set for the batch RL agent, and Zhang et al [15] study adaptive reward-poisoning attack on the Q-learning RL agent.…”

Section: Discussionmentioning

confidence: 99%

“…[14][15][16]. For example, Ma et al [14] poison rewards in the training set for the batch RL agent, and Zhang et al [15] study adaptive reward-poisoning attack on the Q-learning RL agent. Rakhsha et al [16] poisons either reward values or transition functions to attack RL agents performing in cyclic tasks (i.e., the tasks without termination states).…”

Section: Discussionmentioning

confidence: 99%

“…Thus, the proposed environment poisoning attack can manipulate the RL agent's policy without any access to the RL agent's perception or the environment dynamics model. Moreover, unlike those training-time attacks [14][15][16] which are designed for a specific RL algorithm, our attack is proposed for a model-free RL agent but it is not restricted by the types of the RL algorithms.…”

Section: Discussionmentioning

confidence: 99%

“…More recently, training-time attacks [14][15][16][17][18] have shown intensifying concern about security threats to the learning of RL policies. For example, there exists the possibility that the chatbot can be mistaught by a small group of Twitter users to make misogynistic and racist comments [19]; it is possible for autonomous cars to misread traffic signs if the signs are contaminated by adversarial stickers [20]; it has been observed that recommendation systems can be tricked by fake clicks and comments in order to rank products higher than they ought to be.…”

Section: List Of Figuresmentioning

confidence: 99%

“…Therefore, it is imperative to investigate the threats posed by environment hyper-parameters as well as to develop protection solutions for RL policies. Furthermore, the success of existing training-time attacks [14][15][16] relies on comprehensive prior knowledge of the RL system, including RL agent's learning mechanism (i.e., learning algorithm and policy model) and/or its environment model (i.e., transition dynamics and reward functions). Unfortunately, assuming such an omniscient attacker makes attack approaches somewhat unrealistic, which limits threats to real-world RL-based applications.…”

Section: List Of Figuresmentioning

confidence: 99%

See 4 more Smart Citations

Environment poisoning in reinforcement learning: attacks and resilience

Xu¹

View full text Add to dashboard Cite

Upon finishing my Ph.D. thesis, gratitude is the strongest feeling inside of me. I would like to express my sincere appreciation to all those who have provided me with invaluable help in this process. I wish to express my greatest gratitude to my supervisor, Prof. Zinovi Rabinovich, not only for his expert guidance and scholarly advice on my research work, but also for his strong support and encouragement throughout my Ph.D. journey. His insightful comments and critical thinking always inspire me in tackling research problems. His attitude towards the academic research provides me with a good model of what a true researcher should be. Without his patient instruction and constant encouragement, I could not fulfil these research works and complete the Ph.D. study. I would like to thank all talented members in Prof. Rabinovich's team, including Rundong Wang, Ridhima Bector and Wei Qiu, for the kind help and support that made my study in NTU a wonderful time. Besides, I would like to thank my collaborators, Xinghua Qu and Lev Raizman, for their valuable feedback and precious suggestions on my research studies. Many thanks to my friends, Yanling Li and Shuyang Ding, for the joy and the encouragement they gave me. I would like to thank my Thesis Advisory Committee (TAC) members, Prof. Sinno Jialin Pan and Dr. Fedor Duzhin, for their insightful comments and suggestions on my research work. I also want to thank Mr. Kesavan Asaithambi for his support in Computational Intelligence Lab.Most importantly, I would like to thank my parents, Zhaoxin Xu and Jiaojun Han, for their unconditional love and support throughout my life. I would like to express special thanks to my husband and best friend, Hantao Huang, for always being there for me and supporting me at all times. Having you in my life makes me feel incredibly lucky every single day. This thesis is dedicated to all of you.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: List Of Figuresmentioning

confidence: 99%

Section: List Of Figuresmentioning

confidence: 99%

See 3 more Smart Citations

Environment poisoning in reinforcement learning: attacks and resilience

Xu¹

View full text Add to dashboard Cite

show abstract

An Attackability Perspective on No-Sensing Adversarial Multi-player Multi-armed Bandits

Shi

Shen

2021

2021 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Most of the existing federated multi-armed bandits (FMAB) designs are based on the presumption that clients will implement the specified design to collaborate with the server. In reality, however, it may not be possible to modify the client's existing protocols. To address this challenge, this work focuses on clients who always maximize their individual cumulative rewards, and introduces a novel idea of "reward teaching", where the server guides the clients towards global optimality through implicit local reward adjustments. Under this framework, the server faces two tightly coupled tasks of bandit learning and target teaching, whose combination is non-trivial and challenging. A phased approach, called Teaching-After-Learning (TAL), is first designed to encourage and discourage clients' explorations separately. General performance analyses of TAL are established when the clients' strategies satisfy certain mild requirements. With novel technical approaches developed to analyze the warmstart behaviors of bandit algorithms, particularized guarantees of TAL with clients running UCB or ε-greedy strategies are then obtained. These results demonstrate that TAL achieves logarithmic regrets while only incurring logarithmic adjustment costs, which is order-optimal w.r.t. a natural lower bound. As a further extension, the Teaching-While-Learning (TWL) algorithm is developed with the idea of successive arm elimination to break the non-adaptive phase separation in TAL. Rigorous analyses demonstrate that when facing clients with UCB1, TWL outperforms TAL in terms of the dependencies on sub-optimality gaps thanks to its adaptive design. Experimental results demonstrate the effectiveness and generality of the proposed algorithms.

show abstract

ShieldFL: Mitigating Model Poisoning Attacks in Privacy-Preserving Federated Learning

Miao

et al. 2022

IEEE Trans.Inform.Forensic Secur.

View full text Add to dashboard Cite

Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks. The multi-agent structure addresses the major concern of data-hungry in traditional RL, while the federated mechanism protects the data privacy of individual agents. However, the federated mechanism also exposes the system to poisoning by malicious agents that can mislead the trained policy. Despite the advantage brought by FL, the vulnerability of Federated Reinforcement Learning (FRL) has not been well-studied before. In this work, we propose the first general framework to characterize FRL poisoning as an optimization problem constrained by a limited budget and design a poisoning protocol that can be applied to policy-based FRL and extended to FRL with actor-critic as a local RL algorithm by training a pair of private and public critics. We also discuss a conventional defense strategy inherited from FL to mitigate this risk. We verify our poisoning effectiveness by conducting extensive experiments targeting mainstream RL algorithms and over various RL OpenAI Gym environments covering a wide range of difficulty levels. Our results show that our proposed defense protocol is successful in most cases but is not robust under complicated environments. Our work provides new insights into the vulnerability of FL in RL training and poses additional challenges for designing robust FRL algorithms.

show abstract

The Sample Complexity of Teaching by Reinforcement on Q-Learning

Cited by 12 publications

References 23 publications

Environment poisoning in reinforcement learning: attacks and resilience

Environment poisoning in reinforcement learning: attacks and resilience

An Attackability Perspective on No-Sensing Adversarial Multi-player Multi-armed Bandits

ShieldFL: Mitigating Model Poisoning Attacks in Privacy-Preserving Federated Learning

Contact Info

Product

Resources

About