ShieldFL: Mitigating Model Poisoning Attacks in Privacy-Preserving Federated Learning

Abstract

Federated learning (FL) has become a popular tool for solving traditional Reinforcement Learning (RL) tasks. The multi-agent structure addresses the major concern of data-hungry in traditional RL, while the federated mechanism protects the data privacy of individual agents. However, the federated mechanism also exposes the system to poisoning by malicious agents that can mislead the trained policy. Despite the advantage brought by FL, the vulnerability of Federated Reinforcement Learning (FRL) has not been well-studied before. In this work, we propose the first general framework to characterize FRL poisoning as an optimization problem constrained by a limited budget and design a poisoning protocol that can be applied to policy-based FRL and extended to FRL with actor-critic as a local RL algorithm by training a pair of private and public critics. We also discuss a conventional defense strategy inherited from FL to mitigate this risk. We verify our poisoning effectiveness by conducting extensive experiments targeting mainstream RL algorithms and over various RL OpenAI Gym environments covering a wide range of difficulty levels. Our results show that our proposed defense protocol is successful in most cases but is not robust under complicated environments. Our work provides new insights into the vulnerability of FL in RL training and poses additional challenges for designing robust FRL algorithms.

Poisoning Attacks in Federated Learning: A Survey

“…Based on these data, we can draw the following conclusions. [38], [49], [51]. The biggest problem with the latter is that current research is based on the predictive accuracy of the global model on the validation dataset to determine whether an attack has been poisoned, and targeted poisoning attacks may evade this defense.…”

“…The first one is a similarity-based malicious model detection algorithm, which considers that statistical methods (such as euclidean distance [40], [52], cosine similarity [37], [38], k-means [46], Pearson correlation coefficient [49], [51], etc.) can be used to measure the differences between benign and malicious models [36], [80].…”

“…To solve this problem, [38], [49], [51] used homomorphic encryption to evaluate the local model securely.…”

“…Zhuoran et al [38] designed a privacy-preserving defense strategy called ShieldFL, which uses two-trapdoor homomorphic encryption to defend against poisoning attacks. They used two servers in ShieldFL to compute the cosine similarity of the encrypted local models.…”

“…Finally, S a removes the noise in the cosine similarity to get the final result. The method of [49], [51] is similar to [38]. The difference is that [49] and [51] calculate the Pearson correlation coefficient between the models.…”