Federated Learning with Differential Privacy: Algorithms and Performance Analysis

Abstract

Federated learning (FL), as a manner of distributed machine learning, is capable of significantly preserving clients' private data from being exposed to external adversaries. Nevertheless, private information can still be divulged by analyzing on the differences of uploaded parameters from clients, e.g., weights trained in deep neural networks. In this paper, to effectively prevent information leakage, we propose a novel framework based on the concept of differential privacy (DP), in which artificial noises are added to the parameters at the clients side before aggregating, namely, noising before model aggregation FL (NbAFL). First, we prove that the NbAFL can satisfy DP under distinct protection levels by properly adapting different variances of artificial noises. Then we develop a theoretical convergence bound of the loss function of the trained FL model in the NbAFL. Specifically, the theoretical bound reveals the following three key properties: 1) There is a tradeoff between the convergence performance and privacy protection levels, i.e., a better convergence performance leads to a lower protection level; 2) Given a fixed privacy protection level, increasing the number N of overall clients participating in FL can improve the convergence performance; 3) There is an optimal number of maximum aggregation times (communication rounds) in terms of convergence performance for a given protection level. Furthermore, we propose a K-random scheduling strategy, where K (1 < K < N ) clients are randomly selected from the N overall clients to participate in each aggregation. We also develop the corresponding convergence bound of the loss function in this case and the K-random scheduling strategy can also retain the above three properties. Moreover, we find that there is an optimal K that achieves the best convergence performance at a fixed privacy level. Evaluations demonstrate that our theoretical results are consistent with simulations, thereby facilitating the designs on various privacy-preserving FL algorithms with different tradeoff requirements on convergence performance and privacy levels.

A Compressive Sensing Approach for Federated Learning over Massive MIMO Communication Systems

“…where the second equality is obtained from (26). By using the expression in (36), computing the estimated transmitted signal x(i) is not required at each iteration.…”

“…There also exist recent studies that seek to optimize federated learning over wireless com-munication systems [14]- [26]. In these studies, the physical characteristics of wireless networks are considered to improve the performance of federated learning under a practical scenario.…”

The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learning

“…Precautions like adding noise or quantization can somewhat reduce the risk. However, some experiments [318] have shown that there exists trade-offs between privacy and performance. Therefore, it is essential to understand this trade-off better and propose methods that can improve both simultaneously.…”

“…Therefore, it is essential to understand this trade-off better and propose methods that can improve both simultaneously. Differential privacy may be a valuable tool along this line [318]. iv.…”

FedOCR: Communication-Efficient Federated Learning for Scene Text Recognition

“…Following McMahan et al [23], many researchers are working on improving the federated learning with more efficient parameter transmission and higher-level privacy-preserving. To improve privacy security, Wei et al [36] propose a federated learning framework based on differential privacy, in which artificial noises are added to the local parameters of participants before the model aggregation. To improve communication efficiency, Reisizadeh et al [27] propose a communication-efficient federated learning method with periodic averaging and quantization.…”