Communication Cost Reduction with Partial Structure in Federated Learning

Abstract

Federated learning is a distributed learning algorithm designed to train a single server model on a server using different clients and their local data. To improve the performance of the server model, continuous communication with clients is required, and since the number of clients is very large, the algorithm must be designed in consideration of the cost required for communication. In this paper, we propose a method for distributing a model with a structure different from that of the server model, distributing a model suitable for clients with different data sizes, and training a server model using the reconstructed model trained by the client. In this way, the server model deploys only a subset of the sequential model, collects gradient updates, and selectively applies updates to the server model. This method of delivering the server model at a lower cost to clients who only need smaller models can reduce the communication cost of training server models compared to standard methods. An image classification model was designed to verify the effectiveness of the proposed method via three data distribution situations and two datasets, and it was confirmed that training was accomplished only with a cost 0.229 times smaller than the standard method.

Efficient Gradient Updating Strategies with Adaptive Power Allocation for Federated Learning over Wireless Backhaul

“…We note that, in many cases, data sharing is not free from security, regulatory and privacy issues [8]. We also note that the communication cost for the centralized learning depends on the number/size of the collected data [22,23]. In contrast, the communication cost for the federated learning is independent with the data size, but depends on the CNN architecture (specifically, the number of weights in the CNN).…”

Fed2A: Federated Learning Mechanism in Asynchronous and Adaptive Modes

“…Finally, as for the last method, each client can train and upload a subset of the model (i.e., the shallow and deep layers can be split) to save related costs. Kang et al [21], for example, distributed a model according to data sizes of clients, and Wang et al [22] divided the global model into branches according to sample categories.…”

“…Caldas et al [12] Dropout compression Wang et al [13] SVD compression Sattler et al [14] Sparse compression Asad et al [15] Sparce compression Lu et al [16] Threshold compression Zhu et al [17] Frequency reduction Huang et al [18] Frequency reduction Wang et al [19] Frequency reduction Wang et al [20] Frequency reduction Kang et al [21] Training split Wang et al [22] Training split…”

Federated Learning Design and FunctionalModels: Survey

“…Continuous communication in federated learning requires an extensive number of communication rounds between the client and server due to the sizeable data and number of clients. This necessitates the use of effective algorithms and models to reduce the cost between the client and the global model [120]. To that end, a number of approaches have been used to decrease communication rounds and improve performance.…”