CoCo-FL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization

Abstract

Devices participating in federated learning (FL) typically have heterogeneous communication and computation resources. However, all devices need to finish training by the same deadline dictated by the server when applying synchronous FL, as we consider in this paper. Reducing the complexity of the trained neural network (NN) at constrained devices, i.e., by dropping neurons/filters, is insufficient as it tightly couples reductions in communication and computation requirements, wasting resources. Quantization has proven effective to accelerate inference, but quantized training suffers from accuracy losses. We present a novel mechanism that quantizes during training parts of the NN to reduce the computation requirements, freezes them to reduce the communication and computation requirements, and trains the remaining parts in full precision to maintain a high convergence speed and final accuracy. Using this mechanism, we present the first FL technique that independently optimizes for specific communication and computation constraints in FL: CoCo-FL. We show that CoCo-FL reaches a much higher convergence speed than the state of the art and a significantly higher final accuracy.

Federated Learning for Computationally Constrained Heterogeneous Devices: A Survey

“…In ZeroFL [79], dropout masks in combination with sparse convolutions are used to lower the computational complexity in training (FLOPS) and reduce the communication volume, although special hardware and software support is required to enable real-world gains. Lastly, in CoCoFL [77], a technique is presented that does not use subsets of an NN for training. Instead, only for some layers per round gradients get calculated while the remainder of the layers are frozen.…”

“…The attributes scale and granularity are often neglected, are hidden behind the technique, and lack discussion in the papers. The reported scale in the resources supported by the techniques ranges from 4× − 25× [12,41,52,61,71,77,79,80,85,101] up to 100× − 250× [25,87], yet it remains unclear whether training at such high scales is still effective. Hence, while all approaches show the effectiveness of their solution in certain scenarios, it often remains unclear whether devices with low resources or stale devices can make a meaningful contribution that advances the global model.…”

“…the resource model. This is especially the case in techniques using subsets, where some model resources in terms of power usage [104], while others count the number of parameters [2,25,41,79], the number of multiplyaccumulate-operations [41,80], or the required training time [77]. Therefore, the supported scale of heterogeneity by the techniques is not comparable.…”