Finite Bit Quantization for Decentralized Learning Under Subspace Constraints

Abstract

In this paper, we consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus optimization as special case, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. In order to cope with communication constraints, we propose and study a quantized differential based approach where the communicated estimates among agents are quantized. The analysis shows that, under some general conditions on the quantization noise, and for sufficiently small step-sizes µ, the strategy is stable in the meansquare error sense. The analysis also reveals the influence of the gradient and quantization noises on the performance.

i-WSN League: Clustered Distributed Learning in Wireless Sensor Networks

“…In order to reduce the communication bottleneck, the model updates are compressed (quantized or sparsified) by the nodes. Similar approach is proposed in [19] for decentralized learning in which the communication estimates among the users are quantized. Similar compression and quantization solutions are studied in [20] and [21] to solve distributed optimization problems with communication constraints.…”