Finite-Bit Quantization For Distributed Algorithms With Linear Convergence

Michelusi, Nicolò; Scutari, Gesualdo; Lee, Chang-Shen

doi:10.48550/arxiv.2107.11304

Cited by 2 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, it was shown in [33] that, for a fixed number of output levels, the choice in ( 16) maximizes the range of the input variable while still satisfying the error bound (17). This property justifies the efficiency of the choice (16) in terms of quantization bit rate.…”

Section: )mentioning

confidence: 89%

“…As explained in [30], deterministic quantizers can lead to severe estimation biases in inference problems. To overcome this issue, randomized quantizers Q(•) are commonly used to compensate for the bias (on average, over time) [12], [29]- [33]. This section is devoted to describing the general class of randomized quantizers considered throughout the study.…”

Section: Randomized Quantizersmentioning

confidence: 99%

“…Two useful results were established in [33] regarding this choice. First, it was shown that, by setting the constants a and b according to:…”

Section: Randomized Quantizersmentioning

confidence: 99%

“…Designers are typically limited by an upload bandwidth of 1MB/s or less [2]. While there have been significant works in the literature on solving optimization and inference problems in a decentralized manner [8]- [13], [13]- [33], and with some exceptions [12], [25]- [33], the large majority of these works is not tailored to the specific challenge of limited communication capabilities.…”

Section: Introductionmentioning

confidence: 99%

“…While there exist several useful works in the literature that study decentralized learning approaches in the presence of differential quantization [12], [29]- [34], these works investigate standard consensus or single-task optimization, and do not consider the subspace constrained formulation (3) or multitask variants. Moreover, with some exceptions that consider deterministic optimization [33], [34], the analyses conducted in these works assume that some quantities (e.g., the norm or some components of the vector to be quantized) are represented with very high precision (e.g., machine precision) and the associated quantization error is neglected. On the other hand, the analysis in the current work is not limited to standard consensus optimization and does not assume a machine precision representation of some quantities -a detailed discussion of related work is provided in Sec.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Quantization for decentralized learning under subspace constraints

Nassif¹,

Vlaski²,

Carpentiero³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. In order to cope with communication constraints, we propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates before communicating with their neighbors. The analysis shows that, under some general conditions on the quantization noise, and for sufficiently small step-sizes µ, the strategy is stable both in terms of mean-square error and average bit rate: by reducing µ, it is possible to keep the estimation errors small (on the order of µ) without increasing indefinitely the bit rate as µ → 0. Simulations illustrate the theoretical findings and the effectiveness of the proposed approach, revealing that decentralized learning is achievable at the expense of only a few bits.

show abstract

Section: )mentioning

confidence: 89%

Section: Randomized Quantizersmentioning

confidence: 99%

“…Two useful results were established in [33] regarding this choice. First, it was shown that, by setting the constants a and b according to:…”

Section: Randomized Quantizersmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Quantization for decentralized learning under subspace constraints

Nassif¹,

Vlaski²,

Carpentiero³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

A Compressed Gradient Tracking Method for Decentralized Optimization With Linear Convergence

Liao

Huang

et al. 2022

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

Communication compression techniques are of growing interests for solving the decentralized optimization problem under limited communication, where the global objective is to minimize the average of local cost functions over a multiagent network using only local computation and peer-to-peer communication. In this article, we propose a novel compressed gradient tracking algorithm (C-GT) that combines gradient tracking technique with communication compression. In particular, C-GT is compatible with a general class of compression operators that unifies both unbiased and biased compressors. We show that C-GT inherits the advantages of gradient tracking-based algorithms and achieves linear convergence rate for strongly convex and smooth objective functions. Numerical examples complement the theoretical findings and demonstrate the efficiency and flexibility of the proposed algorithm.

show abstract

Finite-Bit Quantization For Distributed Algorithms With Linear Convergence

Cited by 2 publications

References 0 publications

Quantization for decentralized learning under subspace constraints

Quantization for decentralized learning under subspace constraints

A Compressed Gradient Tracking Method for Decentralized Optimization With Linear Convergence

Contact Info

Product

Resources

About