vqSGD: Vector Quantized Stochastic Gradient Descent

Gandikota, Venkata; Kane, Daniel M.; Maity, Raj Kumar; Mazumdar, Arya

doi:10.1109/tit.2022.3161620

Cited by 23 publications

(29 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…non-subtractive dither quantization. In some literature, see [9], the dimensionality-reduction is performed on the whole gradient vector. More naively, constraints in the communication capabilities between the remote users and the PS can also be addressed by restricting the number of communication iterations between gradient updates [13], [14].…”

Section: A Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?

Salehkalaibar¹,

Rini²

2022

Preprint

View full text Add to dashboard Cite

In federated learning (FL), a global model is trained at a Parameter Server (PS) by aggregating model updates obtained from multiple remote learners. Critically, the communication between the remote users and the PS is limited by the available power for transmission, while the transmission from the PS to the remote users can be considered unbounded. This gives rise to the distributed learning scenario in which the updates from the remote learners have to be compressed so as to meet communication rate constraints in the uplink transmission toward the PS. For this problem, one would like to compress the model updates so as to minimize the resulting loss in accuracy. In this paper, we take a rate-distortion approach to answer this question for the distributed training of a deep neural network (DNN). In particular, we define a measure of the compression performance, the per-bit accuracy, which addresses the ultimate model accuracy that a bit of communication brings to the centralized model. In order to maximize the per-bit accuracy, we consider modeling the gradient updates at remote learners as a generalized normal distribution. Under this assumption on the model update distribution, we propose a class of distortion measures for the design of quantizer for the compression of the model updates. We argue that this family of distortion measures, which we refer to as "M -magnitude weighted L2" norm, capture the practitioner intuition in the choice of gradient compressor. Numerical simulations are provided to validate the proposed approach.

show abstract

Section: A Literature Reviewmentioning

confidence: 99%

“…Another approach for gradient compression is uniform quantization with non-subtractive dithering [9], [23]- [25], for unbiased quantization. This approach finds its theoretical foundations in works such as [26, eq.…”

Section: B Gradient Distortionmentioning

confidence: 99%

Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?

Salehkalaibar¹,

Rini²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…distributed SGD and FL systems, e.g., sketches, can achieve provable privacy benefits [125], [180]. Therefore, a novel sketch-based framework (DiffSketch) for distributed learning has been proposed, improving absolute test accuracy while offering a certain privacy guarantees and communication compression.…”

Section: • Model Compression Model Compression Techniques Formentioning

confidence: 99%

“…Therefore, a novel sketch-based framework (DiffSketch) for distributed learning has been proposed, improving absolute test accuracy while offering a certain privacy guarantees and communication compression. Moreover, the work in [180] has presented a family of vector quantization schemes, termed Vector-Quantized Stochastic Gradient Descent (VQSGD), which provides an asymptotic reduction in the communication cost and automatic privacy guarantees. • Encryption.…”

Section: • Model Compression Model Compression Techniques Formentioning

confidence: 99%

Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

Ma¹,

Li²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Motivated by the advancing computational capacity of distributed end-user equipments (UEs), as well as the increasing concerns about sharing private data, there has been considerable recent interest in machine learning (ML) and artificial intelligence (AI) that can be processed on on distributed UEs. Specifically, in this paradigm, parts of an ML process are outsourced to multiple distributed UEs, and then the processed ML information is aggregated on a certain level at a central server, which turns a centralized ML process into a distributed one, and brings about significant benefits. However, this new distributed ML paradigm raises new risks of privacy and security issues. In this paper, we provide a survey of the emerging security and privacy risks of distributed ML from a unique perspective of information exchange levels, which are defined according to the key steps of an ML process, i.e.: i) the level of preprocessed data, ii) the level of learning models, iii) the level of extracted knowledge and, iv) the level of intermediate results. We explore and analyze the potential of threats for each information exchange level based on an overview of the current state-of-the-art attack mechanisms, and then discuss the possible defense methods against such threats. Finally, we complete the survey by providing an outlook on the challenges and possible directions for future research in this critical area.

show abstract

“…Sparsification methods highly rely on fixed or variable rate elimination of the dimensions of the gradient vector based on a specific criterion such as magnitude or variance [2]- [5]. This is while quantization methods focus on discretizing the gradient vectors through dimension-wise [6] or vector quantization [7]. Data privacy in FL model has been mainly addressed through DP as a context-free notion evaluating the privacy loss incurred by membership attacks to extract information about the individual sample points [8].…”

mentioning

confidence: 99%

Wireless Federated Learning with Limited Communication and Differential Privacy

Sonee¹,

Rini²,

Huang³

2021

Preprint

View full text Add to dashboard Cite

This paper investigates the role of dimensionality reduction in efficient communication and differential privacy (DP) of the local datasets at the remote users for over-the-air computation (AirComp)-based federated learning (FL) model. More precisely, we consider the FL setting in which clients are prompted to train a machine learning model by simultaneous channel-aware and limited communications with a parameter server (PS) over a Gaussian multiple-access channel (GMAC), so that transmissions sum coherently at the PS globally aware of the channel coefficients. For this setting, an algorithm is proposed based on applying (i) federated stochastic gradient descent (FedSGD) for training the minimum of a given loss function based on the local gradients, (ii) Johnson-Lindenstrauss (JL) random projection for reducing the dimension of the local updates and (iii) artificial noise to further aid user's privacy. For this scheme, our results show that the local DP performance is mainly improved due to injecting noise of greater variance on each dimension while keeping the sensitivity of the projected vectors unchanged. This is while the convergence rate is slowed down compared to the case without dimensionality reduction. As the performance outweighs for the slower convergence, the trade-off between privacy and convergence is higher but is shown to lessen in high-dimensional regime yielding almost the same trade-off with much less communication cost.

show abstract

vqSGD: Vector Quantized Stochastic Gradient Descent

Cited by 23 publications

References 19 publications

Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?

Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?

Trusted AI in Multi-agent Systems: An Overview of Privacy and Security for Distributed Learning

Wireless Federated Learning with Limited Communication and Differential Privacy

Contact Info

Product

Resources

About