WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization

Elthakeb, Ahmed T.; Pilligundla, Prannoy; Mireshghallah, Fatemehsadat; Elgindi, Tarek M.; Deledalle, Charles-Alban; Esmaeilzadeh, Hadi

doi:10.48550/arxiv.2003.00146

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Executing machine learning algorithms in reduced precision, i.e. quantization [30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45], and through compression, i.e. sparsity/pruning [46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64], enables us to optimize the NN significantly.…”

Section: Reduced Precision and Compressionmentioning

confidence: 99%

Smart sensors using artificial intelligence for on-detector electronics and ASICs

Carini¹,

Deptuch²,

Dickinson³

et al. 2022

Preprint

View full text Add to dashboard Cite

Cutting edge detectors push sensing technology by further improving spatial and temporal resolution, increasing detector area and volume, and generally reducing backgrounds and noise. This has led to a explosion of more and more data being generated in next-generation experiments. Therefore, the need for nearsensor, at the data source, processing with more powerful algorithms is becoming increasingly important to more efficiently capture the right experimental data, reduce downstream system complexity, and enable faster and lower-power feedback loops. In this paper, we discuss the motivations and potential applications for on-detector AI. Furthermore, the unique requirements of particle physics can uniquely drive the development of novel AI hardware and design tools. We describe existing modern work for particle physics in this area. Finally, we outline a number of areas of opportunity where we can advance machine learning techniques, codesign workflows, and future microelectronics technologies which will accelerate design, performance, and implementations for next generation experiments.

show abstract

Section: Reduced Precision and Compressionmentioning

confidence: 99%

Smart sensors using artificial intelligence for on-detector electronics and ASICs

Carini¹,

Deptuch²,

Dickinson³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Pouransari et al [31] proposed the least squares quantization method that searches proper scale factors for binary quantization. Elthakeb et al [32] attempted to quantize weights and activations by applying sinusoidal regularization. The regularization involves two hyperparameters that determine weight-quantization and bitwidth-regularization strengths.…”

Section: Related Workmentioning

confidence: 99%

CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method

Kim¹,

Jeong²

2021

Preprint

View full text Add to dashboard Cite

Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on the pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints -binary, ternary, one-bit shift, and two-bit shift weight constraints. As a posttraining method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For most cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6%, 74.4%, and 64.0% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions. The code for CBP is publicly available at https://github.com/dooseokjeong/CBP.

show abstract

WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization

Cited by 2 publications

References 9 publications

Smart sensors using artificial intelligence for on-detector electronics and ASICs

Smart sensors using artificial intelligence for on-detector electronics and ASICs

CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method

Contact Info

Product

Resources

About