How to Train your DNN: The Network Operator Edition

Chang, Michael Alan; Bottini, Domenic; Jian, Lisa; Kumar, Pankaj; Panda, Aurojit; Shenker, Scott

doi:10.48550/arxiv.2004.10275

Search citation statements

Order By: Relevance

Paper Sections

Select...

Introduction1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2023

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, while performance continuously improves, the models become larger with a massive increase in the number of parameters. In fact, modern Neural Network (NN) models may have billions and even trillions of parameters, which makes the deployment of these models a challenging task (Chang et al, 2020). One way to mitigate this issue is compressing the model's parameters to reduce its overall memory footprint while satisfying an accuracy constraint.…”

Section: Introductionmentioning

confidence: 99%

Rotation Invariant Quantization for Model Compression

Kampeas¹,

Nahshan²,

Kremer³

et al. 2023

Preprint

View full text Add to dashboard Cite

Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion tradeoff for NN model compression. First, we suggest a Rotation-Invariant Quantization (RIQ) technique that utilizes a single parameter to quantize the entire NN model, yielding a different rate at each layer, i.e., mixed-precision quantization. Then, we prove that our rotation-invariant approach is optimal in terms of compression. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates ×19.4 and ×52.9 compression ratios on pretrained VGG dense and pruned models, respectively, with < 0.4% accuracy degradation. Code: https://github.com/ehaleva/RIQ.

show abstract

Section: Introductionmentioning

confidence: 99%