Weight Quantization for Multi-layer Perceptrons Using Soft Weight Sharing

Köksal, Fatih; Alpaydın, Ethem; Dündar, Günhan

doi:10.1007/3-540-44668-0_30

Cited by 7 publications

(2 citation statements)

References 9 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The studies in [9,10,11] discretize the weights of a neural network according to the weights' ranges. The methods in [12] and [13] use uniform scalar parameter quantization to implement fixed-point versions of the networks.…”

Section: Introductionmentioning

confidence: 99%

Ranking the parameters of deep neural networks using the fisher information

Tao

Berisha

Woolf

et al. 2016

2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The large number of parameters in deep neural networks (DNNs) often makes them prohibitive for low-power devices, such as field-programmable gate arrays (FPGA). In this paper, we propose a method to determine the relative importance of all network parameters by measuring the amount of information that the network output carries about each of the parameters -the Fisher Information. Based on the importance ranking, we design a complexity reduction scheme that discards unimportant parameters and assigns more quantization bits to more important parameters. For evaluation, we construct a deep autoencoder and learn a non-linear dimensionality reduction scheme for accelerometer data measuring the gait of individuals with Parkinson's disease. Experimental results confirm that the proposed ranking method can help reduce the complexity of the network with minimal impact on performance.

show abstract

Section: Introductionmentioning

confidence: 99%

Ranking the parameters of deep neural networks using the fisher information

Tao

Berisha

Woolf

et al. 2016

2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…For fixed-point implementations of DNNs, parameter quantization is also required. The studies in [19], [20] discretized the weights of a neural network according to the range of the weights. The methods in [21] and [22] used uniform scalar parameter quantization to implement fixed-point versions of the networks.…”

Section: Introductionmentioning

confidence: 99%

Reducing the Model Order of Deep Neural Networks Using Information Theory

Tao

Berisha²,

Cao

et al. 2016

2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI)

View full text Add to dashboard Cite

Abstract-Deep neural networks are typically represented by a much larger number of parameters than shallow models, making them prohibitive for small footprint devices. Recent research shows that there is considerable redundancy in the parameter space of deep neural networks. In this paper, we propose a method to compress deep neural networks by using the Fisher Information metric, which we estimate through a stochastic optimization method that keeps track of secondorder information in the network. We first remove unimportant parameters and then use non-uniform fixed point quantization to assign more bits to parameters with higher Fisher Information estimates. We evaluate our method on a classification task with a convolutional neural network trained on the MNIST data set. Experimental results show that our method outperforms existing methods for both network pruning and quantization.

show abstract

Defect‐tolerant nanoelectronic pattern classifiers

Lee

Likharev

2007

Circuit Theory & Apps

View full text Add to dashboard Cite

SUMMARYMixed-signal neuromorphic networks ('CrossNets'), based on hybrid CMOS/nanodevice circuits, may provide unprecedented performance for important pattern classification tasks. The synaptic weights necessary for such tasks may be imported from an external 'precursor' network with either continuous or discrete synaptic weights (in the former case, with the quantization-'clipping'-due to the binary character of the elementary synaptic nanodevices-latching switches.) Alternatively, the weights may be adjusted 'in situ' (inside the CrossNet) using a pseudo-stochastic method, or set-up using a mixed-mode method partly employing external circuitry. Our calculations have shown that CrossNet pattern classifiers, using any of these synaptic weight adjustment methods, may be remarkably resilient. For example, in a CrossNet with synapses in the form of two small square arrays with 4 × 4 nanodevices each, the resulting weight discreteness may have a virtually negligible effect on the classification fidelity, while the fraction of defective devices which affects the performance substantially ranges from ∼20% to as high as 90% (!), depending on the training method.

show abstract

Weight Quantization for Multi-layer Perceptrons Using Soft Weight Sharing

Cited by 7 publications

References 9 publications

Ranking the parameters of deep neural networks using the fisher information

Ranking the parameters of deep neural networks using the fisher information

Reducing the Model Order of Deep Neural Networks Using Information Theory

Defect‐tolerant nanoelectronic pattern classifiers

Contact Info

Product

Resources

About