Computational Cost Reduction in Learned Transform Classifications

Machado, Emerson Lopes; Miosso, Cristiano Jacques; Borries, Ricardo von; Coutinho, Murilo G.; Berger, Pedro de Azevedo; Marques, Thiago; Jacobi, Ricardo Pezzuol

doi:10.48550/arxiv.1504.06779

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The approach still requires a real-valued, full precision training phase, however, so the benefits of reducing computations does not apply to training. Similarly, Machado et al (2015) manage to get acceptable accuracy on sparse representation classification by replacing all floating-point multiplications by integer shifts. Bit-stream networks (Burge et al, 1999) also provides a way of binarizing neural network connections, by substituting weight connections with logical gates.…”

Section: Introductionmentioning

confidence: 99%

Neural Networks with Few Multiplications

Lin¹,

Courbariaux²,

Memisevic³

et al. 2015

Preprint

View full text Add to dashboard Cite

For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardwarefriendly training of neural networks.

show abstract

Section: Introductionmentioning

confidence: 99%