DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks

Wiedemann, Simon; Kirchhoffer, Heiner; Matlage, Stefan; Haase, Paul; Marbán, Arturo; Marinč, Talmaj; Neumann, David; Nguyễn, Tùng Lâm; Schwarz, Heiko; Wiegand, Thomas; Marpe, Detlev; Samek, Wojciech

doi:10.1109/jstsp.2020.2969554

Cited by 86 publications

(69 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To obtain well-trained models that can still be employed productively, DL models have intensive memory and computational requirements due to their huge complexity and large numbers of parameters [ 193 , 194 ]. One of the fields that is characterized as data-intensive is the field of healthcare and environmental science.…”

Section: Challenges (Limitations) Of Deep Learning and Alternate Solumentioning

confidence: 99%

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

et al. 2021

View full text Add to dashboard Cite

In the last few years, the deep learning (DL) computing paradigm has been deemed the Gold Standard in the machine learning (ML) community. Moreover, it has gradually become the most widely used computational approach in the field of ML, thus achieving outstanding results on several complex cognitive tasks, matching or even beating those provided by human performance. One of the benefits of DL is the ability to learn massive amounts of data. The DL field has grown fast in the last few years and it has been extensively used to successfully address a wide range of traditional applications. More importantly, DL has outperformed well-known ML techniques in many domains, e.g., cybersecurity, natural language processing, bioinformatics, robotics and control, and medical information processing, among many others. Despite it has been contributed several works reviewing the State-of-the-Art on DL, all of them only tackled one aspect of the DL, which leads to an overall lack of knowledge about it. Therefore, in this contribution, we propose using a more holistic approach in order to provide a more suitable starting point from which to develop a full understanding of DL. Specifically, this review attempts to provide a more comprehensive survey of the most important aspects of DL and including those enhancements recently added to the field. In particular, this paper outlines the importance of DL, presents the types of DL techniques and networks. It then presents convolutional neural networks (CNNs) which the most utilized DL network type and describes the development of CNNs architectures together with their main features, e.g., starting with the AlexNet network and closing with the High-Resolution network (HR.Net). Finally, we further present the challenges and suggested solutions to help researchers understand the existing research gaps. It is followed by a list of the major DL applications. Computational tools including FPGA, GPU, and CPU are summarized along with a description of their influence on DL. The paper ends with the evolution matrix, benchmark datasets, and summary and conclusion.

show abstract

Section: Challenges (Limitations) Of Deep Learning and Alternate Solumentioning

confidence: 99%

Review of deep learning: concepts, CNN architectures, challenges, applications, future directions

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Figure 3g shows that the error, as expected, decreases with increasing N. However, for similar N, the error due to uniform quantization is significantly higher for normally distributed weights when compared to uniformly distributed weights. Since weight distributions in practical scenarios [55][56][57][58] are more likely to follow a normal distribution, uniform quantization can lead to significantly high inference inaccuracy. In order to mitigate the challenges associated with uniform quantization, we propose k-means clustering based quantization.…”

Section: G (S) G(s) G(s) G (S) G(s) G(s) G(s) G(s)mentioning

confidence: 99%

Graphene memristive synapses for high precision neuromorphic computing

2020

View full text Add to dashboard Cite

Memristive crossbar architectures are evolving as powerful in-memory computing engines for artificial neural networks. However, the limited number of non-volatile conductance states offered by state-of-the-art memristors is a concern for their hardware implementation since trained weights must be rounded to the nearest conductance states, introducing error which can significantly limit inference accuracy. Moreover, the incapability of precise weight updates can lead to convergence problems and slowdown of on-chip training. In this article, we circumvent these challenges by introducing graphene-based multi-level (>16) and non-volatile memristive synapses with arbitrarily programmable conductance states. We also show desirable retention and programming endurance. Finally, we demonstrate that graphene memristors enable weight assignment based on k-means clustering, which offers greater computing accuracy when compared with uniform weight quantization for vector matrix multiplication, an essential component for any artificial neural network.

show abstract

“…6 shows, that NNR achieves high compression even for high performance qualities and outperforms comparable methods significantly. As an example, cr = 2.98% for VGG16 NCTM, while for VGG16 BZIP cr = 7.74%, as given in [38]. The graphs also show that much higher compression ratios (far below 3%) of compressed NNs can be achieved for lossy coding scenarios, i.e.…”

Section: Compression Performancementioning

confidence: 83%

“…In the NNR standard, the quantization indices that are output by the selected quantization method and all other syntax elements are entropy coded, using DeepCABAC [38]. This method is based on context-adaptive binary arithmetic coding (CABAC) [39], [40], which was originally developed and optimized for video compression.…”

Section: Entropy Codingmentioning

confidence: 99%