A Customized Convolutional Neural Network Design Using Improved Softmax Layer for Real-time Human Emotion Recognition

Wang, Kai-Yen; Huang, Yuan Ming; Ho, Yun-Lung; Fang, Wai-Chi

doi:10.1109/aicas.2019.8771616

Cited by 9 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This value can be approximated to z j − m which is used for further calculation. Similar to this, Wang et al proposed another method with EEG signals [22]. This version of algorithm for an optimized softmax layer implementation and its hardware designs are suitable for human emotion recognition algorithm while the input is EEG signal.…”

Section: Kouretas Et Al [21]mentioning

confidence: 97%

See 1 more Smart Citation

From DNNs to GANs: Review of efficient hardware architectures for deep learning

Bhattacharya¹

2021

Preprint

View full text Add to dashboard Cite

In recent times, the trend in very large scale integration (VLSI) industry is multi-dimensional, e.g. reduction of energy consumption, occupancy of less space, precise result, less power dissipation, faster response etc. To meet these needs, the hardware architecture should be reliable and robust to these problems. Recently, neural network and deep learning has been started to impact the present research paradigm significantly which consists of parameters in the order of millions, nonlinear function for activation, convolutional operation for feature extraction, softmax regression for classification, generative adversarial networks, etc. These operations involve huge calculation and memory overhead. Presently available DSP processors are incapable of performing these operations and they most face the problems e.g. memory overhead, performance drop and compromised accuracy. Moreover, if a huge silicon area is powered to accelerate the operation using parallel computation, the IC's will be having significant chance of burning out due to the considerable generation of heat. Hence, novel dark silicon constraint is developed to reduce the heat dissipation without sacrificing the accuracy. Similarly, different algorithms have been adapted to design a DSP processor compatible for fast performance in neural network, activation function, convolutional neural network and generative adversarial network. In this review, we illustrate the recent developments in hardware for accelerating the efficient implementation of deep learning networks with enhanced performance. The techniques investigated in this review are expected to direct future research challenges of hardware optimization for high-performance computations.

show abstract

Section: Kouretas Et Al [21]mentioning

confidence: 97%

“…Wang et al [22] This hardware design adds threshold layers to accelerate the training speed and replace the Euler's base value with a dynamic base value to improve the network accuracy. [19].…”

Section: Kouretas Et Al [21]mentioning

confidence: 99%

From DNNs to GANs: Review of efficient hardware architectures for deep learning

Bhattacharya¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Moreover, to obtain the final value of softmax the division is still used. In [31] it is proposed to add threshold layers to accelerate the training speed and replace the Euler's base value with a dynamic base value to improve the network accuracy. Such approach allowed to save up to 15% of training model convergence time and also increase by 3 to 5% the average accuracy.…”

Section: Related Work and Key Contributionsmentioning

confidence: 99%

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

Vasyltsov

Chang

2021

Preprint

View full text Add to dashboard Cite

There has been a rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). Previously, the softmax layer was not a main concern of DNN accelerating HW, because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms are widely used in various modern DNNs, a cost-efficient implementation of softmax layer is becoming very important. In this paper, we propose two methods to approximate softmax computation, which are based on the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input. We have validated the proposed technique over different AI tasks (object detection, machine translation, sentiment analysis, and semantic equivalence) and DNN models (DETR, Transformer, BERT) by a variety of benchmarks (COCO17, WMT14, WMT17, GLUE). We showed that 8-bit approximation allows to obtain acceptable accuracy loss below 1.0%.

show abstract

Lightweight Approximation of Softmax Layer for On-Device Inference

Vasyltsov

Chang²

2021

Transactions on Computational Science and Computational Intelligence

View full text Add to dashboard Cite

A Customized Convolutional Neural Network Design Using Improved Softmax Layer for Real-time Human Emotion Recognition

Cited by 9 publications

References 8 publications

From DNNs to GANs: Review of efficient hardware architectures for deep learning

From DNNs to GANs: Review of efficient hardware architectures for deep learning

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

Lightweight Approximation of Softmax Layer for On-Device Inference

Contact Info

Product

Resources

About