Lightweight Approximation of Softmax Layer for On-Device Inference

Vasyltsov, Ihor; Chang, Wooseok

doi:10.1007/978-3-030-70296-0_41

Cited by 1 publication

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After optimization, their final design includes only shift operations, leading one detector, and adders. Finally, there are some extreme approximation cases represented in [13] and [29], where logarithmic computation and subtraction are skipped at all.…”

Section: Related Work and Key Contributionsmentioning

confidence: 99%

“…The original reciprocal exponentiation method was proposed in [29], where the authors used the inverse way of max-normalization and the reciprocal of exponential function as below:…”

Section: Normalization Of Reciprocal Exponentiationmentioning

confidence: 99%

“…Since we are interested in the methods which are not using a division operation, we have considered prior arts where logarithmic transformation was applied. Thus, we have adopted available pre-trained models from https://github.com/facebookresearch/detr by substituting a conventional softmax layer by the methods described in [32], [35], [29], and [13]. All computations were conducted in FP32 precision.…”

Section: A Appendixmentioning

confidence: 99%

“…There are several methods, which strongly approximate the original softmax formula and which showed good results for conventional CV tasks: [35], [29], and [13]. Note, that Eq.…”

Section: A Appendixmentioning

confidence: 99%

See 3 more Smart Citations

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism

Vasyltsov

Chang

2021

Preprint

Self Cite

View full text Add to dashboard Cite

There has been a rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). Previously, the softmax layer was not a main concern of DNN accelerating HW, because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms are widely used in various modern DNNs, a cost-efficient implementation of softmax layer is becoming very important. In this paper, we propose two methods to approximate softmax computation, which are based on the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input. We have validated the proposed technique over different AI tasks (object detection, machine translation, sentiment analysis, and semantic equivalence) and DNN models (DETR, Transformer, BERT) by a variety of benchmarks (COCO17, WMT14, WMT17, GLUE). We showed that 8-bit approximation allows to obtain acceptable accuracy loss below 1.0%.

show abstract

Section: Related Work and Key Contributionsmentioning

confidence: 99%

“…The original reciprocal exponentiation method was proposed in [29], where the authors used the inverse way of max-normalization and the reciprocal of exponential function as below:…”

Section: Normalization Of Reciprocal Exponentiationmentioning

confidence: 99%

Section: A Appendixmentioning

confidence: 99%

“…There are several methods, which strongly approximate the original softmax formula and which showed good results for conventional CV tasks: [35], [29], and [13]. Note, that Eq.…”

Section: A Appendixmentioning

confidence: 99%

See 2 more Smart Citations