EFA-Trans: An Efficient and Flexible Acceleration Architecture for Transformers

Yang, Xin; Su, Tao

doi:10.3390/electronics11213550

Cited by 6 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the hardware implementation of this function is resource-intensive, researchers have proposed various approaches to optimize it. In papers [29][30][31][32], the exponential function is implemented by storing its points in an internal memory of FPGA (look-up table, LUT). This is a simple and effective solution.…”

Section: Current State Of Hardware Implementation Of the Softmax Acti...mentioning

confidence: 99%

See 1 more Smart Citation

Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators

Shatravin,

Shashev,

Shidlovskiy

2023

Applied Sciences

View full text Add to dashboard Cite

In recent decades, machine-learning algorithms have been extensively utilized to tackle various complex tasks. To achieve the high performance and efficiency of these algorithms, various hardware accelerators are used. Typically, these devices are specialized for specific neural network architectures and activation functions. However, state-of-the-art complex autonomous and mobile systems may require different algorithms for different tasks. Reconfigurable accelerators can be used to resolve this problem. They possess the capability to support diverse neural network architectures and allow for significant alterations to the implemented model at runtime. Thus, a single device can be used to address entirely different tasks. Our research focuses on dynamically reconfigurable accelerators based on reconfigurable computing environments (RCE). To implement the required neural networks on such devices, their algorithms need to be adapted to the homogeneous structure of RCE. This article proposes the first implementation of the widely used SoftMax activation for hardware accelerators based on RCE. The implementation leverages spatial distribution and incorporates several optimizations to enhance its performance. The timing simulation of the proposed implementation on FPGA shows a high throughput of 1.12 Gbps at 23 MHz. The result is comparable to counterparts lacking reconfiguration capability. However, this flexibility comes at the expense of the increased consumption of logic elements.

show abstract

Section: Current State Of Hardware Implementation Of the Softmax Acti...mentioning

confidence: 99%

“…The wide input range of the SoftMax activation increases the complexity of its hardware implementation. To reduce the range, most of the mentioned papers suggest subtracting the maximum input value from all inputs [29,31,32,34]. This is possible due to Equation (2).…”

Section: Current State Of Hardware Implementation Of the Softmax Acti...mentioning

confidence: 99%