Sigmoid Activation Implementation for Neural Networks Hardware Accelerators Based on Reconfigurable Computing Environments for Low-Power Intelligent Systems

Abstract

The remarkable results of applying machine learning algorithms to complex tasks are well known. They open wide opportunities in natural language processing, image recognition, and predictive analysis. However, their use in low-power intelligent systems is restricted because of high computational complexity and memory requirements. This group includes a wide variety of devices, from smartphones and Internet of Things (IoT)smart sensors to unmanned aerial vehicles (UAVs), self-driving cars, and nodes of Edge Computing systems. All of these devices have severe limitations to their weight and power consumption. To apply neural networks in these systems efficiently, specialized hardware accelerators are used. However, hardware implementation of some neural network operations is a challenging task. Sigmoid activation is popular in the classification problem and is a notable example of such a complex operation because it uses division and exponentiation. The paper proposes efficient implementations of this activation for dynamically reconfigurable accelerators. Reconfigurable computing environments (RCE) allow achieving reconfigurability of accelerators. The paper shows the advantages of applying such accelerators in low-power systems, proposes the centralized and distributed hardware implementations of the sigmoid, presents comparisons with the results of other studies, and describes application of the proposed approaches to other activation functions. Timing simulations of the developed Verilog modules show low delay (14–18.5 ns) with acceptable accuracy (average absolute error is 0.004).

Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators

“…The literature review allowed us to identify the following set of operations: "signal source" (SRC), "signal transfer" (TRS), "multiply and accumulate" (MAC), "parametric ReLU" (PRL) "maximum" (MAX), "minimum" (MIN), "gate" (GAT), "union" (U), "delay" (DEL), and "block" (BLK) (Figure 3). These operations are sufficient for implementing the key layers of neural networks (dense, convolution, pooling) as well as activations (sigmoid, tanh, ELU, and other) [21,[23][24][25]. This paper focuses on the implementation of SoftMax activation.…”

“…We briefly discuss the implementation of PLA in the RCE using the example of sigmoid activation described in detail in the paper [25]. This implementation is based on the approximation with equal subranges-the entire approximation range is divided into several subranges of equal width.…”

“…The a and b values are chosen to reduce the difference between the original and approximated functions in this subrange. The piecewise linear approximation is a popular approach to implement complex functions in hardware accelerators [25,35,36].…”

“…Thus, the desired algorithms must be implemented as a spatially distributed composition of the given operations. In the research, we propose the PE with 10 simple operations: signal source, signal transfer, multiply-accumulate, parametric ReLU, maximum, minimum, gate, union, delay, and block [21,[23][24][25]. There is currently no implementation of SoftMax activation based on spatial distribution and these operations.…”

“…This paper considers the implementation of SoftMax activation for the RCE. Activation has a lot in common with sigmoid activation, the implementation of which is presented in [25]. However, due to its higher computational complexity, SoftMax activation requires more optimizations, as will be discussed further.…”