FPGA Implementation for the Sigmoid with Piecewise Linear Fitting Method Based on Curvature Analysis

Abstract

The sigmoid activation function is popular in neural networks, but its complexity limits the hardware implementation and speed. In this paper, we use curvature values to divide the sigmoid function into different segments and employ the least squares method to solve the expressions of the piecewise linear fitting function in each segment. We then adopt an optimization method with maximum absolute errors and average absolute errors to select an appropriate function expression with a specified number of segments. Finally, we implement the sigmoid function on the field-programmable gate array (FPGA) development platform and apply parallel operations of arithmetic (multiplying and adding) and range selection at the same time. The FPGA implementation results show that the clock frequency of our design is up to 208.3 MHz, while the end-to-end latency is just 9.6 ns. Our piecewise linear fitting method based on curvature analysis (PWLC) achieves recognition accuracy on the MNIST dataset of 97.51% with a deep neural network (DNN) and 98.65% with a convolutional neural network (CNN). Experimental results demonstrate that our FPGA design of sigmoid function can obtain the lowest latency, reduce absolute errors, and achieve high recognition accuracies, while the hardware cost is acceptable in practical applications.

Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays

“…Both functions contain exponential functions, making it difficult to implement them on resource-constrained hardware and requiring a large chip area [55]. Therefore, function approximation techniques are required in place of the exact functions to realize them in the FPGA, and to reduce the overall computational complexity [55], [56], [57], [58]. In this article, we focus on approximating the sigmoid and tanh.…”

“…The PWL approximation, introduced in [60], is a combination of linear segments that approximates the activation or nonlinear function [56], [61]. Increasing the number of linear segments to represent the nonlinear function allows us to achieve better accuracy.…”

“…Increasing the number of linear segments to represent the nonlinear function allows us to achieve better accuracy. The PWL approximation is a promising method to reach a higher processing speed since it consumes fewer resources on FPGA 8 compared to the Taylor approximation: to reach higher accuracy, the Taylor approach fits the nonlinear function with high-order expressions, which results in the consumption of resources, while the PWL can reach the same level of accuracy with the use of more segments, but without employing highorder operations [56].…”