Implementation of a digital neuron with nonlinear activation function using piecewise linear approximation technique

Mishra, Amit; Din, Zaheer ud; Raj, Krishna

doi:10.1109/icm.2007.4497664

Cited by 12 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the first case, the design optimization depends on the type of memories available in the device; in the second case, it is possible to develop logical simplifications. Other approximation form is the Piece Wise Lineal (PWL), which approaches each section with a straight line, in this case a multiplication and a sum are necessary [18,19,21,38,44,45,63,66,68,69,70].…”

Section: State Of the Art Of Hardware Implementation For Sigmoidal Fumentioning

confidence: 99%

Fixed-point design methodology of the hyperbolic tangent function using lookup tables with performance evaluation on FPGA

Suárez¹

2018

EasyChair Preprints

View full text Add to dashboard Cite

The sigmoid functions, including the hyperbolic tangent, are widely used in artificial neural networks. Usually, artificial neural networks are implemented on a computer, in training and testing phases, using floating point arithmetic. In some situations it is necessary to improve the performance of neural networks, in such cases the implementations are designed for specific digital devices, using fixed point arithmetic. This strategy increases the speed, and reduces the hardware resources and consumed power. The sigmoid functions are non-linear systems because include exponentials and divisions operations, so they are the bottleneck of the artificial neuron design and cannot be implemented directly in fixed-point format. For this reason, several methods of approximation are used, mainly based on piecewise approximations. One of the most common technique is to use Look-up Tables, which store samples of the function. This method needs a lot of hardware resources, but it reaches the highest speed. The first objective, using lookup table approximations, is to explore the set of solutions and study the effect of the number of bits. This can only be achieved if an advanced design methodology is used for experimentation, a second objective is to expose this fast and flexible design method. Thirdly, the measure of functionality associated with the error is reviewed. A new functionality measure is proposed associated with the concept of signal to noise power relation. The new measure allows compare different approximations; besides, some linearities are observed against the number of bits. Afterwards, the implementations are developed on some Field Programmable Gate Arrays and the performances are evaluated: hardware resources, speed and consumed power. Again, some linear behaviours are observed. Finally, a quality factor is proposed, which includes functionality and physical performances, this factor allows comparison between different implementations.

show abstract

Section: State Of the Art Of Hardware Implementation For Sigmoidal Fumentioning

confidence: 99%

Fixed-point design methodology of the hyperbolic tangent function using lookup tables with performance evaluation on FPGA

Suárez¹

2018

EasyChair Preprints

View full text Add to dashboard Cite

show abstract

“…Several approaches to approximate the hyperbolic tangent activation function have been reported in literature [18]- [21]. An efficient approach to approximate the hyperbolic tangent activation function was proposed in [2].…”

Section: A Hyperbolic Tangent Activation Functionmentioning

confidence: 99%

FPGA implementation of a multilayer Artificial Neural Network using System-on-Chip design methodology

Biradar

Chatterjee

Mishra

et al. 2015

2015 International Conference on Cognitive Computing and Information Processing(CCIP)

View full text Add to dashboard Cite

Artificial NeuralNetworks (ANNs) find applications in engineering solutions to complex problems. The usefulness of ANNs in real-time embedded applications can be enhanced if feasible architectures for their hardware implementation can be customized to provide an attractive tradeoff between area and performance. In this paper, we present a real-time embedded hardware implementation of a feed-forward neural network which employs backpropagation algorithm for training. Custom modules are designed for the activation functions, the neurons and a finite state machine that co-ordinate activities for training. The proposed ANN is implemented on a Virtex-5 Field Programmable Gate Array. Use of System-on-Chip design methodology enables design reuse while improving the performance metrics.

show abstract

“…An important part of any ANN is the non-linear activation function (AF) and its derivative, which are used for the feedforward and training operations of the ANN, respectively. With the advent of powerful FPGAs, as well as continued interest in both embedded systems and large-scale, high-performance VLSI implementations, there is a constant effort to design better AFs in hardware [2], [3], [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Performance analysis of table-based approximations of the hyperbolic tangent activation function

Leboeuf

Muscedere

Ahmadi

2011

2011 IEEE 54th International Midwest Symposium on Circuits and Systems (MWSCAS)

View full text Add to dashboard Cite

When designing an artificial neural network system in hardware, the implementation of the activation function is an important consideration. The hyperbolic tangent activation function is the most popular, and many approaches exist to ap proximate it, with varying trade-otis between area utilization and delay. Unfortunately, there is little data available reporting the minimum accuracy required of the activation function approxi mation in order to obtain good system-level performance; this is particularly the case for table-based approximation methods.In this paper, we demonstrate that table-based approximation methods are very well suited for implementing the tanh activation function, as well as its derivative in a variety of feed-forward artificial neural network topologies which employ the popular RPROP or Levenberg-Marquardt training algorithms. It is shown that when these training methods are used, an activation function possessing a relatively high maximum error can be used to obtain results comparable to 80ating point. This discovery suggests that these table-based methods can be employed with extreme efficiency in terms of area and speed, rendering them a promising option for any VLSI or FPGA artificial neural network hardware design.Artificial neural networks (ANNs) are a powerful com putational model originally inspired by the workings of the human brain. They are used extensively in many applications for function approximation and pattern classification [1]. An important part of any ANN is the non-linear activation function (AF) and its derivative, which are used for the feedforward and training operations of the ANN, respectively. With the advent of powerful FPGAs, as well as continued interest in both embedded systems and large-scale, high-performance VLSI implementations, there is a constant effort to design better AFs in hardware [2], [3], [4], [5].It is generally accepted that a maximum representation error of 1 % or less is needed in order to function properly [6], [7], [8], however these results seem to apply specifically to when the classic backpropagation algorithm is used for training. The majority of these works implement the AF and its derivative using piecewise linear approximations, or a custom function generator. Another approach is to store these functions in lookup tables (LUTs), however this is often shunned due to the perception that very large table sizes are required to achieve acceptable levels of performance [9].Recently, some table-based approaches have been proposed [10], [11] which are very competitive in terms speed and area utilization, as long as the maximum approximation error is no less than approximately 1%. It is desirable to determine how these AFs and their derivatives perform in a system-level test in order to characterize their performance and identify their strengths and weeknesses.In this work, a wide series of experiments are carried out in order to determine how well an ANN system performs when table-based AFs are employed. Performance is characterized over an ar...

show abstract

Implementation of a digital neuron with nonlinear activation function using piecewise linear approximation technique

Cited by 12 publications

References 13 publications

Fixed-point design methodology of the hyperbolic tangent function using lookup tables with performance evaluation on FPGA

Fixed-point design methodology of the hyperbolic tangent function using lookup tables with performance evaluation on FPGA

FPGA implementation of a multilayer Artificial Neural Network using System-on-Chip design methodology

Performance analysis of table-based approximations of the hyperbolic tangent activation function

Contact Info

Product

Resources

About