Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks

Ji, Li; Yuan, Zihao; Li, Zhe; Ding, Caiwen; Ren, Ao; Qiu, Qinru; Draper, J.; Wang, Yanzhi

doi:10.1109/ijcnn.2017.7965993

Cited by 43 publications

(31 citation statements)

References 27 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The survey of logistic, ReLU, and tanh networks is reported in Ref. [65]. Two bestperforming examples were extracted for SC-DCNN [61], and HEIF is reported in Ref.…”

Section: Evaluation and Resultsmentioning

confidence: 99%

“…The first stochastic computing implementation of a deep convolutional neural network, a particularly important neural network topology with broad applications to image processing, was proposed by Ren et alia [61] and further optimized in Refs. [62,65]. These works draw heavily on new ideas in the stochastic computing literature, including massively parallel generation of pseudorandom bitstreams [66], state-machine based nonlinear activation functions [63,67], and aggressive use of correlation insensitivity [68].…”

Section: Application To Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Energy-Efficient Stochastic Computing with Superparamagnetic Tunnel Junctions

et al. 2020

View full text Add to dashboard Cite

Superparamagnetic tunnel junctions have emerged as a competitive, realistic nanotechnology to support novel forms of stochastic computation in CMOS-compatible platforms. One of their applications is to generate random bitstreams suitable for use in stochastic computing implementations. We describe a method for digitally programmable bitstream generation based on pre-charge sense amplifiers which is more energy efficient than previously explored alternatives. The energy savings offered by this digital generator survive when we use them as the fundamental units of a neural network architecture. To take advantage of the potential savings, we codesign the algorithm with the circuit, rather than directly transcribing a classical neural network into hardware. The flexibility of the neural network mathematics compensates for explicitly energy efficient choices we make at the device level. The result is a convolutional neural network design operating at ≈ 150 nJ per inference with 97 % performance on MNIST-nearly an order of magnitude more energy efficiency than comparable proposals in the recent literature. * matthew.daniels@nist.gov † mark.stiles@nist.govThe low energy, truly random behavior, ease of control, and established compatibility with complementarymetal-oxide-semiconductor (CMOS) circuitry has led to the use of SMTJs as the basis for a number of novel computing schemes [11][12][13]. SMTJs were proposed to implement the concept of probabilistic bits, or "p-bits," which were leveraged for applications as Bayesian neural networks [14][15][16], invertible Boolean logic [17,18], reservoir computing [19], and Ising network models applied to optimization problems [20,21]. SMTJs were also proposed as stochastic neural units [22] that can interact with synaptic units, emulated by crossbar arrays of mag-

show abstract

“…The survey of logistic, ReLU, and tanh networks is reported in Ref. [65]. Two bestperforming examples were extracted for SC-DCNN [61], and HEIF is reported in Ref.…”

Section: Evaluation and Resultsmentioning

confidence: 99%

Section: Application To Neural Networkmentioning

confidence: 99%

Energy-Efficient Stochastic Computing with Superparamagnetic Tunnel Junctions

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Besides multiplications and additions, SC-based activation functions are also developed [19], [20]. As a result, SC has become an interesting and promising approach to implement large-scale neural networks [11], [12], [21], [22] with high performance/energy efficiency and minor accuracy degradation.…”

Section: Stochastic Computing and Scnnsmentioning

confidence: 99%

Universal Approximation Property and Equivalence of Stochastic Computing-Based Neural Networks and Binary Neural Networks

Wang

Zhan

Zhao

et al. 2019

AAAI

Self Cite

View full text Add to dashboard Cite

Large-scale deep neural networks are both memory and computation-intensive, thereby posing stringent requirements on the computing platforms. Hardware accelerations of deep neural networks have been extensively investigated. Specific forms of binary neural networks (BNNs) and stochastic computing-based neural networks (SCNNs) are particularly appealing to hardware implementations since they can be implemented almost entirely with binary operations.Despite the obvious advantages in hardware implementation, these approximate computing techniques are questioned by researchers in terms of accuracy and universal applicability. Also it is important to understand the relative pros and cons of SCNNs and BNNs in theory and in actual hardware implementations. In order to address these concerns, in this paper we prove that the "ideal" SCNNs and BNNs satisfy the universal approximation property with probability 1 (due to the stochastic behavior). The proof is conducted by first proving the property for SCNNs from the strong law of large numbers, and then using SCNNs as a "bridge" to prove for BNNs. Based on the universal approximation property, we further prove that SCNNs and BNNs exhibit the same energy complexity. In other words, they have the same asymptotic energy consumption with the growing of network size. We also provide a detailed analysis of the pros and cons of SCNNs and BNNs for hardware implementations and conclude that SCNNs are more suitable.

show abstract

“…Having hundreds of these modules in a design would require multiple megabits of storage. Indeed, in [11], the authors compare 8-bit neurons ReLU and tanh/sigmoid activation functions. They show that replacing the ReLU with tanh increases the neuron area by 20% and neuron energy by 36%.…”

Section: Introductionmentioning

confidence: 99%

Design Space Exploration of Neural Network Activation Function Circuits

Yang

Wei

et al. 2019

IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst.

View full text Add to dashboard Cite

The widespread application of artificial neural networks has prompted researchers to experiment with FPGA and customized ASIC designs to speed up their computation. These implementation efforts have generally focused on weight multiplication and signal summation operations, and less on activation functions used in these applications. Yet, efficient hardware implementations of nonlinear activation functions like Exponential Linear Units (ELU), Scaled Exponential Linear Units (SELU), and Hyperbolic Tangent (tanh), are central to designing effective neural network accelerators, since these functions require lots of resources. In this paper, we explore efficient hardware implementations of activation functions using purely combinational circuits, with a focus on two widely used nonlinear activation functions, i.e., SELU and tanh. Our experiments demonstrate that neural networks are generally insensitive to the precision of the activation function. The results also prove that the proposed combinational circuit based approach is very efficient in terms of speed and area, with negligible accuracy loss on the MNIST, CIFAR-10 and IMAGENET benchmarks. Synopsys Design Compiler synthesis results show that circuit designs for tanh and SELU can save between 3.13 7.69 and 4.45 8.45 area compared to the LUT/memory based implementations, and can operate at 5.14GHz and 4.52GHz using the 28nm SVT library, respectively. The implementation is available at: https://github.com/ThomasMrY/ActivationFunctionDemo.

show abstract

Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks

Cited by 43 publications

References 27 publications

Energy-Efficient Stochastic Computing with Superparamagnetic Tunnel Junctions

Energy-Efficient Stochastic Computing with Superparamagnetic Tunnel Junctions

Universal Approximation Property and Equivalence of Stochastic Computing-Based Neural Networks and Binary Neural Networks

Design Space Exploration of Neural Network Activation Function Circuits

Contact Info

Product

Resources

About