Universal Approximation Property and Equivalence of Stochastic Computing-Based Neural Networks and Binary Neural Networks

Abstract

Large-scale deep neural networks are both memory and computation-intensive, thereby posing stringent requirements on the computing platforms. Hardware accelerations of deep neural networks have been extensively investigated. Specific forms of binary neural networks (BNNs) and stochastic computing-based neural networks (SCNNs) are particularly appealing to hardware implementations since they can be implemented almost entirely with binary operations.Despite the obvious advantages in hardware implementation, these approximate computing techniques are questioned by researchers in terms of accuracy and universal applicability. Also it is important to understand the relative pros and cons of SCNNs and BNNs in theory and in actual hardware implementations. In order to address these concerns, in this paper we prove that the "ideal" SCNNs and BNNs satisfy the universal approximation property with probability 1 (due to the stochastic behavior). The proof is conducted by first proving the property for SCNNs from the strong law of large numbers, and then using SCNNs as a "bridge" to prove for BNNs. Based on the universal approximation property, we further prove that SCNNs and BNNs exhibit the same energy complexity. In other words, they have the same asymptotic energy consumption with the growing of network size. We also provide a detailed analysis of the pros and cons of SCNNs and BNNs for hardware implementations and conclude that SCNNs are more suitable.

Universal Approximation Theorems of Fully Connected Binarized Neural Networks

“…Differently to that, the authors in [22] focused on stochastic-computing based NNs and show UA properties in almost sure sense. The work of [7] considers UA properties of quantized ReLU networks for locally integratable functions on the Sobolev space.…”

“…Modifications and extensions have been established in [11] or [1], for example. UA properties to quantized NNs have been considered in [20], [7] and [22], for example. The first work focused on the uniform approximation capabilities of QNNs for Lipschitz-continuous functions.…”

A stochastic-computing based deep learning framework using adiabatic quantum-flux-parametron superconducting technology

“…Compared with precise computing applications [3], SC is more suitable for approximate computing where DNN is an example. Moreover, it has been proved recently on the equivalence between SC-based DNN and binary neural networks (BNN) [50]. As the latter originates from the deep learning society [11] and many accuracy enhancement techniques have been developed, these advances can be migrated to enhance the accuracy in SC-based DNNs to be close to the software-based, floating point accuracy levels.…”

“…This is because the final classification result depends on the relative score/logit values of different classes, instead of absolute values. Recent work [35] [3] have pointed out the suitability of SC for DNN acceleration, and [50] has further proved the equivalence between SC-based DNN and binary neural networks, where the latter originate from deep learning society [11]. All the above discussions suggest the potential to build SC-based DNN acceleration using AQFP technology.…”

Demystifying and Generalizing BinaryConnect

“…Impressive performance has been achieved with quantized networks, for example, on object detection [44] and natural language processing [43] tasks. The theoretical underpinnings of quantized neural networks, such as when and why their performance remains reasonably well, have been actively studied [3,13,22,41,46].…”