Energy-Reliability Limits in Nanoscale Feedforward Neural Networks and Formulas

Abstract

Due to energy-efficiency requirements, computational systems are now being implemented using noisy nanoscale semiconductor devices whose reliability depends on energy consumed. We study circuit-level energy-reliability limits for deep feedforward neural networks (multilayer perceptrons) built using such devices, and en route also establish the same limits for formulas (boolean tree-structured circuits). To obtain energy lower bounds, we extend Pippenger's mutual information propagation technique for characterizing the complexity of noisy circuits, since small circuit complexity need not imply low energy. Many device technologies require all gates to have the same electrical operating point; in circuits of such uniform gates, we show that the minimum energy required to achieve any non-trivial reliability scales superlinearly with the number of inputs. Circuits implemented in emerging device technologies like spin electronics can, however, have gates operate at different electrical points; in circuits of such heterogeneous gates, we show energy scaling can be linear in the number of inputs. Building on our extended mutual information propagation technique and using crucial insights from convex optimization theory, we develop an algorithm to compute energy lower bounds for any given boolean tree under heterogeneous gates. This algorithm runs in linear time in number of gates, and is therefore practical for modern circuit design. As part of our development we find a simple procedure for energy allocation across circuit gates with different operating points and neural networks with differently-operating layers.

Power-Efficient Deep Neural Networks with Noisy Memristor Implementation

“…In this work, we investigate this key issue in the context of in-memory computing, by first aiming to predict the effect of noisy memristor values onto the final DNN computation. The physical substrate of memristors and their computational and noise properties make the mathematical problem quite different from the above works [6]- [9].…”

“…It is shown in [6] that noise can sometimes help in the inference performance of neural networks by getting them unstuck from local minima. Several works in the literature also investigate the robustness of DNNs to undesired noise [7]- [9], though without considering the in-memory computation framework. As a key issue, the DNN robustness is strongly related to the power consumption of its electrical circuit implementation [9].…”

“…Several works in the literature also investigate the robustness of DNNs to undesired noise [7]- [9], though without considering the in-memory computation framework. As a key issue, the DNN robustness is strongly related to the power consumption of its electrical circuit implementation [9]. In this work, we investigate this key issue in the context of in-memory computing, by first aiming to predict the effect of noisy memristor values onto the final DNN computation.…”