Shannon-Inspired Statistical Computing for the Nanoscale Era

Shanbhag, Naresh R.; Verma, Naveen; Kim, Yongjune; Patil, Ameya; Varshney, Lav R.

doi:10.1109/jproc.2018.2869867

Cited by 28 publications

(23 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In neurobiology, we show that neural connectivity, reliability, and energy characteristics are matched to one another, as per our theory [26]. In circuit design, [8] details several circuits designed for practical problems.…”

mentioning

confidence: 54%

“…After all, one could consider making a fixed number of individual gates less noisy with more energy, or one could construct larger and more redundant circuit designs with gates that remain noisy. Determining best design strategies is useful not just as a proof technique, but also for informing practical circuit design and explaining the nature of biological neural networks in sensory cortex, as we detail in separate works [8], [26]. In neurobiology, we show that neural connectivity, reliability, and energy characteristics are matched to one another, as per our theory [26].…”

mentioning

confidence: 72%

“…industry has been exploring promising new energy-efficient nanoscale devices as computational substrates [8]. Specialpurpose nanoscale hardware is faster and more energy-efficient than alternate approaches, making deep learning suitable for applications ranging from voice recognition on mobile devices to in-sensor health monitoring [9], [10].…”

mentioning

confidence: 99%

See 2 more Smart Citations

Energy-Reliability Limits in Nanoscale Feedforward Neural Networks and Formulas

Chatterjee

Varshney

2020

IEEE J. Sel. Areas Inf. Theory

Self Cite

View full text Add to dashboard Cite

Due to energy-efficiency requirements, computational systems are now being implemented using noisy nanoscale semiconductor devices whose reliability depends on energy consumed. We study circuit-level energy-reliability limits for deep feedforward neural networks (multilayer perceptrons) built using such devices, and en route also establish the same limits for formulas (boolean tree-structured circuits). To obtain energy lower bounds, we extend Pippenger's mutual information propagation technique for characterizing the complexity of noisy circuits, since small circuit complexity need not imply low energy. Many device technologies require all gates to have the same electrical operating point; in circuits of such uniform gates, we show that the minimum energy required to achieve any non-trivial reliability scales superlinearly with the number of inputs. Circuits implemented in emerging device technologies like spin electronics can, however, have gates operate at different electrical points; in circuits of such heterogeneous gates, we show energy scaling can be linear in the number of inputs. Building on our extended mutual information propagation technique and using crucial insights from convex optimization theory, we develop an algorithm to compute energy lower bounds for any given boolean tree under heterogeneous gates. This algorithm runs in linear time in number of gates, and is therefore practical for modern circuit design. As part of our development we find a simple procedure for energy allocation across circuit gates with different operating points and neural networks with differently-operating layers.

show abstract

mentioning

confidence: 54%

mentioning

confidence: 72%

See 1 more Smart Citation

Energy-Reliability Limits in Nanoscale Feedforward Neural Networks and Formulas

Chatterjee

Varshney

2020

IEEE J. Sel. Areas Inf. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…Neuromorphic computing, a novel concept proposed in 1990s, suggests that future computer system can imitate the operation principles of human brains by processing analog signals in parallel directly on the physical level . It is a promising way to move beyond the deterministic von Neumann model of computation due to its three significant advantages—high parallelism, low power consumption, and in‐memory computing—and can be included in a Shannon‐inspired statistical computation model . Guided by information theory, it is possible to explore its design principles of circuits and architectures that approach the limits of energy efficiency, latency, and accuracy.…”

Section: Introductionmentioning

confidence: 99%

Memristor‐Based Biologically Plausible Memory Based on Discrete and Continuous Attractor Networks for Neuromorphic Systems

Wang

Liu

et al. 2020

Advanced Intelligent Systems

View full text Add to dashboard Cite

To approach an advanced neuromorphic system, a significant unsettled problem is how to realize biologically plausible memory structures that are dramatically different from classical computers. Herein, a physical system based on memristors is simulated to realize associative memory based on discrete attractor networks, which is essentially content‐based storage, and the influence of device characteristics on network performance is systematically studied. An in situ unsupervised learning method is applied to make greater use of array structure and competitions between neurons, demonstrating significant performance improvement in memory capacity and noise tolerance compared with existing supervised approaches. By extending to continuous attractor neural networks (CANNs), working memory is realized based on memristors for the first time via simulation, and the write and read noises in memristor arrays are found to have different impacts on the ability of CANN in maintaining dynamic information. This work lays a foundation for the construction of future advanced neuromorphic computing systems.

show abstract

“…Recently, artificial intelligence (AI) systems have begun to approach and exceed human performance in many intelligent tasks: AlexNet [1] and ResNet [2] achieving human-level accuracy in recognition tasks, and AlphaGo [3] beating human champions in Go [4]. These acclaimed successes of AI are mainly based on computations using massive amounts of data.…”

Section: Introductionmentioning

confidence: 99%