Neuromorphic System for Spatial and Temporal Information Processing

Zyarah, Abdullah M.; Gómez, Kevin; Kudithipudi, Dhireesha

doi:10.1109/tc.2020.3000183

Cited by 8 publications

(6 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We suggest that future widespread deployment of AI/ML will require the development of compute-efficient L2 architectures. Rapid progress towards this goal is being made through the creation of new hardware substrates, notably neuromorphic accelerators that emulate neural processing 209,[217][218][219][220][221][222][223][224][225][226][227][228] . In particular, bio-plausible L2 models can be well-suited for these neuromorphic accelerators.…”

Section: Discussionmentioning

confidence: 99%

Biological underpinnings for lifelong learning machines

Aguilar-Simon²,

et al. 2022

View full text Add to dashboard Cite

Biological organisms learn from interactions with their environment throughout their lifetime. For artificial systems to successfully act and adapt in the real world, it is desirable to similarly be able to learn on a continual basis. This challenge is known as lifelong learning, and remains to a large extent unsolved. In this perspective article, we identify a set of key capabilities that artificial systems will need to achieve lifelong learning. We describe a number of biological mechanisms, both neuronal and non-neuronal, that help explain how organisms solve these challenges, and present examples of biologically inspired models and biologically plausible mechanisms that have been applied to artificial intelligence systems in the quest towards development of lifelong learning machines. We discuss opportunities to further our understanding and advance the state of the art in lifelong learning, aiming to bridge the gap between natural and artificial intelligence.

show abstract

Section: Discussionmentioning

confidence: 99%

Biological underpinnings for lifelong learning machines

Aguilar-Simon²,

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Another approach is to implement a scatter-compute-gather module to aggregate operands based upon the indices of their non-zero values [77]. In [94] the authors implemented a completely custom memristor-based mixed signal architecture. They demonstrate large performance gains and energy efficiencies for embedded applications using a biologically inspired sparse-sparse learning algorithm.…”

Section: Accelerating Sparse Network On Other Platformsmentioning

confidence: 99%

“…Most of the training work has focused on weight sparsity with a few papers focused on activation sparsity. There is relative lack of research on networks that have both forms of sparsity (exceptions are [1,94]). In some scenarios networks trained without explicit activation sparsity end up with highly sparse activations anyway [8,19,20,33].…”

Section: Deploying Complex Sparse-sparse Systemsmentioning

confidence: 99%

Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks

Hunter¹,

Spracklen²,

Ahmad³

2022

Neuromorph. Comput. Eng.

View full text Add to dashboard Cite

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today’s neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100X improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.

show abstract

“…In [86] the authors implemented a completely custom memristor-based mixed signal architecture. They demonstrate large performance gains and energy efficiencies for embedded applications using a biologically inspired sparse-sparse learning algorithm.…”

Section: Accelerating Sparse Network On Other Platformsmentioning

confidence: 99%

“…Most of that work has focused on weight sparsity with a few papers focused on activation sparsity. There is relative lack of research on networks that have both forms of sparsity (exceptions are [1,86]). In some scenarios networks trained without explicit activation sparsity end up with highly sparse activations anyway [19,33,7].…”

Section: Deploying Complex Sparse-sparse Systemsmentioning

confidence: 99%

Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

Hunter¹,

Spracklen²,

Ahmad³

2021

Preprint

View full text Add to dashboard Cite

In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today's neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100X improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.

show abstract

Neuromorphic System for Spatial and Temporal Information Processing

Cited by 8 publications

References 35 publications

Biological underpinnings for lifelong learning machines

Biological underpinnings for lifelong learning machines

Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks

Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks

Contact Info

Product

Resources

About