How Amdahl’s Law limits the performance of large artificial neural networks

Végh, János

doi:10.1186/s40708-019-0097-2

Cited by 25 publications

(30 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This step involves (mostly) losing phase information. Furthermore, as detailed in [11], it introduces severe payload performance limits for neuronal operations.…”

Section: Data Delivery Methodsmentioning

confidence: 99%

See 1 more Smart Citation

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

Végh¹,

Berki²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Both the growing demand to cope with "big data" (based on, or assisted by, artificial intelligence) and the interest in understanding the operation of our brain more completely, stimulated the efforts to build biology-mimicking computing systems from inexpensive conventional components and build different ("neuromorphic") computing systems. On one side, those systems require an unusually large number of processors, which introduces performance limitations and nonlinear scaling. On the other side, the neuronal operation drastically differs from the conventional workloads. The conduction time (transfer time) is ignored in both in conventional computing and "spatiotemporal" computational models of neural networks, although von Neumann warned: "In the human nervous system the conduction times along the lines (axons) can be longer than the synaptic delays, hence our procedure of neglecting them aside of the processing time would be unsound" [1], section 6.3. This difference alone makes imitating biological behavior in technical implementation hard. Besides, the recent issues in computing called the attention to that the temporal behavior is a general feature of computing systems, too. Some of their effects in both biological and technical systems were already noticed. Instead of introducing some "looks like" models, the correct handling of the transfer time is suggested here. Introducing the temporal logic, based on the Minkowski transform, gives quantitative insight into the operation of both kinds of computing systems, furthermore provides a natural explanation of decades-old empirical phenomena. Without considering their temporal behavior correctly, neither effective implementation nor a true imitation of biological neural systems are possible.

show abstract

“…This step involves (mostly) losing phase information. Furthermore, as detailed in [11], it introduces severe payload performance limits for neuronal operations.…”

Section: Data Delivery Methodsmentioning

confidence: 99%

“…It was not suspected that the physical implementations of the electronic components have a temporal behavior [9]. However, it was the final reason of many experienced issues, from the payload per-formance limit of supercomputers [10] and brain simulation [11], to the weeks-long training time in deep learning [12,13].…”

Section: Introductionmentioning

confidence: 99%

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

Végh¹,

Berki²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Notice, that neurons at different Minkowski-distance "see" the local frequency at different Minkowski-times, so they set their internal phase angle to the Minkowski-distance. That is, "at the same time" means different absolute times 11 ent neurons, depending on their location: the biology uses Minkowski four-coordinates (Minkowski-distance).…”

Section: Manifestations Of the Role Of Time In Biologymentioning

confidence: 99%

“…Thanks to the decreasing density and the increasing frequency, the speed of changing the electronic states in a computing system, more and more approached the limiting speed. It was not suspected that the physical implementations of the electronic components have a temporal behavior [9], although it was the final reason of many experienced issues, from the payload performance limit of supercomputers [10] and brain simulation [11], to the weeks-long training time in deep learning [12,13].…”

Section: Introductionmentioning

confidence: 99%

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

Végh

Berki

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Both the growing demand to cope with ”big data” (based on, or assisted by, artiﬁcial intelligence) and the interest in understanding the operation of our brain more completely, stimulated the eﬀorts to build biology-mimicking computing systems from inexpensive conventional components and build diﬀerent (”neuro-morphic”) computing systems. On one side, those systems require an unusually large number of processors, which introduces performance limitations and nonlinear scaling. On the other side, the neuronal operation drastically diﬀers from the conventional workloads. The conduction time (transfer time) is ignored in both in conventional computing and ”spatiotemporal” computational models of neural networks, although von Neumann warned: ”In the human nervous system the con-duction times along the lines (axons) can be longer than the synaptic delays, hence our above procedure of neglecting them aside of τ [the processing time] would be unsound” [1], section 6.3. This diﬀerence alone makes imitating biological behavior in technical implementation hard. Besides, the recent issues in computing called the attention to that the temporal behavior is a general feature of computing systems, too. Some of their eﬀects in both biological and technical systems were al-ready noticed. Instead of introducing some ”looks like” models, the correct handling of the transfer time is suggested here. Introducing the temporal logic, based on the Minkowski transform, gives quantitative insight into the operation of both kinds of computing systems, furthermore provides a natural explanation of decades-old empirical phenomena. Without considering their temporal behavior correctly, neither eﬀective implementation nor a true imitation of biological neural systems are possible.

show abstract

“…As discussed in details in [52], two of the mentioned effects can become dominating in those applications. Since the operating time scale of the biological networks lies in the msec range, brain simulation applications commonly use integration time about 1 ms [53].…”

Section: What Factors Dominate the Performance Gainmentioning

confidence: 99%

Can parallelization save the (computing) world?

Végh¹,

Vásárhelyi²,

Drótos³

2019

Adv. sci. technol. eng. syst. j.

Self Cite

View full text Add to dashboard Cite

Single-processor approachAs all other laws of the growth in computing, the growth of computing performance also shows a "logistic curve"-like behavior, rather than an unlimited exponential growth. The stalling of the single-processor performance experienced nearly two decades ago forced computer experts to look for alternative methods, mainly for some kind of parallelization. Solving the task needs different parallelization methods, and the wide range of those distributed systems limits the computing performance in very different ways. Some general limitations are shortly discussed, and a (by intention strongly simplified) general model of performance of parallelized systems is introduced. The model enables to highlight bottlenecks of parallelized systems of different kind and with the published performance data enables to predict performance limits of strongly parallelized systems like large scale supercomputers and neural networks. Some alternative solution possibilities of increasing computing performance are also discussed.

show abstract

How Amdahl’s Law limits the performance of large artificial neural networks

Cited by 25 publications

References 29 publications

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

On the Spatiotemporal Behavior in Biology-Mimicking Computing Systems

Can parallelization save the (computing) world?

Contact Info

Product

Resources

About