Inexactness and a future of computing

One of the central concerns of computer science is how the resources needed to perform a given computation depend on that computation. Moreover, one of the major resource requirements of computers -ranging from biological cells to human brains to highperformance (engineered) computers -is the energy used to run them, i.e., the thermodynamic costs of running them. Those thermodynamic costs of performing a computation have been a long-standing focus of research in physics, going back (at least) to the early work of Landauer, in which he argued that the thermodynamic cost of erasing a bit in any physical system is at least kT ln [2].One of the most prominent aspects of computers is that they are inherently nonequilibrium systems. However, the research by Landauer and co-workers was done when non-equilibrium statistical physics was still in its infancy, requiring them to rely on equilibrium statistical physics. This limited the breadth of issues this early research could address, leading them to focus on the number of times a bit is erased during a computation -an issue having little bearing on the central concerns of computer science theorists.Since then there have been major breakthroughs in nonequilibrium statistical physics, leading in particular to the new subfield of "stochastic thermodynamics". These breakthroughs have allowed us to put the thermodynamic analysis of bit erasure on a fully formal (nonequilibrium) footing. They are also allowing us to investigate the myriad aspects of the relationship between statistical physics and computation, extending well beyond the issue of how much work is required to erase a bit.In this paper I review some of this recent work on the "stochastic thermodynamics of computation". After reviewing the salient parts of information theory, computer science theory, and stochastic thermodynamics, I summarize what has been learned about the entropic costs of performing a broad range of computations, extending from bit erasure to loop-free circuits to logically reversible circuits to information ratchets to Turing machines. These results reveal new, challenging engineering problems for how to design computers to have minimal thermodynamic costs. They also allow us to start to combine computer science theory and stochastic thermodynamics at a foundational level, thereby expanding both.dergoing arbitrary external driving. In particular, we can now decompose the time-derivative of the (Shannon) entropy of such a system into an "entropy production rate", quantifying the rate of change of the total entropy of the system and its environment, minus a "entropy flow rate", quantifying the rate of entropy exiting the system into its environment. Crucially, the entropy production rate is non-negative, regardless of the CTMC. So if it ever adds a nonzero amount to system entropy, its subsequent evolution cannot undo that increase in entropy. (For this reason it is sometimes referred to as irreversible entropy production.) This is the modern understanding of the second law of thermodynamics, for...

show abstract

“…Those convergences of trajectories are many-to-one maps, and therefore incur entropic costs, which are reflected in Eq. (66).…”

Section: Kolmogorov Complexity and The Entropy Dynamics Of Turinmentioning

confidence: 99%

“…(66) is uncomputable, and the two functions in Eq. (66) are nonrecursive [84]. So the sum of those three terms cannot be computed.…”

Section: Kolmogorov Complexity and The Entropy Dynamics Of Turinmentioning

confidence: 99%

The stochastic thermodynamics of computation

Wolpert¹

2019

J. Phys. A: Math. Theor.

139

View full text Add to dashboard Cite

show abstract

“…Thus, concepts from inexact computing can be used to choose D such that precision is traded for large savings in computational resources, which can be then reinvested into more simulations, higher resolutions for critical processes etc. [14,15,17,21,18].…”

Section: Short-term Prediction: Scaling Of Rc-esn Performance With Rementioning

confidence: 99%

Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM

Chattopadhyay¹,

Hassanzadeh²,

Subramanian³

2019

Preprint

View full text Add to dashboard Cite

In this paper, the performance of three deep learning methods for predicting short-term evolution and for reproducing the long-term statistics of a multi-scale spatio-temporal Lorenz 96 system is examined. The methods are: echo state network (a type of reservoir computing, RC-ESN), deep feed-forward artificial neural network (ANN), and recurrent neural network with long short-term memory (RNN-LSTM). This Lorenz 96 system has three tiers of nonlinearly interacting variables representing slow/largescale (X), intermediate (Y ), and fast/small-scale (Z) processes. For training or testing, only X is available; Y and Z are never known or used. We show that RC-ESN substantially outperforms ANN and RNN-LSTM for short-term prediction, e.g., accurately forecasting the chaotic trajectories for hundreds of numerical solver's time steps, equivalent to several Lyapunov timescales. The RNN-LSTM and ANN show some prediction skills as well; RNN-LSTM bests ANN. Furthermore, even after losing the trajectory, data predicted by RC-ESN and RNN-LSTM have probability density functions (PDFs) that closely match the true PDF, even at the tails. The PDF of the data predicted using ANN, however, deviates from the true PDF. Implications, caveats, and applications to data-driven and data-assisted surrogate modeling of complex nonlinear dynamical systems such as weather/climate are discussed.

show abstract

“…By turning down the voltage, processors can be switched from deterministic to probabilistic calculators. For example, based on contemporary chip technology, a fourfold reduction in power can result in less than a 1% chance that a computational step will be incorrect 4 .…”

Section: More With Lessmentioning

confidence: 99%

Modelling: Build imprecise supercomputers

Palmer

2015

Nature

View full text Add to dashboard Cite

Energy-optimized hybrid computers with a range of processor accuracies will advance modelling in fields from climate change to neuroscience, says Tim Palmer.T oday's supercomputers lack the power to model accurately many aspects of the real world, from the impact of cloud systems on Earth's climate to the processing ability of the human brain. Rather than wait decades for sufficiently powerful supercomputers -with their potentially unsustainable energy demands -it is time for researchers to reconsider the basic concept of the computer. We must move beyond the idea of a computer as a fast but otherwise traditional 'Turing machine' , churning through calculations bit by bit in a sequential, precise and reproducible manner.In particular, we should question whether all scientific computations need to be performed deterministically -that is, always producing the same output given the same input -and with the same high level of precision. I argue that for many applications they do not.Energy-efficient hybrid supercomputers with a range of processor accuracies need to be developed. These would combine conventional energy-intensive processors with low-energy, non-deterministic processors, able to analyse data at variable levels of precision. The demand for such machines could be substantial, across diverse sectors of the scientific community. MORE WITH LESSTake climate change, for example. Estimates of Earth's future climate are based on solving nonlinear (partial differential) equations for fluid flow in the atmosphere and oceans. Current climate simulators -typically with grid cells of 100 kilometres in width -can resolve the large, low-pressure weather systems typical of mid-latitudes, but not individual clouds. Yet modelling cloud systems accurately is crucial for reliable estimates of the impact of anthropogenic emissions on global temperature 1 .The resolution of this computational grid is determined by the available computing power. Current petaflop computers can perform up to 10 15 additions or multiplications -floating-point operationsper second (flops). By the early 2020s, next-generation exaflop supercomputers, capable of 10 18 operations per second, will be able to resolve the largest and most vigorous types of thunderstorm 2 . But cloud physics on scales smaller than a grid cell will still have to be approximated, or

show abstract

Inexactness and a future of computing

Cited by 31 publications

References 57 publications

The stochastic thermodynamics of computation

The stochastic thermodynamics of computation

Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM

Modelling: Build imprecise supercomputers

Contact Info

Product

Resources

About