Stochastic Thermodynamics of Learning

Goldt, Sebastian; Seifert, Udo

doi:10.1103/physrevlett.118.010601

Cited by 51 publications

(56 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worthwhile to revisit the results of our earlier work [21] in the light of these results. In this previous paper, we studied a different learning problem, namely the learning of P mappings } carry no information about the label of a previously unseen input.…”

Section: Concluding Perspectivesmentioning

confidence: 90%

“…Our inequality(17) still applies to this process, but it is not very sharp anymore: I : 1 T s s( ) and S w 1 n D( ) , but a steady state comes with a non-zero rate of heat dissipation, such that Q t D~. This issue was not addressed in our previous work [21]. In this section, we derive a sharper bound using concepts from steady state thermodynamics [42].…”

Section: Learning In Large Network and A Second Boundmentioning

confidence: 93%

“…If it is possible to construct a teacher T , the rule implicitly defined by the mappings is realisable and can, at least in theory, be learned. Even in that case, however, the issue remains for the scenario considered in [21] that the number of samples from which the neuron learns is limited and might not be sufficient to learn the underlying 'rule' effectively. On the other hand, learning the mappings…”

Section: Concluding Perspectivesmentioning

confidence: 99%

“…Neural networks, well known from statistical physics and machine learning [18][19][20], form a mature framework to investigate learning and generalising. We have recently introduced the methods of stochastic thermodynamics to study the thermodynamic efficiency of the second step, building an efficient representation of uncorrelated data [21], and a recent study has looked at the non-equilibrium thermodynamics of unsupervised learning with restricted Boltzmann machines [22].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Thermodynamic efficiency of learning a rule in neural networks

Goldt

Seifert

2017

New J. Phys.

Self Cite

View full text Add to dashboard Cite

Biological systems have to build models from their sensory input data that allow them to efficiently process previously unseen inputs. Here, we study a neural network learning a binary classification rule for these inputs from examples provided by a teacher. We analyse the ability of the network to apply the rule to new inputs, that is to generalise from past experience. Using stochastic thermodynamics, we show that the thermodynamic costs of the learning process provide an upper bound on the amount of information that the network is able to learn from its teacher for both batch and online learning. This allows us to introduce a thermodynamic efficiency of learning. We analytically compute the dynamics and the efficiency of a noisy neural network performing online learning in the thermodynamic limit. In particular, we analyse three popular learning algorithms, namely Hebbian, Perceptron and AdaTron learning. Our work extends the methods of stochastic thermodynamics to a new type of learning problem and might form a suitable basis for investigating the thermodynamics of decision-making. T s =  . This rule or function is implemented by a neural network, called the teacher. Another network, called the student, has to infer this rule from a number of examples , T x s ( ) supplied by the teacher. Our focus is on the final step of information processing: how well can the network emulate the function after a training period, i.e. how well do the outputs of the student, s, match the correct output of the teacher T s for the same input? We will show that the ability of the network to generalise such a rule from the examples it has seen to previously unseen inputs is OPEN ACCESS RECEIVED

show abstract

Section: Concluding Perspectivesmentioning

confidence: 90%

Section: Learning In Large Network and A Second Boundmentioning

confidence: 93%

Section: Concluding Perspectivesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Thermodynamic efficiency of learning a rule in neural networks

Goldt

Seifert

2017

New J. Phys.

Self Cite

View full text Add to dashboard Cite

show abstract

“…For θ → 0, Eq. [32][33][34]. Similarly, we can evaluate the efficiency in terms of sensitivity and precision with Eq.…”

Section: A Derivation Of Uncertainty Relationmentioning

confidence: 99%

Uncertainty relations in stochastic processes: An information inequality approach

Hasegawa

2019

Phys. Rev. E

117

View full text Add to dashboard Cite

The thermodynamic uncertainty relation is an inequality stating that it is impossible to attain higher precision than the bound defined by entropy production. In statistical inference theory, information inequalities assert that it is infeasible for any estimator to achieve an error smaller than the prescribed bound. Inspired by the similarity between the thermodynamic uncertainty relation and the information inequalities, we apply the latter to systems described by Langevin equations and derive the bound for the fluctuation of thermodynamic quantities. When applying the Cramér-Rao inequality, the obtained inequality reduces to the fluctuation-response inequality. We find that the thermodynamic uncertainty relation is a particular case of the Cramér-Rao inequality, in which the Fisher information is the total entropy production. Using the equality condition of the Cramér-Rao inequality, we find that the stochastic total entropy production is the only quantity which can attain equality in the thermodynamic uncertainty relation. Furthermore, we apply the Chapman-Robbins inequality and obtain a relation for the lower bound of the ratio between the variance and the sensitivity of systems in response to arbitrary perturbations.

show abstract

Teaching Complexity as Transdisciplinarity

Demerath

Suarez

2019

Understanding Complex Systems

View full text Add to dashboard Cite

Stochastic Thermodynamics of Learning

Cited by 51 publications

References 40 publications

Thermodynamic efficiency of learning a rule in neural networks

Thermodynamic efficiency of learning a rule in neural networks

Uncertainty relations in stochastic processes: An information inequality approach

Teaching Complexity as Transdisciplinarity

Contact Info

Product

Resources

About