Time’s Barbed Arrow: Irreversibility, Crypticity, and Stored Information

Crutchfield, James P.; Ellison, Christopher J.; Mahoney, John R.

doi:10.1103/physrevlett.103.094101

Cited by 99 publications

(198 citation statements)

References 17 publications

Supporting

Mentioning

195

Contrasting

Order By: Relevance

“…Given a generic pattern P( X, X), its statistical complexity C µ = H(S) = I( X ; S) is generally strictly greater than the amount of information its past contains about it future, I( X ; X). That is, for most patterns, even the most-efficient prescient memory -the causal states -contain superfluous information about the past [20,35]. For these patterns, I(R ; X) > I( X ; X) for all choices of prescient memory R. This yields us our third result:…”

mentioning

confidence: 84%

Thermodynamics of complexity and pattern manipulation

et al. 2017

View full text Add to dashboard Cite

Many organisms capitalize on their ability to predict the environment to maximize available free energy, and reinvest this energy to create new complex structures. This functionality relies on the manipulation of patterns-temporally ordered sequences of data. Here, we propose a framework to describe pattern manipulators -devices that convert thermodynamic work to patterns or vice versa -and use them to build a 'pattern engine that facilitates a thermodynamic cycle of pattern creation and consumption. We show that the least heat dissipation is achieved by the provably simplest devices; the ones that exhibit desired operational behaviour while maintaining the least internal memory. We derive the ultimate limits of this heat dissipation, and show that it is generally non-zero and connected with the patterns intrinsic crypticity -a complexity theoretic quantity that captures the puzzling difference between the amount of information the pattern's past behaviour reveals about its future, and the amount one needs to communicate about this past to optimally predict the future.The manipulation of patterns is as important to living organisms as it is for computation. Living things capitalize on structure in their environment for available energy, and use this energy to generate new complex structures. Similarly, a crucial task in the modern era of big data is to identify patterns in large data sets in order to make predictions about future events -often at great energetic cost. Here, we consider the thermodynamic costs intrinsic to this sort of pattern manipulation and ask: is there a preferred method by which this manipulation should be done? Our intuition is that simpler is better, a longstanding tenant of natural philosophy known as Occam's razor. To formalize this, we first qualify what is meant both by simpler and by better.In complexity science, computational mechanics formalizes what is simpler in the context of pattern manipulation [1][2][3]. The premise is that everything we observe in the environment can be considered to be a patterna temporal sequence of data exhibiting certain statistical structure. Much of science then deals with building models that can explain such statistics -machines that take information from past observations, and use it to generate statistically coinciding conditional future predictions. Given two machines that exhibit same pattern of behaviour, the one that stores less information from the past is considered simpler, the motivation being that it better isolates indicators of future behaviour. The simplest such machine then defines exactly how much memory is required to produce a given pattern, and thus * ajpgarner@nus.edu.sg † cqtmileg@nus.edu.sg quantifies the pattern's intrinsic structure. Known as statistical complexity, this measure has been applied to quantify structure in diverse contexts [4][5][6].Meanwhile in thermodynamics, better originally described heat engines that produce more work with less wasted heat. This carries through to modern thermodynamics: the best approach fo...

show abstract

mentioning

confidence: 84%

Thermodynamics of complexity and pattern manipulation

et al. 2017

View full text Add to dashboard Cite

show abstract

“…[12,13] shows that E = I[S + ; S − ] by applying the causal shielding relations in Eqs. (1) and (2).…”

Section: Background and Notationmentioning

confidence: 99%

“…Together, a process' causal irreversibility [12,13] is defined as the difference between the forward and reversetime statistical complexities:…”

Section: B Informational Architecturementioning

confidence: 99%

Informational and Causal Architecture of Continuous-time Renewal Processes

Marzen

Crutchfield

2017

J Stat Phys

Self Cite

View full text Add to dashboard Cite

We introduce the minimal maximally predictive models ( -machines) of processes generated by certain hidden semi-Markov models. Their causal states are either hybrid discrete-continuous or continuous random variables and causal-state transitions are described by partial differential equations. Closed-form expressions are given for statistical complexities, excess entropies, and differential information anatomy rates. We present a complete analysis of the -machines of continuous-time renewal processes and, then, extend this to processes generated by unifilar hidden semi-Markov models and semi-Markov models. Our information-theoretic analysis leads to new expressions for the entropy rate and the rates of related information measures for these very general continuous-time process classes.

show abstract

“…Within computational mechanics, a process P( ↔ X) is viewed as a communication channel that transmits information from the past to the future, storing information in the present-presumably in some internal states, variables, or degrees of freedom [24]. One can ask a simple question, then: how much information does the past share with the future?…”

Section: Causal Statesmentioning

confidence: 99%

Optimal causal inference: Estimating stored information and approximating causal architecture

Still

Crutchfield

Ellison

2010

Chaos: An Interdisciplinary Journal of Nonlinear Science

Self Cite

102

View full text Add to dashboard Cite

We introduce an approach to inferring the causal architecture of stochastic dynamical systems that extends rate distortion theory to use causal shielding-a natural principle of learning. We study two distinct cases of causal inference: optimal causal filtering and optimal causal estimation.Filtering corresponds to the ideal case in which the probability distribution of measurement sequences is known, giving a principled method to approximate a system's causal structure at a desired level of representation. We show that, in the limit in which a model complexity constraint is relaxed, filtering finds the exact causal architecture of a stochastic dynamical system, known as the causal-state partition. From this, one can estimate the amount of historical information the process stores. More generally, causal filtering finds a graded model-complexity hierarchy of approximations to the causal architecture. Abrupt changes in the hierarchy, as a function of approximation, capture distinct scales of structural organization.For nonideal cases with finite data, we show how the correct number of underlying causal states can be found by optimal causal estimation. A previously derived model complexity control term allows us to correct for the effect of statistical fluctuations in probability estimates and thereby avoid over-fitting. Natural systems compute intrinsically and produce information. This organization, often only indirectly accessible to an observer, is reflected to varying degrees in measured time series. Nonetheless, this information can be used to build models of varying complexity that capture the causal architecture of the underlying system and allow one to estimate its information processing capabilities. We investigate two cases. The first is when a model builder wishes to find a more compact representation than the true one. This occurs, for example, when one is willing to incur the cost of a small increase in error for a large reduction in model size. The second case concerns the empirical setting in which only a finite amount of data is available. There one wishes to avoid over-fitting a model to a particular data set.

show abstract

Time’s Barbed Arrow: Irreversibility, Crypticity, and Stored Information

Cited by 99 publications

References 17 publications

Thermodynamics of complexity and pattern manipulation

Thermodynamics of complexity and pattern manipulation

Informational and Causal Architecture of Continuous-time Renewal Processes

Optimal causal inference: Estimating stored information and approximating causal architecture

Contact Info

Product

Resources

About