Predictability, Complexity, and Learning

Abstract

We define predictive information I(pred)(T) as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times T:I(pred)(T) can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then I(pred)(T) grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power-law growth is associated, for example, with the learning of infinite parameter (or nonparametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and the analysis of physical systems through statistical mechanics and dynamical systems theory. Furthermore, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of I(pred)(T) provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in problems in physics, statistics, and biology.

Statistical physics of a model binary genetic switch with linear feedback

“…Substituting (27) in (26) we deduce, using the definition of the hypergeometric function (24) and noting…”

“…An alternative approach is to study the dynamics of stochastic flipping between two stable states using stochastic simulations [19,20,21], by numerically integrating the master equation [22], or by path integral-type approaches [23]. This dynamical problem bears some resemblance to the Kramers problem of escape from a free energy minimum [24,25], and one expects on general grounds that the typical time spent in one of the bistable states should be exponentially large in the typical number of proteins present in the state. This has been confirmed, at least for cooperative toggle switches formed of mutually repressing genes [19,20].…”

Multivariate multiscale entropy: A tool for complexity analysis of multichannel data

“…Among them, the time delay embedded reconstruction [1] provides a general framework for the estimation of invariant quantities (in terms of smooth transformations of the state space of the attractor) of the original system, such as attractor dimensions, Lyapunov exponents and entropies [2][3] [4]. Information theoretic measures of structural complexity include effective measure complexity (EMC) [5], excess entropy [6], and predictive information [7] [8].…”

Predictive Information in Gaussian Processes with Application to Music Analysis

“…For example, the excess entropy [1] is the mutual information between the semi-infinite past and future of a random process. Addressing the observation that some processes with long-range dependencies have infinite excess entropy [2], Bialek et al [3] introduced the predictive information as the mutual information between a finite segment of a process and the infinite future following it, and studied its behaviour, especially in relation to learning in statistical models. In previous work [4], we defined the predictive information rate (PIR) of a random process as the average information in one observation about future observations yet to be made given the observations made so far; thus, it quantifies the new information in observations made sequentially.…”