Generic predictions of output probability based on complexities of inputs and outputs

Abstract

For a broad class of input-output maps, arguments based on the coding theorem from algorithmic information theory (AIT) predict that simple (low Kolmogorov complexity) outputs are exponentially more likely to occur upon uniform random sampling of inputs than complex outputs are. Here, we derive probability bounds that are based on the complexities of the inputs as well as the outputs, rather than just on the complexities of the outputs. The more that outputs deviate from the coding theorem bound, the lower the complexity of their inputs. Our new bounds are tested for an RNA sequence to structure map, a finite state transducer and a perceptron. These results open avenues for AIT to be more widely used in physics. PACS numbers:Deep links between physics and theories of computation [1, 2] are being increasingly exploited to uncover new fundamental physics and to provide novel insights into theories of computation. For example, advances in understanding quantum entanglement are often expressed in sophisticated information theoretic language, while providing new results in computational complexity theory such as polynomial time algorithms for integer factorization [3]. These connections are typically expressed in terms of Shannon information, with its natural analogy with thermodynamic entropy.There is, however, another branch of information theory, called algorithmic information theory (AIT) [4], which is concerned with the information content of individual objects. It has been much less applied in physics (although notable exceptions occur, see [5] for a recent overview). Reasons for this relative lack of attention include that AIT's central concept, the Kolmogorov complexity K U (x) of a string x, defined as the length of the shortest program that generates x on a universal Turing machine (UTM) U , is formally uncomputable due to its link to the famous halting problem of UTMs [6]. Moreover, many important results, such as the invariance theorem which states that for two UTMs U and W , the Kolmogorov complexities K U (x) = K W (x) + O(1) are equivalent, hold asymptotically up to O(1) terms that are independent of x, but not always well understood, and therefore hard to control.Another reason applications of AIT to many practical problems have been hindered can be understood in terms of hierarchies of computing power. For example, one of the oldest such categorisations, the Chomsky hierarchy [7], ranks automata into four different classes, of which the UTMs are the most powerful, and finite state machines (FSMs) are the least. Many key results in AIT are derived by exploiting the power of UTMs. Interestingly, if physical processes can be mapped onto UTMs, then certain properties can be shown to be uncom-putable [8, 9]. However, many problems in physics are fully computable, and therefore lower on the Chomsky hierarchy than UTMs. For example, finite Markov processes are equivalent to FSMs, and RNA secondary structure (SS) folding algorithms can be recast as context-free grammars, the second level in the hiearchy. Th...

Symmetry and simplicity spontaneously emerge from the algorithmic nature of evolution

“…Similarly, a larger robustness may also enhance the ability of a system to encode cryptic variation, facilitating access to new phenotypes [44]. A natural tendency towards simpler and more robust structures may therefore facilitate the emergence of modularity, where individual components can evolve independently [45], and so make living systems more globally evolvable.…”

“…( 17) only provides an upper bound. Nevertheless, a statistical lower bound can be derived [11,45] showing that most of the probability weight in P (x) will be close to the bound (17). Interestingly, outputs that are far from the bound can be shown to have inputs (in this case genomes or bonding patterns) that themselves are unusually simple [45].…”

“…whereK(p) is an appropriate approximation to the true Kolmogorov complexity K(p), and a and b are constants that depend on the map, but not on p. While Eq. ( 1) is only an upper bound, it can be shown [22] that outputs generated by uniform random sampling of inputs are likely to be close to the bound. In extensive tests, Eq.…”

“…where the simplest strings 0 n and 1 n are separated out because N w (x) which assigns complexity K = 1 to the string 0 or 1, but complexity 2 to 0 n or 1 n for n ≥ 2, whereas the true Kolmogorov complexity of such a trivial string actually scales as log 2 (n), as one only needs to encode n. In ref. [11,24,45] this complexity measure was empirically shown to work well for a wide diversity of maps, ranging from sets of coupled differential equations, to the RNA SS GP map, to a finite-state transducer, to neural network models for deep learning. This success gives us confidence to use it here for the RNA and the GRN.…”

A Note on A Priori Forecasting and Simplicity Bias in Time Series

“…Moreover, algorithmic probability estimates have successfully been made via a weaker form of Levin's coding theorem [13], applicable in real-world contexts. This weaker form was applied in a range of inputoutput maps to make a priori predictions regarding the probability of different shapes and patterns, such as the probability of different RNA shapes appearing on a random choice of genetic sequence, or the probability of differential equation solution profile shapes, on random choice of input parameters, and several other examples [13,14]. Surprisingly, it was found that probability estimates could be made directly from the complexities of the shapes themselves, without recourse to the details of the map or reference to how the shapes were generated.…”