Compression is Comprehension and the Unreasonable Effectiveness of Digital Computation in the Natural World

Zenil, Héctor

doi:10.1142/9789811200076_0011

Cited by 6 publications

(13 citation statements)

References 34 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Figure 5 , equivalent to the pixel attacks for discrete objects, we show that the parameter identification is robust to even more than 25% of additive noise. Operating in a low complexity regime—as above—is compatible with a principal of parsimony such as Ockham’s razor, which is empirically found to be able to explain data simplicity bias ( Zenil & Delahaye, 2010 ; Dingle et al, 2018 ; Zenil et al, 2018 ), suggesting that the best explanation is the simplest, but also that what is modeled is not algorithmically random ( Zenil, 2020 ).…”

Section: Resultsmentioning

confidence: 52%

“…The assumption that the optimum parameters have an underlying simplicity bias is strong, but has been investigated ( Dingle et al, 2018 ; Zenil, 2020 ) and is compatible with principles of parsimony. This bias favors objects of interest that are of low algorithmic complexity, though they may appear random, For example, the decimal expansions of the constant π or e to an accuracy of 32 bits have a BDM value of 666.155 and 674.258, respectively, while the expected BDM for a random binary string of the same size is significantly larger:

.…”

Section: Algorithmic Optimization Methodologymentioning

confidence: 99%

See 1 more Smart Citation

Algorithmic Probability-Guided Machine Learning on Non-Differentiable Spaces

Hernández-Orozco

Zenil

Riedel

et al. 2021

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

We show how complexity theory can be introduced in machine learning to help bring together apparently disparate areas of current research. We show that this model-driven approach may require less training data and can potentially be more generalizable as it shows greater resilience to random attacks. In an algorithmic space the order of its element is given by its algorithmic probability, which arises naturally from computable processes. We investigate the shape of a discrete algorithmic space when performing regression or classification using a loss function parametrized by algorithmic complexity, demonstrating that the property of differentiation is not required to achieve results similar to those obtained using differentiable programming approaches such as deep learning. In doing so we use examples which enable the two approaches to be compared (small, given the computational power required for estimations of algorithmic complexity). We find and report that 1) machine learning can successfully be performed on a non-smooth surface using algorithmic complexity; 2) that solutions can be found using an algorithmic-probability classifier, establishing a bridge between a fundamentally discrete theory of computability and a fundamentally continuous mathematical theory of optimization methods; 3) a formulation of an algorithmically directed search technique in non-smooth manifolds can be defined and conducted; 4) exploitation techniques and numerical methods for algorithmic search to navigate these discrete non-differentiable spaces can be performed; in application of the (a) identification of generative rules from data observations; (b) solutions to image classification problems more resilient against pixel attacks compared to neural networks; (c) identification of equation parameters from a small data-set in the presence of noise in continuous ODE system problem, (d) classification of Boolean NK networks by (1) network topology, (2) underlying Boolean function, and (3) number of incoming edges.

show abstract

Section: Resultsmentioning

confidence: 52%

.…”

Section: Algorithmic Optimization Methodologymentioning

confidence: 99%

Algorithmic Probability-Guided Machine Learning on Non-Differentiable Spaces

Hernández-Orozco

Zenil

Riedel

et al. 2021

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…An open question is why output probability distributions generated by completely different reference universal Turing machines turn out very similar, despite concerns related to optimality and additive—or other (if not optimal)—constants which resonates with Chaitin’s optimism when learning about CTM [ 54 ]. I call this the “Unreasonable Effectiveness of Digital Computation in the Natural World” [ 77 ]. As is the case in other areas of mathematics, it could be that we are too careful (as no doubt we should be), whereas in practice, things do not correspond to theoretical expectations.…”

Section: Discussionmentioning

confidence: 99%

A Review of Methods for Estimating Algorithmic Complexity: Options, Challenges, and New Directions

Zenil

2020

Entropy

Self Cite

View full text Add to dashboard Cite

Some established and also novel techniques in the field of applications of algorithmic (Kolmogorov) complexity currently co-exist for the first time and are here reviewed, ranging from dominant ones such as statistical lossless compression to newer approaches that advance, complement and also pose new challenges and may exhibit their own limitations. Evidence suggesting that these different methods complement each other for different regimes is presented and despite their many challenges, some of these methods can be better motivated by and better grounded in the principles of algorithmic information theory. It will be explained how different approaches to algorithmic complexity can explore the relaxation of different necessary and sufficient conditions in their pursuit of numerical applicability, with some of these approaches entailing greater risks than others in exchange for greater relevance. We conclude with a discussion of possible directions that may or should be taken into consideration to advance the field and encourage methodological innovation, but more importantly, to contribute to scientific discovery. This paper also serves as a rebuttal of claims made in a previously published minireview by another author, and offers an alternative account.

show abstract

“…In this regard, we mention how in a traditional view, a model is better when it can explain more with less, according to Chaitin's compression is comprehension [37].…”

Section: Incomprehensibilitymentioning

confidence: 99%

On Theoretical Incomprehensibility

Minati¹

2019

Philosophies

View full text Add to dashboard Cite

This contribution tentatively outlines the presumed conceptual duality between the issues of incompleteness and incomprehensibility—The first being more formal in nature and able to be declined in various ways until specified in the literature as theoretical incompleteness. This is theoretical and not temporary, which is admissible and the completion prosecutable. As considered in the literature, theoretical incompleteness refers to uncertainty principles in physics, incompleteness in mathematics, oracles for the Turing Machine, logical openness as the multiplicity of models focusing on coherence more than the optimum selections, fuzziness, quasiness, e.g., quasi-crystals, quasi-systems, and quasi-periodicity, which are intended as the space of equivalences that allow for coherent processes of emergence. The issue of incomprehensibility cannot be considered without reference to an agent endowed with cognitive abilities. In this article, we consider incomprehensibility as understood here as not generally scientifically explicable, i.e., with the available knowledge, as such incomprehensibility may be temporary, pending theoretical and technological advances, or deemed to be absolute as coincident with eventual definitive, theoretical non-explicability, and incomprehensibility. We considered the theoretically incomprehensibility mostly in three main ways: as the inexhaustibility of the multiplicity of constructivist reality as given by the theoretically incomprehensible endless loop of incomprehensible–comprehensible, and by existential questions. Moreover, theoretical incomprehensibility is intended as evidence of the logical openness of both the world and of understanding itself. The role of theoretical incomprehensibility is intended as a source of theoretical research issues such as paradoxes and paradigm shifts, where it is a matter of having cognitive strategies and approaches to look for, cohabit, combine, and use comprehensibility and (theoretical) incomprehensibility. The usefulness of imaginary numbers comes to mind. Can we support such research for local, temporary, and theoretical incomprehensibility with suitable approaches such as software tools, for instance, that simulate the logical frameworks of incomprehensibility? Is this a step toward a kind of artificial creativity leading to paradigm shifts? The most significant novelty of the article lies in the focus on the concept of theoretical incomprehensibility and distinguishing it from incomprehensibility and considering different forms of understanding. It is a matter of identifying strategies to act and coexist with the theoretically incomprehensible, to represent and use it, for example when dealing with imaginary numbers and quantum contexts where classical comprehensibility is theoretically impossible. Can we think of forms of non-classical understanding? In this article, these topics are developed in conceptual and philosophical ways.

show abstract

Compression is Comprehension and the Unreasonable Effectiveness of Digital Computation in the Natural World

Cited by 6 publications

References 34 publications

Algorithmic Probability-Guided Machine Learning on Non-Differentiable Spaces

Algorithmic Probability-Guided Machine Learning on Non-Differentiable Spaces

A Review of Methods for Estimating Algorithmic Complexity: Options, Challenges, and New Directions

On Theoretical Incomprehensibility

Contact Info

Product

Resources

About