The extended Smale's 9th problem -- On computational barriers and paradoxes in estimation, regularisation, computer-assisted proofs and learning

Bastounis, Alexander; Hansen, Anders C.; Vlačić, Verner

doi:10.48550/arxiv.2110.15734

Cited by 2 publications

(6 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This extended model yields an extended version of Smale's 9th problem from the list of mathematical problems for the 21st century [23]). A very simplified summary of the results in [19] are as follows.…”

Section: Theoretical Preliminaries and Generalised Hardness Of Approx...mentioning

confidence: 99%

“…HA is almost exclusively associated with combinatorial computational problems, however -as was discovered in [19] and subsequently in [20,21] and [22] -a generalised form of this phenomenon, namely GHA, can happen in other areas of the computational sciences regardless of the P vs NP question. GHA can be explained as follows: Given an ϵ > 0, the ϵ-approximate computational problem is the problem of computing an approximation that is no more than ϵ away from the true solution -in some appropriate predefined metric.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Which neural networks can be computed by an algorithm? – Generalised hardness of approximation meets Deep Learning

Thesing

Hansen

2023

Proc Appl Math and Mech

View full text Add to dashboard Cite

Classical hardness of approximation (HA) is the phenomenon that, assuming P ≠ NP, one can easily compute an ϵ‐approximation to the solution of a discrete computational problem for ϵ > ϵ0 > 0, but for ϵ < ϵ0 – where ϵ0 is the approximation threshold – it becomes intractable. Recently, a similar yet more general phenomenon has been documented in AI: Generalised hardness of approximation (GHA). This phenomenon includes the following occurrence: For any approximation threshold ϵ1 > 0, there are AI problems for which provably there exist stable neural networks (NNs) that solve the problem, but no algorithm can compute any NN that approximates the AI problem to ϵ1‐accuracy. Moreover, this issue is independent of the P vs NP question and thus is a rather different mathematical phenomenon than HA. GHA implies that the universal approximation theorem for NNs only provides a partial understanding of the power of NNs in AI. Thus, a classification theory describing which NNs can be computed by algorithms to particular accuracies is needed to fill this gap. We initiate such a theory by showing the correspondence between the functions that can be computed to ϵ‐accuracy by an algorithm and those functions that can be approximated by NNs which can be computed to ϵ̂‐accuracy by an algorithm. In particular, the approximation thresholds ϵ and ϵ̂ cannot differ by more than a factor of 12. This means that computing function approximations through NNs will be optimal – in the sense of best approximation accuracy achievable by an algorithm – up to a small constant, compared to any other computational technique.

show abstract

Section: Theoretical Preliminaries and Generalised Hardness Of Approx...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Which neural networks can be computed by an algorithm? – Generalised hardness of approximation meets Deep Learning

Thesing

Hansen

2023

Proc Appl Math and Mech

View full text Add to dashboard Cite

show abstract

“…The existence and computability of neural network has been studied before, namely, for classification problems [8] as well as for inverse problems [23]. The crucial difference to our approach lies in the applied computation model.…”

Section: Previous Workmentioning

confidence: 99%

“…Thus, the question studied in [8] and [23] is the following: Given arbitrarily accurate approximations to any complex number such as, in particular, the training samples, do there exist algorithms that yield accurate neural networks or are there computational barriers in the training of neural networks under these conditions? Although these are interesting questions on their own and the results indicate that indeed computational barriers exist, the assumptions are not realistic for autonomous digital computations.…”

Section: Previous Workmentioning

confidence: 99%

“…As discussed before, this constitutes a realistic model for autonomous digital computations. Consequently, for showing non-computability results on Turing machines, we have to exploit different proof techniques as applied in [8], [23]. In particular, we require various methods from the theory of recursive functions and mathematical logic to establish our result.…”

Section: Previous Workmentioning

confidence: 99%

See 1 more Smart Citation

Limitations of Deep Learning for Inverse Problems on Digital Hardware

Boche¹,

Fono²,

Kutyniok³

2022

Preprint

View full text Add to dashboard Cite

Deep neural networks have seen tremendous success over the last years. Since the training is performed on digital hardware, in this paper, we analyze what actually can be computed on current hardware platforms modeled as Turing machines, which would lead to inherent restrictions of deep learning. For this, we focus on the class of inverse problems, which, in particular, encompasses any task to reconstruct data from measurements. We prove that finite-dimensional inverse problems are not Banach-Mazur computable for small relaxation parameters. In fact, our result even holds for Borel-Turing computability., i.e., there does not exist an algorithm which performs the training of a neural network on digital hardware for any given accuracy. This establishes a conceptual barrier on the capabilities of neural networks for finite-dimensional inverse problems given that the computations are performed on digital hardware.

show abstract

The extended Smale's 9th problem -- On computational barriers and paradoxes in estimation, regularisation, computer-assisted proofs and learning

Cited by 2 publications

References 82 publications

Which neural networks can be computed by an algorithm? – Generalised hardness of approximation meets Deep Learning

Which neural networks can be computed by an algorithm? – Generalised hardness of approximation meets Deep Learning

Limitations of Deep Learning for Inverse Problems on Digital Hardware

Contact Info

Product

Resources

About