The Modern Mathematics of Deep Learning

Berner, Julius; Kutyniok, Gitta; Petersen, Philipp

doi:10.48550/arxiv.2105.04026

Cited by 20 publications

(28 citation statements)

References 70 publications

(85 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the tremendous success of deep learning in various applications, also several downsides can be observed. Particular problematic for their applicability are, for instance, instabilities due to adversarial examples in image classification [64] and image reconstruction [4], the fact that deep neural networks still act as a black box lacking explainability [40], [71], and in general a vast gap between theory and practise (see, for instance, [1], [10]). This led to the acknowledgement that for reliable deep learning a better understanding as well as a mathematical theory of deep learning is in great demand.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Limitations of Deep Learning for Inverse Problems on Digital Hardware

Boche¹,

Fono²,

Kutyniok³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks have seen tremendous success over the last years. Since the training is performed on digital hardware, in this paper, we analyze what actually can be computed on current hardware platforms modeled as Turing machines, which would lead to inherent restrictions of deep learning. For this, we focus on the class of inverse problems, which, in particular, encompasses any task to reconstruct data from measurements. We prove that finite-dimensional inverse problems are not Banach-Mazur computable for small relaxation parameters. In fact, our result even holds for Borel-Turing computability., i.e., there does not exist an algorithm which performs the training of a neural network on digital hardware for any given accuracy. This establishes a conceptual barrier on the capabilities of neural networks for finite-dimensional inverse problems given that the computations are performed on digital hardware.

show abstract

Section: Contributionsmentioning

confidence: 99%

“…In this section, we give a short introduction to deep learning with a particular focus on solving inverse problems [2], [37], [47], [50], [58]. For a comprehensive depiction of deep learning theory we refer to [30] and [10].…”

Section: Deep Learning For Inverse Problemsmentioning

confidence: 99%

Limitations of Deep Learning for Inverse Problems on Digital Hardware

Boche¹,

Fono²,

Kutyniok³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Obstacles in the theoretical foundation include the higher-order nonlinear structures due to the stacking of multiple layers and the excessive number of network parameters in state of the art networks. For some recent surveys, see [5,6].…”

Section: 𝑅( A) − 𝑅 𝑛 ( A)mentioning

confidence: 99%

On generalization bounds for deep networks based on loss surface implicit regularization

Imaizumi¹,

Schmidt-Hieber²

2022

Preprint

View full text Add to dashboard Cite

RIKEN AIP A. The classical statistical learning theory says that fitting too many parameters leads to overfitting and poor performance. That modern deep neural networks generalize well despite a large number of parameters contradicts this finding and constitutes a major unsolved problem towards explaining the success of deep learning. The implicit regularization induced by stochastic gradient descent (SGD) has been regarded to be important, but its specific principle is still unknown. In this work, we study how the local geometry of the energy landscape around local minima affects the statistical properties of SGD with Gaussian gradient noise. We argue that under reasonable assumptions, the local geometry forces SGD to stay close to a low dimensional subspace and that this induces implicit regularization and results in tighter bounds on the generalization error for deep neural networks. To derive generalization error bounds for neural networks, we first introduce a notion of stagnation sets around the local minima and impose a local essential convexity property of the population risk. Under these conditions, lower bounds for SGD to remain in these stagnation sets are derived. If stagnation occurs, we derive a bound on the generalization error of deep neural networks involving the spectral norms of the weight matrices but not the number of network parameters. Technically, our proofs are based on controlling the change of parameter values in the SGD iterates and local uniform convergence of the empirical loss functions based on the entropy of suitable neighborhoods around local minima. Our work attempts to better connect non-convex optimization and generalization analysis with uniform convergence.

show abstract

“…, (5,6,7). In the second level, the first and third nodes fire if one or more dependent nodes fire, and the second (dominant) node fires if two or more dependent nodes fire.…”

Section: Small-batch Trainingmentioning

confidence: 99%

“…Deep learning is recognized as a monumentally successful approach to many data-extensive applications in image recognition, natural language processing, and board game programs [1,2,3]. Despite extensive efforts [4,5,6], however, our theoretical understanding of how this increasingly popular machinery works and why it is so effective remains incomplete. This is exemplified by the substantial vacuum between the highly sophisticated training paradigm of modern neural networks and the capabilities of existing theories.…”

Section: Introductionmentioning

confidence: 99%

Neurashed: A Phenomenological Model for Imitating Deep Learning Training

Su¹

2021

Preprint

View full text Add to dashboard Cite

To advance deep learning methodologies in the next decade, a theoretical framework for reasoning about modern neural networks is needed. While efforts are increasing toward demystifying why deep learning is so effective, a comprehensive picture remains lacking, suggesting that a better theory is possible. We argue that a future deep learning theory should inherit three characteristics: a hierarchically structured network architecture, parameters iteratively optimized using stochastic gradient-based methods, and information from the data that evolves compressively. As an instantiation, we integrate these characteristics into a graphical model called neurashed. This model effectively explains some common empirical patterns in deep learning. In particular, neurashed enables insights into implicit regularization, information bottleneck, and local elasticity. Finally, we discuss how neurashed can guide the development of deep learning theories.

show abstract

The Modern Mathematics of Deep Learning

Cited by 20 publications

References 70 publications

Limitations of Deep Learning for Inverse Problems on Digital Hardware

Limitations of Deep Learning for Inverse Problems on Digital Hardware

On generalization bounds for deep networks based on loss surface implicit regularization

Neurashed: A Phenomenological Model for Imitating Deep Learning Training

Contact Info

Product

Resources

About