The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and Smale’s 18th problem

Colbrook, Matthew J.; Antun, Vegard; Hansen, Anders C.

doi:10.1073/pnas.2107151119

Cited by 73 publications

(57 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is a measure of efficiency for our network construction, where in order to guarantee reconstruction within an error proportional to χ, the required depth of the network scales logarithmically in χ. In particular, this is analogous to what is observed in [29,Thm. 5.10].…”

Section: Discussionsupporting

confidence: 85%

“…There are several key points worth mentioning about Theorem 1, specifically how it can be generalized and comparisons to the influential work in [29].…”

Section: Discussionmentioning

confidence: 99%

“…21]). In [29], the authors studied the case of Fourier imaging with the Discrete Fourier Transform (DFT) and binary imaging with the Discrete Walsh-Hadamard Transform (DHT). To provide performance guarantees, they considered a model class consisting of images that are approximately sparse in levels in the discrete orthonormal Haar wavelet basis.…”

Section: Contributions and Related Workmentioning

confidence: 99%

“…Like [29], our work involves unrolling an optimization algorithm. Unrolling is an important concept in deep learning for inverse problems, one which has been used in some of the best-performing networks.…”

Section: Contributions and Related Workmentioning

confidence: 99%

“…For imaging, various different optimization algorithms have been unrolled. These include ISTA [41,42], ADMM [43][44][45], the primal-dual iteration [29,46,47], block coordinate descent [48], POCS [49], as well as approximate message passing algorithms [50], Neumann series [51], field of experts models [52] and various others, including [53]. While successful, we remark that simply unrolling an optimization solver does not ensure stability and robustness of a deep learning procedure [14].…”

Section: Contributions and Related Workmentioning

confidence: 99%

See 4 more Smart Citations

NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems

Neyra-Nesterenko¹,

Adcock²

2022

Preprint

View full text Add to dashboard Cite

Solving inverse problems is a fundamental component of science, engineering and mathematics. With the advent of deep learning, deep neural networks have significant potential to outperform existing state-of-theart, model-based methods for solving inverse problems. However, it is known that current data-driven approaches face several key issues, notably instabilities and hallucinations, with potential impact in critical tasks such as medical imaging. This raises the key question of whether or not one can construct stable and accurate deep neural networks for inverse problems. In this work, we present a novel construction of an accurate, stable and efficient neural network for inverse problems with general analysis-sparse models. To construct the network, we unroll NESTA, an accelerated first-order method for convex optimization. Combined with a compressed sensing analysis, we prove accuracy and stability. Finally, a restart scheme is employed to enable exponential decay of the required network depth, yielding a shallower, and consequently more efficient, network. We showcase this approach in the case of Fourier imaging, and verify its stability and performance via a series of numerical experiments. The key impact of this work is to provide theoretical guarantees for computing and developing stable neural networks in practice.

show abstract

Section: Discussionsupporting

confidence: 85%

“…There are several key points worth mentioning about Theorem 1, specifically how it can be generalized and comparisons to the influential work in [29].…”

Section: Discussionmentioning

confidence: 99%

Section: Contributions and Related Workmentioning

confidence: 99%

Section: Contributions and Related Workmentioning

confidence: 99%

Section: Contributions and Related Workmentioning

confidence: 99%

See 3 more Smart Citations

NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems

Neyra-Nesterenko¹,

Adcock²

2022

Preprint

View full text Add to dashboard Cite

show abstract

Coevolution of machine learning and process‐based modelling to revolutionize Earth and environmental sciences: A perspective

Razavi

Hannah

Elshorbagy

et al. 2022

Hydrological Processes

View full text Add to dashboard Cite

Machine learning (ML) applications in Earth and environmental sciences (EES) have gained incredible momentum in recent years. However, these ML applications have largely evolved in ‘isolation’ from the mechanistic, process‐based modelling (PBM) paradigms, which have historically been the cornerstone of scientific discovery and policy support. In this perspective, we assert that the cultural barriers between the ML and PBM communities limit the potential of ML, and even its ‘hybridization’ with PBM, for EES applications. Fundamental, but often ignored, differences between ML and PBM are discussed as well as their strengths and weaknesses in light of three overarching modelling objectives in EES, (1) nowcasting and prediction, (2) scenario analysis, and (3) diagnostic learning. The paper ponders over a ‘coevolutionary’ approach to model building, shifting away from a borrowing to a co‐creation culture, to develop a generation of models that leverage the unique strengths of ML such as scalability to big data and high‐dimensional mapping, while remaining faithful to process‐based knowledge base and principles of model explainability and interpretability, and therefore, falsifiability.

show abstract

Knowledge‐driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un‐supervised learning

Wang,

Wu,

Jia

et al. 2024

Magnetic Resonance in Med

View full text Add to dashboard Cite

Deep learning (DL) has emerged as a leading approach in accelerating MRI. It employs deep neural networks to extract knowledge from available datasets and then applies the trained networks to reconstruct accurate images from limited measurements. Unlike natural image restoration problems, MRI involves physics‐based imaging processes, unique data properties, and diverse imaging tasks. This domain knowledge needs to be integrated with data‐driven approaches. Our review will introduce the significant challenges faced by such knowledge‐driven DL approaches in the context of fast MRI along with several notable solutions, which include learning neural networks and addressing different imaging application scenarios. The traits and trends of these techniques have also been given which have shifted from supervised learning to semi‐supervised learning, and finally, to unsupervised learning methods. In addition, MR vendors' choices of DL reconstruction have been provided along with some discussions on open questions and future directions, which are critical for the reliable imaging systems.

show abstract

The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and Smale’s 18th problem

Cited by 73 publications

References 74 publications

NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems

NESTANets: Stable, accurate and efficient neural networks for analysis-sparse inverse problems

Coevolution of machine learning and process‐based modelling to revolutionize Earth and environmental sciences: A perspective

Knowledge‐driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un‐supervised learning

Contact Info

Product

Resources

About