Deep Learning Techniques for Inverse Problems in Imaging

Ongie, Greg; Jalal, Ajil; Metzler, Christopher A.; Baraniuk, Richard G.; Dimakis, Alexandros G.; Willett, Rebecca

doi:10.1109/jsait.2020.2991563

Cited by 442 publications

(294 citation statements)

References 116 publications

Supporting

Mentioning

260

Contrasting

Unclassified

Order By: Relevance

“…Generally, such systems are characterized by how the information is encoded (forward process) and decoded (inverse problem) from the measurements. Recently, physics-based learning [1] has demonstrated the ability to directly optimize a computational imaging system's performance [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]. Physics-based learning takes advantage of both the known physics of the system's forward model process and the architecture of the decoder's iterative optimizer to build a differentiable neural network that is efficiently parameterized by only a limited number of learnable variables, thereby enabling training using less data [11], [8], [9], while still retaining the robustness and interpretability associated with conventional physics-based inverse problems.…”

Section: Introductionmentioning

confidence: 99%

“…Learning for Computational Imaging: Methods can be categorized into physics-based and physics-free approaches. Physics-based [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14] methods consider the inclusion of the forward model process and the structure of inverse problem optimization in the physics-based network. Physics-free approaches use a black box architecture (e.g.…”

Section: Introductionmentioning

confidence: 99%

“…In addition, physics-based methods are more robust in experimental settings and inherit the interpretability associated with classic inverse problems. The recent work by Ongie et al [14] has a comprehensive review of these methods.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Memory-Efficient Learning for Large-Scale Computational Imaging

Kellman

Zhang

Markley

et al. 2020

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

Critical aspects of computational imaging systems, such as experimental design and image priors, can be optimized through deep networks formed by the unrolled iterations of classical physics-based reconstructions. Termed physics-based networks, they incorporate both the known physics of the system via its forward model, and the power of deep learning via datadriven training. However, for realistic large-scale physics-based networks, computing gradients via backpropagation is infeasible due to the memory limitations of graphics processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable physics-based learning for large-scale computational imaging systems. We demonstrate our method on a compressed sensing example, as well as two large-scale real-world systems: 3D multichannel magnetic resonance imaging and super-resolution optical microscopy.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Memory-Efficient Learning for Large-Scale Computational Imaging

Kellman

Zhang

Markley

et al. 2020

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

show abstract

“…Differentiable model-based approaches for image reconstruction have been introduced in several domains of imaging [21], for example in ptychography [31]. Instead of directly optimizing model parameters, an additional deep neural network (a deep image prior [41,42]) has also been introduced, for example for phase imaging [32,33] or ptychography [34].…”

Section: Discussionmentioning

confidence: 99%

“…Similar situations where models based on a well known underlying physical process are learned from data are also encountered in other imaging modalities [21], and more broadly many areas of engineering and physics (for example [22][23][24][25][26][27][28][29][30][31][32][33][34]). To take advantage of such prior information, methods have been developed that combine physical process models with machine learning optimization.…”

Section: Introductionmentioning

confidence: 99%

Differentiable model-based adaptive optics with transmitted and reflected light

Vishniakou¹,

Seelig²

2020

Opt. Express

View full text Add to dashboard Cite

Aberrations limit optical systems in many situations, for example when imaging in biological tissue. Machine learning offers novel ways to improve imaging under such conditions by learning inverse models of aberrations. Learning requires datasets that cover a wide range of possible aberrations, which however becomes limiting for more strongly scattering samples, and does not take advantage of prior information about the imaging process. Here, we show that combining model-based adaptive optics with the optimization techniques of machine learning frameworks can find aberration corrections with a small number of measurements. Corrections are determined in a transmission configuration through a single aberrating layer and in a reflection configuration through two different layers at the same time. Additionally, corrections are not limited by a predetermined model of aberrations (such as combinations of Zernike modes). Focusing in transmission can be achieved based only on reflected light, compatible with an epidetection imaging configuration.

show abstract

Regionalization in a global hydrologic deep learning model: from physical descriptors to random vectors

Khandelwal

Jia

et al. 2022

Preprint

View full text Add to dashboard Cite

Streamflow prediction is a long-standing hydrologic problem. Development of models for streamflow prediction often requires incorporation of catchment physical descriptors to characterize the associated complex hydrological processes. Across different scales of catchments, these physical descriptors also allow models to extrapolate hydrologic information from one catchment to others, a process referred to as "regionalization". Recently, in gauged basin scenarios, deep learning models have been shown to achieve state of the art regionalization performance by building a global hydrologic model. These models predict streamflow given catchment physical descriptors and weather forcing data. However, these physical descriptors are by their nature uncertain, sometimes incomplete, or even unavailable in certain cases, which limits the applicability of this approach. In this paper, we show that by assigning a vector of random values as a surrogate for catchment physical descriptors, we can achieve robust regionalization performance under a gauged prediction scenario. Our results show that the deep learning model using our proposed random vector approach achieves a predictive performance comparable to that of the model using actual physical descriptors. The random vector approach yields robust performance under different data sparsity scenarios and deep learning model selections. Furthermore, based on the use of random vectors, high-dimensional characterization identifies the uniqueness of catchments, thereby improving regionalization performance in gauged basin scenario when physical descriptors are uncertain, or insufficient.

show abstract

Deep Learning Techniques for Inverse Problems in Imaging

Cited by 442 publications

References 116 publications

Memory-Efficient Learning for Large-Scale Computational Imaging

Memory-Efficient Learning for Large-Scale Computational Imaging

Differentiable model-based adaptive optics with transmitted and reflected light

Regionalization in a global hydrologic deep learning model: from physical descriptors to random vectors

Contact Info

Product

Resources

About