A Regularized Gauss–Newton Trust Region Approach to Imaging in Diffuse Optical Tomography

Sturler, Eric de; Kilmer, Misha E.

doi:10.1137/100798181

Cited by 17 publications

(37 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A specific question is if the improvements in the forward propagation and regularization proposed in this paper will lead to better generalization properties of subsampled second-order methods such as [13,44]. Another thrust of future work is the development of automatic parameter selection strategies for deep learning based on the approaches presented, e.g., in [19,26,48,24,18]. A particular challenge in this application is the nontrivial relationship between the regularization parameters chosen for the classification and forward propagation parameters.…”

Section: Discussionmentioning

confidence: 99%

Stable architectures for deep neural networks

2017

View full text Add to dashboard Cite

Deep neural networks have become invaluable tools for supervised machine learning, e.g., classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks.The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

show abstract

Section: Discussionmentioning

confidence: 99%

Stable architectures for deep neural networks

2017

View full text Add to dashboard Cite

show abstract

“…In the standard SAA approach, when convergence slows down or a minimum is found for the chosen sample (but not for the true problem), a new sample is chosen to improve the approximate solution. However, for our problem this approach leads to slow convergence and stagnation, unless we obtain more accurate estimates of the dominant singular components of the Jacobian and the corresponding components of the gradient (see also [10]). Hence, after exploiting the relatively fast initial convergence for our problem, we want to avoid stagnation of convergence in the next phase.…”

Section: A Stochastic Optimization Approachmentioning

confidence: 99%

“…This is the same noise level as used in [9]. Then, we reconstruct the absorption images using the PaLS [1] repesentation and TREGS [10] for optimization. 2D Experiment.…”

Section: Numerical Experimentsmentioning

confidence: 99%

Randomized Approach to Nonlinear Inversion Combining Random and Optimized Simultaneous Sources and Detectors

Aslan¹,

Sturler²,

Kilmer³

2019

SIAM J. Sci. Comput.

Self Cite

View full text Add to dashboard Cite

In partial differential equations-based (PDE-based) inverse problems with many measurements, many largescale discretized PDEs must be solved for each evaluation of the misfit or objective function. In the nonlinear case, evaluating the Jacobian requires solving an additional set of systems. This leads to a tremendous computational cost, and this is by far the dominant cost for these problems. Several authors have proposed randomization and stochastic programming techniques to drastically reduce the number of system solves by estimating the objective function using only a few appropriately chosen random linear combinations of the sources. While some have reported good solution quality at a greatly reduced cost, for our problem of interest, diffuse optical tomography, the approach often does not lead to sufficiently accurate solutions.We propose two improvements. First, to efficiently exploit Newton-type methods, we modify the stochastic estimates to include random linear combinations of detectors, drastically reducing the number of adjoint solves. Second, after solving to a modest tolerance, we compute a few simultaneous sources and detectors that maximize the Frobenius norm of the sampled Jacobian to improve the rate of convergence and obtain more accurate solutions. We complement these optimized simultaneous sources and detectors by random simultaneous sources and detectors constrained to a complementary subspace. Our approach leads to solutions of the same quality as obtained using all sources and detectors but at a greatly reduced computational cost, as the number of large-scale linear systems to be solved is significantly reduced.

show abstract

“…The compact support of the basis functions is an important advantage for nonlinear optimization, since not all parameters may need to be updated at each iteration (see [1]). The TREGS (pronounced tē reks) method [25] has proved to be fast and reliable at solving the nonlinear least squares problem for the parameter Downloaded 07/17/15 to 165.123.34.86. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php vector describing the absorption images, and we therefore use this algorithm for all our numerical results.…”

Section: Palsmentioning

confidence: 99%

“…Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php vector describing the absorption images, and we therefore use this algorithm for all our numerical results. The interested reader is directed to [25] for further details of the optimization algorithm. …”

Section: Palsmentioning

confidence: 99%

Nonlinear Parametric Inversion Using Interpolatory Model Reduction

Sturler¹,

Gugercin²,

Kilmer³

et al. 2015

SIAM J. Sci. Comput.

Self Cite

View full text Add to dashboard Cite

Nonlinear parametric inverse problems appear in several prominent applications; one such application is diffuse optical tomography (DOT) in medical image reconstruction. Such inverse problems present huge computational challenges, mostly due to the need for solving a sequence of large-scale discretized, parametrized partial differential equations in the forward model. In this paper, we show how interpolatory parametric model reduction can significantly reduce the cost of the inversion process in DOT by drastically reducing the computational cost of solving the forward problems. The key observation is that function evaluations for the underlying optimization problem may be viewed as transfer function evaluations along the imaginary axis; a similar observation holds for Jacobian evaluations as well. This motivates the use of system-theoretic model order reduction methods. We discuss the construction and use of interpolatory parametric reduced models as surrogates for the full forward model. Within the DOT setting, these surrogate models can approximate both the cost functional and the associated Jacobian with very little loss of accuracy while significantly reducing the cost of the overall inversion process. Four numerical examples illustrate the efficiency of the proposed approach. Introduction.Nonlinear inverse problems, as exemplified by medical image reconstruction or identification and localization of anomalous regions (e.g., tumors in the body [21], contaminant pools in the earth [38], or cracks in a material sample [55]), are commonly encountered yet remain very expensive to solve. In such inverse problems, one wishes to recover information identifying an unknown spatial distribution (the image) of some quantity of interest within a given medium that is not directly observable. For example, identifying anomalous regions of electrical conductivity in a sample of muscle tissue aids in identification and localization of potential tumor sites.The principal tool linking the images that are sought to correlated data that may be observed and measured is a mathematical model, the forward model. Within the context considered here, forward models are large-scale, discretized, two-or threedimensional, partial differential equations. These forward models constitute the functions to be evaluated for the underlying optimization problem that fits images of interest to observed data, so it is necessary to resolve these large-scale forward problems many times in order to to recover and reconstruct an image to some desired resolution. This constitutes the largest single computational impediment to effective, practical use of some imaging modalities, and low-quality image resolution is an all too common and regrettable outcome. Rapid advances in technology make it possible

show abstract

A Regularized Gauss–Newton Trust Region Approach to Imaging in Diffuse Optical Tomography

Cited by 17 publications

References 23 publications

Stable architectures for deep neural networks

Stable architectures for deep neural networks

Randomized Approach to Nonlinear Inversion Combining Random and Optimized Simultaneous Sources and Detectors

Nonlinear Parametric Inversion Using Interpolatory Model Reduction

Contact Info

Product

Resources

About