Inexact Newton regularization combined with gradient methods in Banach spaces

Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.

“…This assumption is also needed for deterministic iterates, cf. [46,Proposition 6.17] or [35,Lemma 9]. Thus we make the following assumption.…”

Section: Regularising Propertymentioning

confidence: 99%

“…This has since been extended and refined in various aspects, e.g. regarding acceleration [17,45,49,51], nonlinear forward models [12,35], and Gauss-Newton methods [29].…”

Section: Introductionmentioning

confidence: 99%

On the regularizing property of stochastic gradient descent

Jin

2018

“…Newton-like iterative regularization schemes are well-established workhorses for stably solving nonlinear ill-posed problems like (1.1), see, e.g. [10,12,19,21,24]. To simplify the notation let us consider a generic setting where F : D(F) ⊂ V → W denotes the underlying nonlinear operator between Banach spaces.…”

Section: Introductionmentioning

confidence: 99%

Tangential cone condition and Lipschitz stability for the full waveform forward operator in the acoustic regime

Eller¹,

Rieder²

2021

Time-domain full waveform inversion (FWI) in the acoustic regime comprises a parameter identification problem for the acoustic wave equation: Pressure waves are initiated by sources, get scattered by the earth's inner structure, and their reflected parts are picked up by receivers located on the surface. From these reflected wave fields the two parameters, density and sound speed, have to be reconstructed. Mathematically, FWI reduces to the solution of a nonlinear and ill-posed operator equation involving the parameter-to-solution map of the wave equation. Newton-like iterative regularization schemes are well suited and well analyzed to tackle this inverse problem. Their convergence results are often based on an assumption about the nonlinear map known as tangential cone condition. In this paper we verify this assumption for a semi-discrete version of FWI where the searched-for parameters are restricted to a finite dimensional space. As a byproduct we establish that the semi-discrete seismic inverse problem is locally Lipschitz stable, in particular, it is conditionally well-posed.

“…Margotti et al [24] extended these results to strong convergence for a Kaczmarz variation of REGINN using the Landweber method as inner iteration. Recently, Margotti [23] proved strong convergence and stability results for a variant of REGINN in Banach space, which used a generic gradient-like method as inner iteration. As we know, although the number of outer iterations by using the REGINN method is less than the number of iterations by using the Landweber iteration method directly, it still takes a lot of time to compute the inner loop.…”

Section: Introductionmentioning

confidence: 99%

Inexact Newton regularization combined with two-point gradient methods for nonlinear ill-posed problems ^*

Fan¹,

Xu²

2021

In this paper, we propose an inexact Newton regularization combined with two-point gradient methods for nonlinear ill-posed problems. The basic idea of the proposed method is to linearize the equation around each outer iteration and subsequently apply a so-called two-point gradient method in the inner loop to accelerate the iterative process. Under suitable assumptions, we show that the iteration sequence generated by the proposed algorithm converges to a solution of the related problem in the noiseless situation. Furthermore, the stability and regularization properties of the proposed algorithm are analyzed in the noise-data case. Several numerical examples are provided to validate the theoretical results and to demonstrate the efficiency of the proposed method.