Anderson acceleration for contractive and noncontractive operators

Abstract

A one-step analysis of Anderson acceleration with general algorithmic depths is presented. The resulting residual bounds within both contractive and noncontractive settings reveal the balance between the contributions from the higher and lower order terms, which are both dependent on the success of the optimization problem solved at each step of the algorithm. The new residual bounds show the additional terms introduced by the extrapolation produce terms that are of a higher order than was previously understood. In the contractive setting these bounds sharpen previous convergence and acceleration results. The bounds rely on sufficient linear independence of the differences between consecutive residuals, rather than assumptions on the boundedness of the optimization coefficients, allowing the introduction of a theoretically sound safeguarding strategy. Several numerical tests illustrate the analysis primarily in the noncontractive setting, and demonstrate the use of the method, the safeguarding strategy and theory-based guidance on dynamic selection of the algorithmic depth, on a p-Laplace equation, a nonlinear Helmholtz equation and the steady Navier–Stokes equations with high Reynolds number in three spatial dimensions.

One-step convergence of inexact Anderson acceleration for contractive and non-contractive mappings

“…Our analysis is primarily based on linear algebra and on properties of orthogonal and oblique projectors in particular, to gain insight into the effect of the leastsquares minimization in Anderson acceleration. Compared to the main conclusion in [17,Theorem 5.5], our result gives the same convergence factor for the linear term of the nonlinear residual, but we have a simpler upper bound for the higher-order terms that does not exhibit an explicit exponential growth with the acceleration depth and does not involve the squares of the most recent nonlinear residual norm if the minimization at the current step has the largest possible gain. More importantly, our analysis can be extended without difficulty to study the effects of an approximate evaluation of the fixed point mapping for the inexact method by exploring the impact of small perturbations in the relevant projectors, whereas it seems less clear how this idea could be realized based on the analysis in [17].…”

“…The rest of the paper is structured as follows: In Section 2, we consider the fixed point iteration x k+1 = g(x k ) and state assumptions on the mapping g; we also outline Anderson acceleration and then present a few preliminary results for the subsequent analysis. In Section 3, we give a one-step convergence analysis of (exact) Anderson acceleration based on projectors and angles between subspaces, showing a result similar to that in [17] with a new bound for the higher-order terms. In Section 4, we provide an analysis of inexact Anderson acceleration, where each update g(x k ) is allowed to be evaluated with an error proportional to the residual norm w k = g(x k ) − x k without obviously affecting the convergence rate of the algorithm.…”

“…The idea of this approach is to find a proper linear combination of successive previous iterates with coefficients obtained from a constrained minimization to obtain a new iterate that potentially yields a smaller nonlinear residual norm than the new iterate computed from the original fixed point iteration. Since the solution of systems of nonlinear equations by fixed point iterations is ubiquitous in science and engineering, Anderson acceleration has been widely used in a variety of applications; see, e.g., [13,17,28] and the references therein.…”

“…Such a favorable property makes it reasonable to study the convergence of Anderson acceleration within its own framework based on minimization. An early major convergence result was given in [27] for contractive mappings, and this was later improved in [17] for contractive and non-contractive mappings. However, it is shown [9] To save costs for iterative methods for solving large nonlinear systems, an economic strategy is to use inexact methods, which follow the process of their exact counterpart but evaluate each iterate only approximately.…”

“…Our work is motivated by a recent one-step convergence analysis of (exact) Anderson acceleration [17]. To understand the inexact method, we first develop a similar analysis of the exact variant.…”