Compact representations of structured BFGS matrices

Abstract

The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory ("Argonne"). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan. http://energy.gov/downloads/doe-public-accessplan COAP manuscript No.

A structured L-BFGS method and its application to inverse problems

“…In the introduction we briefly described how algorithm TULIP differs from existing structured L-BFGS methods. We now comment on this in more detail for the recently proposed structured L-BFGS methods S-LBFGS-M and S-LBFGS-P from [BDLP21].…”

“…Let us first observe that in general the different methods use different operators B k . The operator B k+1 in algorithm TULIP satisfies the secant equation B k+1 s k = y k = ∇J (x k+1 ) − ∇J (x k ), while the operators in [BDLP21] satisfy, in our notation, the secant equations LBFGS-P). This implies that these operators are generally different from each other.…”

“…The idea to exploit the problem structure for constructing a better approximation of ∇ 2 J (x k ) and use it as a seed matrix in L-BFGS appears in various numerical studies [JBES04, Hel06, Mod09, K Ř13, ACG18, YGJ18, AM21, BDLP21], sometimes under the name preconditioned L-BFGS. Together, these works cover a wide range of real-life problems like molecular energy minimization [JBES04, K Ř13], independent component analysis [ACG18], medical image registration [Hel06, Mod09, AM21], logistic regression and optimal control [BDLP21]. The authors consistently report significant speed ups on these large-scale problems over all methods that are used for comparison, including classical L-BFGS, Gauss-Newton and truncated Newton.…”

“…Also, except for [AM21, BDLP21] the choice of τ k is not studied in the numerical experiments. For B (0) k = τ k I this choice can meaningfully influence the numerical performance of L-BFGS, cf for instance the classical works [Ore82, LN89, GL89], so we investigate it more thoroughly than [AM21,BDLP21]. In comparison to the method in our conference paper [AM21], TULIP is also significantly more effective and we test it on a substantially larger problem set.…”

“…In comparison to the method in our conference paper [AM21], TULIP is also significantly more effective and we test it on a substantially larger problem set. The structured methods from [BDLP21], furthermore, rely on a different formula for the quasi-Newton update and they use compact representations instead of the two-loop recursion that TULIP is built on; we discuss these aspects in detail in section 3.5. On image registration problems TULIP outperforms the methods from [BDLP21], cf section 5.…”