Nonlinearly preconditioned L‐BFGS as an acceleration mechanism for alternating least squares with application to tensor decomposition

Sterck, Hans De; Howse, Alexander J. M.

doi:10.1002/nla.2202

Cited by 18 publications

(28 citation statements)

References 58 publications

(201 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The convergence theory of Nesterov's accelerated gradient method for convex problems does not apply in our case due to the non-convex setting of the CP problem, and because we accelerate ALS steps instead of gradient steps. In fact, in the context of nonlinear convergence acceleration for ALS, few theoretical results on convergence are available [2,3,4]. We will, however, demonstrate numerically, for representative synthetic and real-world test problems, that our Nesterov-accelerated ALS methods are competitive with or outperform existing acceleration methods for ALS.…”

Section: 3mentioning

confidence: 89%

“…(1.4), by an ALS step. Replacing gradient directions by update directions provided by ALS is essentially also the approach taken in [2,3,4] to obtain nonlinear acceleration of ALS by NGMRES, NCG and LBFGS; in the case of Nesterov's method the procedure is extremely simple and easy to implement. However, applying this procedure directly fails for several reasons.…”

Section: 3mentioning

confidence: 99%

See 1 more Smart Citation

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

Mitchell

Sterck

2020

Numerical Linear Algebra App

Self Cite

View full text Add to dashboard Cite

We present Nesterov-type acceleration techniques for Alternating Least Squares (ALS) methods applied to canonical tensor decomposition. While Nesterov acceleration turns gradient descent into an optimal first-order method for convex problems by adding a momentum term with a specific weight sequence, a direct application of this method and weight sequence to ALS results in erratic convergence behaviour or divergence. This is so because the tensor decomposition problem is non-convex and ALS is accelerated instead of gradient descent. We investigate how line search or restart mechanisms can be used to obtain effective acceleration. We first consider a cubic line search (LS) strategy for determining the momentum weight, showing numerically that the combined Nesterov-ALS-LS approach is competitive with or superior to other recently developed nonlinear acceleration techniques for ALS, including acceleration by nonlinear conjugate gradients (NCG) and LBFGS. As an alternative, we consider various restarting techniques, some of which are inspired by previously proposed restarting mechanisms for Nesterov's accelerated gradient method. We study how two key parameters, the momentum weight and the restart condition, should be set. Our extensive empirical results show that the Nesterov-accelerated ALS methods with restart can be dramatically more efficient than the standalone ALS or Nesterov accelerated gradient method, when problems are ill-conditioned or accurate solutions are required. The resulting methods perform competitively with or superior to existing acceleration methods for ALS, and additionally enjoy the benefit of being much simpler and easier to implement. On a large and ill-conditioned 71×1000×900 tensor consisting of readings from chemical sensors used for tracking hazardous gases, the restarted Nesterov-ALS method outperforms any of the existing methods by a large factor.

show abstract

Section: 3mentioning

confidence: 89%

Section: 3mentioning

confidence: 99%

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

Mitchell

Sterck

2020

Numerical Linear Algebra App

Self Cite

View full text Add to dashboard Cite

show abstract

“…In (32), for each iteration step k, one is free to pick H 0 k . In the original implementation of the algorithm, in order to reduce the condition numbers of H k , the diagonal is scaled with the Cholesky factor δ k , [58],…”

Section: Preconditioningmentioning

confidence: 99%

“…In the limited-memory (L-BFGS) variant, [22,23], the approximation to H is constructed from a small number of vectors by a rank-one update formula, see Eqn. (32) below. The resulting algorithm is still considered the state-ofthe-art method when huge systems of equations with a very large number of unknowns need to get solved.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Mathematical analysis of a solution method for finite-strain holonomic plasticity of Cosserat materials

Blesgen

Amendola

2019

Meccanica

View full text Add to dashboard Cite

This article deals with the mathematical derivation and the validation over benchmark examples of a numerical method for the solution of a finite-strain holonomic (rate-independent) Cosserat plasticity problem for materials, possibly with microstructure. Two improvements are made in contrast to earlier approaches: First, the micro-rotations are parameterized with the help of an Euler-Rodrigues formula related to quaternions. Secondly, as main result, a novel two-pass preconditioning scheme for searching the energyminimizing solutions based on the limited memory Broyden-Fletcher-Goldstein-Shanno quasi-Newton method is proposed that consists of a predictor step and a corrector-iteration. After outlining the necessary adaptations to the model, numerical simulations compare the performance and efficiency of the new and the old algorithm. The proposed numerical model can be effectively employed for studying the mechanical response of complicated materials featuring large size effects.

show abstract

Blockwise acceleration of alternating least squares for canonical tensor decomposition

Evans

Yang

2023

Numerical Linear Algebra App

View full text Add to dashboard Cite

The canonical polyadic (CP) decomposition of tensors is one of the most important tensor decompositions. While the well‐known alternating least squares (ALS) algorithm is often considered the workhorse algorithm for computing the CP decomposition, it is known to suffer from slow convergence in many cases and various algorithms have been proposed to accelerate it. In this article, we propose a new accelerated ALS algorithm that accelerates ALS in a blockwise manner using a simple momentum‐based extrapolation technique and a random perturbation technique. Specifically, our algorithm updates one factor matrix (i.e., block) at a time, as in ALS, with each update consisting of a minimization step that directly reduces the reconstruction error, an extrapolation step that moves the factor matrix along the previous update direction, and a random perturbation step for breaking convergence bottlenecks. Our extrapolation strategy takes a simpler form than the state‐of‐the‐art extrapolation strategies and is easier to implement. Our algorithm has negligible computational overheads relative to ALS and is simple to apply. Empirically, our proposed algorithm shows strong performance as compared to the state‐of‐the‐art acceleration techniques on both simulated and real tensors.

show abstract

Nonlinearly preconditioned L‐BFGS as an acceleration mechanism for alternating least squares with application to tensor decomposition

Cited by 18 publications

References 58 publications

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

Nesterov acceleration of alternating least squares for canonical tensor decomposition: Momentum step size selection and restart mechanisms

Mathematical analysis of a solution method for finite-strain holonomic plasticity of Cosserat materials

Blockwise acceleration of alternating least squares for canonical tensor decomposition

Contact Info

Product

Resources

About