A rank-adaptive robust integrator for dynamical low-rank approximation

Abstract

A rank-adaptive integrator for the dynamical low-rank approximation of matrix and tensor differential equations is presented. The fixed-rank integrator recently proposed by two of the authors is extended to allow for an adaptive choice of the rank, using subspaces that are generated by the integrator itself. The integrator first updates the evolving bases and then does a Galerkin step in the subspace generated by both the new and old bases, which is followed by rank truncation to a given tolerance. It is shown that the adaptive low-rank integrator retains the exactness, robustness and symmetry-preserving properties of the previously proposed fixed-rank integrator. Beyond that, up to the truncation tolerance, the rank-adaptive integrator preserves the norm when the differential equation does, it preserves the energy for Schrödinger equations and Hamiltonian systems, and it preserves the monotonic decrease of the functional in gradient flows. Numerical experiments illustrate the behaviour of the rank-adaptive integrator. Keywordsdynamical low-rank approximation • rank adaptivity • structurepreserving integrator • matrix and tensor differential equations Mathematics Subject Classification (2000) 65L05 • 65L20 • 65L70 • 15A69

Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations

“…Instead of evolving individual low-rank factors in time, these methods evolve products of low-rank factors, which yields remarkable stability and exactness properties [27], both in the matrix and the tensor settings [31,40,39,9]. In this work, we employ the "unconventional" basis update & Galerkin step integrator [7] as well as its rank-adaptive extension [5], see also [32,8]. The rank-adaptive unconventional integrator chooses the approximation ranks according to the continuous-time training dynamics and allows us to find highly-performing low-rank subnetworks directly during the training phase, while requiring reduced training cost and memory storage.…”

“…Its solution space contains all elements of the layer's weight matrix. Thus, rather than a discrete update of the weights, we interpret the training phase as a continuous evolution of the weights according to (5), as illustrated in Fig. 1(a-b).…”

“…That is, we wish to make sure that our update remains of rank r k and at the same time minimizes the residual, i.e. solves (5). The rank r attractor of the above system is optimal in the sense that it minimizes the gradient norm…”

“…Constraining these parameters to lie on a manifold defined by low-rank matrices is thus a quite natural approach. By interpreting the training problem as a continuous-time gradient flow, we propose a training algorithm based on the extension of recent Dynamical Low-Rank Approximation (DLRA) algorithms [6,7,5]. This approach allows us to use low-rank numerical integrators for matrix Ordinary Differential Equations (ODEs) to obtain modified forward and backward training phases that only use the small-rank factors in the low-rank representation of the parameter matrices.…”

Dynamical low-rank approximation for Burgers' equation with uncertainty

“…Furthermore, the unconventional integrator inherits the exactness and robustness properties of the classical matrix projector-splitting integrator, see [40]. Additionally, it allows for an efficient use of rank adaptivity [44].…”