Unified frameworks for high order Newton-Schulz and Richardson iterations: a computationally efficient toolkit for convergence rate improvement

Abstract

Convergence rate and robustness improvement together with reduction of computational complexity are required for solving the system of linear equations Aθ * = b in many applications such as system identification, signal and image processing, network analysis, machine learning and many others. Two unified frameworks (1) for convergence rate improvement of high order Newton-Schulz matrix inversion algorithms and (2) for combination of Richardson and iterative matrix inversion algorithms with improved convergence rate for estimation of θ * are proposed. Recursive and computationally efficient version of new algorithms is developed for implementation on parallel computational units. In addition to unified description of the algorithms the frameworks include explicit transient models of estimation errors and convergence analysis. Simulation results confirm significant performance improvement of proposed algorithms in comparison with existing methods. Keywords Richardson iteration • Neumann series • High order Newton-Schulz algorithm • Least squares estimation • Harmonic regressor • Strictly Diagonally Dominant Matrix • Symmetric positive definite matrix • Ill-conditioned matrix • Polynomial preconditioning • Matrix power series factorization • Computationally efficient matrix inversion algorithm • Simultaneous calculations

Convergence Rate Improvement of Richardson and Newton-Schulz Iterations

“…where X, Y, Z are matrices of corresponding dimensions, I is the identity matrix, h = 2, 3, 4, ... . Realization of the algorithm (13) requires h mmm (matrix-by-matrix multiplications) per iteration loop according to Horner's scheme, see for example [21], [40].…”

“…The unified framework for convergence rate improvement of high order Newton-Schulz matrix inversion algorithms was proposed in [40] . The following new composite power series expansion for Newton-Schulz iteration with different expansion rates for further convergence rate improvement extends this framework as follows :…”

“…The error model (31) with composite power series expansion is the same as the error model ( 22) in [40] for a single power series. The advantages of composite expansion are discussed below.…”

“…The challenges associated with computational efficiency are addressed already on the design level in this case, providing new opportunities for high performance parallel processing. This paper introduces new composite power series expansion with optionally chosen rates and high degree of parallelism for further convergence rate improvement in the unified framework described in [40]. New expansion applied to Richardson iteration resulted in significant improvement of the convergence rate.…”

“…New expansion applied to Richardson iteration resulted in significant improvement of the convergence rate. Simulation results are presented for quantification of the improvements of new algorithms compared to recent algorithms described in [40]. Moreover, explicit transient models are derived for all new algorithms described in this paper.…”