Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions

Abstract

Low-rank matrix approximations, such as the truncated singular value decomposition and the rank-revealing QR decomposition, play a central role in data analysis and scientific computing. This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation. These techniques exploit modern computational architectures more fully than classical methods and open the possibility of dealing with truly massive data sets.This paper presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions. These methods use random sampling to identify a subspace that captures most of the action of a matrix. The input matrix is then compressed-either explicitly or implicitly-to this subspace, and the reduced matrix is manipulated deterministically to obtain the desired low-rank factorization. In many cases, this approach beats its classical competitors in terms of accuracy, speed, and robustness. These claims are supported by extensive numerical experiments and a detailed error analysis.The specific benefits of randomized techniques depend on the computational environment. Consider the model problem of finding the k dominant components of the singular value decomposition of an m × n matrix. (i) For a dense input matrix, randomized algorithms require O(mn log(k)) floating-point operations (flops) in contrast with O(mnk) for classical algorithms. (ii) For a sparse input matrix, the flop count matches classical Krylov subspace methods, but the randomized approach is more robust and can easily be reorganized to exploit multi-processor architectures. (iii) For a matrix that is too large to fit in fast memory, the randomized techniques require only a constant number of passes over the data, as opposed to O(k) passes for classical algorithms. In fact, it is sometimes possible to perform matrix approximation with a single pass over the data.

Accuracy of singular vectors obtained by projection-based SVD methods

“…to [1,7]), compute the SVD of the small m × n matrix A := U * A V = U V * and obtain an approximate tall-skinny SVD as A ≈ ( U U ) ( V V ) * , represented explicitly as a low-rank (rank min( m, n)) matrix. Some of the columns of U U and V V then approximate the exact left and right singular vectors of A.…”

“…Some of the state-of-the-art algorithms for an approximate SVD, such as [6,7], rely on one-sided projection methods, instead of two-sided projection as described so far. In this case one would approximate A ≈ ( U U ) V * , where U V * = U * A is the SVD of the m × n matrix, obtained by projecting A only from the left side by U (or from the right by V ; we discuss left projection for definiteness).…”

Approximately-isometric diffusion maps

“…For example, a dominant part of the dictionary construction process is the low-rank Nyström approximation provided by the constructed ONM (see Definition 4.1 and Section 4). An alternative approach for obtaining such an approximation is to use a randomized method such as the methods presented in [18]. Such an implementation will be explored in future works, and the impact of this change on the distance approximation error will be analyzed.…”

Changing computing paradigms towards power efficiency

“…To this end, we can borrow approaches developed in the literature of randomized linear algebra (e.g. [19] and references therein). The main idea is that matrix A is replaced byà = PAP T , where P is a suitably chosen fat sketching matrix P ∈ R k×n , where k n [20].…”