Fast Estimation of Approximate Matrix Ranks Using Spectral Densities

Abstract

In many machine learning and data related applications, it is required to have the knowledge of approximate ranks of large data matrices at hand. In this paper, we present two computationally inexpensive techniques to estimate the approximate ranks of such large matrices. These techniques exploit approximate spectral densities, popular in physics, which are probability density distributions that measure the likelihood of finding eigenvalues of the matrix at a given point on the real line. Integrating the spectral density over an interval gives the eigenvalue count of the matrix in that interval. Therefore the rank can be approximated by integrating the spectral density over a carefully selected interval. Two different approaches are discussed to estimate the approximate rank, one based on Chebyshev polynomials and the other based on the Lanczos algorithm. In order to obtain the appropriate interval, it is necessary to locate a gap between the eigenvalues that correspond to noise and the relevant eigenvalues that contribute to the matrix rank. A method for locating this gap and selecting the interval of integration is proposed based on the plot of the spectral density. Numerical experiments illustrate the performance of these techniques on matrices from typical applications.

Fast Estimation of $tr(f(A))$ via Stochastic Lanczos Quadrature

“…One of the methods proposed by Ubaru et. al [44] for estimating the numerical rank of large matrices is equivalent to the SLQ method discussed here. The function f for this numerical rank estimation problem turns out to be a step function with a value of one above an appropriately chosen threshold.…”

“…Also, the degree or the number of Lancozs steps required might be very high in practice. A workaround of this issue, proposed in [25] (also mentioned in [44]), is to first approximate the step function by a shifted and scaled hyperbolic tangent function of the formf (t) = 1 2 (1 + tanh(αt)), where α is an appropriately chosen constant, and then approximate the trace of this surrogate functionf (t).…”

“…In the following, we discuss some of the works that are closely related to SLQ, particularly those that invoke the stochastic trace estimator. The stochastic trace estimator has been employed for a number of applications in the literature, for example, for estimating the diagonal of a matrix [8], for counting eigenvalues inside an interval [16], for approximating the score function of Gaussian processes [41], and for estimating the numerical rank [43,44]. For the log-determinant computation, a few methods have been proposed, which also invoke the stochastic trace estimator.…”

Find the dimension that counts: Fast dimension estimation and Krylov PCA

“…Low rank approximation is a popular tool used in applications to reduce high dimensional data [16,10,17,30]. Determining the lower dimension (rank k) remains a principal problem in these applications, see [31,32] for discussions. In statistical signal and array processing, detecting the number of signals in the observations of an array of passive sensors is a fundamental problem [33,19,23], which can be posed as the above dimension estimation problem.…”

Spectrum-adapted Polynomial Approximation for Matrix Functions

“…Efficiently computing f (A)b, a function of a large, sparse Hermitian matrix times a vector, is an important component in numerous signal processing, machine learning, applied mathematics, and computer science tasks. Application examples include graph-based semi-supervised learning methods [2]- [4]; graph spectral filtering in graph signal processing [5]; convolutional neural networks / deep learning [6,7]; clustering [8,9]; approximating the spectral density of a large matrix [10]; estimating the numerical rank of a matrix [11,12]; approximating spectral sums such as the log-determinant of a matrix [13] or the trace of a matrix inverse for applications in physics, biology, information theory, and other disciplines [14]; solving semidefinite programs [15]; simulating random walks [16,Chapter 8]; and solving ordinary and partial differential equations [17]- [19].…”