Optimal Shrinkage of Singular Values

Abstract

We consider recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE -square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level σ, the optimal nonlinearity for MSE loss simply shrinks each data singular value y to y 2 − 4nσ 2 (or to 0 if y < 2 √ nσ). This optimal nonlinearity guarantees an asymptotic MSE of 2nrσ 2 , which compares favorably with optimally tuned hard thresholding and optimally tuned soft thresholding, providing guarantees of 3nrσ 2 and 6nrσ 2 , respectively. Our general method also allows one to evaluate optimal shrinkers numerically to arbitrary precision. As an example, we compute optimal shrinkers for the Schatten-p norm loss, for any p > 0.Keywords. Matrix denoising -singular value shrinkage -optimal shrinkage -spiked model -low-rank matrix estimation -nuclear norm loss -unique admissible -Schatten norm loss.

Optimal shrinkage of eigenvalues in the spiked covariance model

“…the rank of the principal subspace is growing rather than fixed. (In the sibling problem of matrix denoising, compare the “spiked” setup [32, 31, 53] with the “fixed fraction” setup of [67]. )…”

“…In the code supplement [41] we provide Matlab code to compute the optimal nonlinearity for each of the 26 loss families discussed. In the sibling problem of singular value shrinkage for matrix denoising, [53] demonstrates numerical evaluation of optimal shrinkers for the Schatten- p norm, where analytical derivation of optimal shrinkers appears to be impossible.…”

Integrative factorization of bidimensionally linked matrices

“…Their estimate is based on grid‐search on a candidate set of . Recently, Gavish and Donoho () proposed another estimator based on random matrix theory, which is defined as the median of the singular values of divided by the square root of the median of the Marcenko‐Pastur distribution. Our simulations, not shown here, revealed that both and well approximate the standard deviation of a true noise matrix when the data matrix consists of low rank signal, and we use as a default throughout this paper for its simplicity.…”

Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences

“…Instead of using the eigenvalues λ i of PQ −1 or its inverse, we use regularized or shrinked eigenvalues [35][36][37]. For example, in light of (8), we can use the following shrinked eigenvalues:…”