Minimax risk of matrix denoising by singular value thresholding

Abstract

An unknown m by n matrix X0 is to be estimated from noisy measurements Y = X0 + Z, where the noise matrix Z has i.i.d. Gaussian entries. A popular matrix denoising scheme solves the nuclear norm penalization problem minX Y − X 2 F /2 + λ X * , where X * denotes the nuclear norm (sum of singular values). This is the analog, for matrices, of ℓ1 penalization in the vector case. It has been empirically observed that if X0 has low rank, it may be recovered quite accurately from the noisy measurement Y .In a proportional growth framework where the rank rn, number of rows mn and number of columns n all tend to ∞ proportionally to each other (rn/mn → ρ, mn/n → β), we evaluate the asymptotic minimax MSE M(ρ, β) = limm n ,n→∞ inf λ sup rank(X)≤rn MSE(X0,X λ ). Our formulas involve incomplete moments of the quarter-and semicircle laws (β = 1, square case) and the Marčenko-Pastur law (β < 1, nonsquare case). For finite m and n, we show that MSE increases as the nonzero singular values of X0 grow larger. As a result, the finite-n worst-case MSE, a quantity which can be evaluated numerically, is achieved when the signal X0 is "infinitely strong."The nuclear norm penalization problem is solved by applying soft thresholding to the singular values of Y . We also derive the minimax threshold, namely the value λ * (ρ), which is the optimal place to threshold the singular values.All these results are obtained for general (nonsquare, nonsymmetric) real matrices. This is an electronic reprint of the original article published by the Institute of Mathematical Statistics in The Annals of Statistics, 2014, Vol. 42, No. 6, 2413-2440. This reprint differs from the original in pagination and typographic detail. 1 2 D. DONOHO AND M. GAVISH1. Introduction. Suppose we observe a single noisy matrix Y , generated by adding noise Z to an unknown matrix X 0 , so that Y = X 0 + Z, where Z is a noise matrix. We wish to recover the matrix X 0 with some bound on the mean squared error (MSE). This is hopeless when X 0 is a completely general matrix, and the noise Z is arbitrary; but when X 0 happens to be of relatively low rank, and the noise matrix is i.i.d. standard Gaussian, one can indeed guarantee quantitatively accurate recovery. This paper provides explicit formulas for the best possible guarantees obtainable by a popular, computationally practical procedure.Specifically, let Y , X 0 and Z be m-by-n real matrices (a set we denote by M m×n ), and suppose that Z has i.i.d. entries, Z i,j ∼ N (0, 1). Consider the following nuclear-norm penalization (NNP) problem:

Optimal shrinkage of eigenvalues in the spiked covariance model

“…the rank of the principal subspace is growing rather than fixed. (In the sibling problem of matrix denoising, compare the “spiked” setup [32, 31, 53] with the “fixed fraction” setup of [67]. )…”

Dynamic Mode Decomposition with Control

“…Equation (2.6) demonstrates how to reduce the dimension of the data matrix X by appropriately choosing a truncation value r of the singular values, thus eliminating the remainder (rem) terms and allowing for the psuedoinverse to be accomplished sinceΣ is square. Choosing the appropriate truncation value r has a rich scientific history; notably, the Eckart-Young theorem provides a rigorous and popular method for choosing r [11,34,15] [10,14].…”

Including inputs and control within equation-free architectures for complex systems

“…The theorem states that the best approximation of X with k modes can be found by retaining the k largest singular values and respective modes. Recent theoretical developments attempt to optimally identify r when X may have additive noise [82,83]. …”