We consider recovery of low-rank matrices from noisy data by shrinkage of singular values, in which a single, univariate nonlinearity is applied to each of the empirical singular values. We adopt an asymptotic framework, in which the matrix size is much larger than the rank of the signal matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. For a variety of loss functions, including Mean Square Error (MSE -square Frobenius norm), the nuclear norm loss and the operator norm loss, we show that in this framework there is a well-defined asymptotic loss that we evaluate precisely in each case. In fact, each of the loss functions we study admits a unique admissible shrinkage nonlinearity dominating all other nonlinearities. We provide a general method for evaluating these optimal nonlinearities, and demonstrate our framework by working out simple, explicit formulas for the optimal nonlinearities in the Frobenius, nuclear and operator norm cases. For example, for a square low-rank n-by-n matrix observed in white noise with level σ, the optimal nonlinearity for MSE loss simply shrinks each data singular value y to y 2 − 4nσ 2 (or to 0 if y < 2 √ nσ). This optimal nonlinearity guarantees an asymptotic MSE of 2nrσ 2 , which compares favorably with optimally tuned hard thresholding and optimally tuned soft thresholding, providing guarantees of 3nrσ 2 and 6nrσ 2 , respectively. Our general method also allows one to evaluate optimal shrinkers numerically to arbitrary precision. As an example, we compute optimal shrinkers for the Schatten-p norm loss, for any p > 0.Keywords. Matrix denoising -singular value shrinkage -optimal shrinkage -spiked model -low-rank matrix estimation -nuclear norm loss -unique admissible -Schatten norm loss.

We show that in a common high-dimensional covariance model, the choice of loss function has a profound effect on optimal estimation. In an asymptotic framework based on the Spiked Covariance model and use of orthogonally invariant estimators, we show that optimal estimation of the population covariance matrix boils down to design of an optimal shrinker η that acts elementwise on the sample eigenvalues. Indeed, to each loss function there corresponds a unique admissible eigenvalue shrinker η* dominating all other shrinkers. The shape of the optimal shrinker is determined by the choice of loss function and, crucially, by inconsistency of both eigenvalues and eigenvectors of the sample covariance matrix. Details of these phenomena and closed form formulas for the optimal eigenvalue shrinkers are worked out for a menagerie of 26 loss functions for covariance estimation found in the literature, including the Stein, Entropy, Divergence, Fréchet, Bhattacharya/Matusita, Frobenius Norm, Operator Norm, Nuclear Norm and Condition Number losses.

In compressed sensing, one takes n < N samples of an N-dimensional vector x 0 using an n × N matrix A, obtaining undersampled measurements y = Ax 0 . For random matrices with independent standard Gaussian entries, it is known that, when x 0 is k-sparse, there is a precisely determined phase transition: for a certain region in the (k/n,n/N)-phase diagram, convex optimization min || x || 1 subject to y = Ax, x ∈ X N typically finds the sparsest solution, whereas outside that region, it typically fails. It has been shown empirically that the same property-with the same phase transition location-holds for a wide range of non-Gaussian random matrix ensembles. We report extensive experiments showing that the Gaussian phase transition also describes numerous deterministic matrices, including Spikes and Sines, Spikes and Noiselets, Paley Frames, Delsarte-Goethals Frames, Chirp Sensing Matrices, and Grassmannian Frames. Namely, for each of these deterministic matrices in turn, for a typical k-sparse object, we observe that convex optimization is successful over a region of the phase diagram that coincides with the region known for Gaussian random matrices. Our experiments considered coefficients constrained to X N for four different sets X ∈ {[0, 1], R + , R, C}, and the results establish our finding for each of the four associated phase transitions.sparse recovery | universality in random matrix theory equiangular tight frames | restricted isometry property | coherence C ompressed sensing aims to recover a sparse vector x 0 ∈ X N from indirect measurements y = Ax 0 ∈ X n with n < N, and therefore, the system of equations y = Ax 0 is underdetermined. Nevertheless, it has been shown that, under conditions on the sparsity of x 0 , by using a random measurement matrix A with Gaussian i.i.d entries and a nonlinear reconstruction technique based on convex optimization, one can, with high probability, exactly recover x 0 (1, 2). The cleanest expression of this phenomenon is visible in the large n; N asymptotic regime. We suppose that the object x 0 is k-sparse-has, at most, k nonzero entries-and consider the situation where k ∼ ρn and n ∼ δN. Fig. 1A depicts the phase diagram ðρ; δ; Þ ∈ ð0; 1Þ 2 and a curve ρ*ðδÞ separating a success phase from a failure phase. Namely, if ρ < ρ*ðδÞ, then with overwhelming probability for large N, convex optimization will recover x 0 exactly; however, if ρ > ρ*ðδÞ, then with overwhelming probability for large N convex optimization will fail. [Indeed, Fig. 1 depicts four curves ρ*ðδjXÞ of this kind for X ∈ f½0; 1; R + ; R; Cg-one for each of the different types of assumptions that we can make about the entries of x 0 ∈ X N (details below).]How special are Gaussian matrices to the above results? It was shown, first empirically in ref. 3 and recently, theoretically in ref. 4, that a wide range of random matrix ensembles exhibits precisely the same behavior, by which we mean the same phenomenon of separation into success and failure phases with the same phase boundary. Such universality, if exhib...

An unknown m by n matrix X0 is to be estimated from noisy measurements Y = X0 + Z, where the noise matrix Z has i.i.d. Gaussian entries. A popular matrix denoising scheme solves the nuclear norm penalization problem minX Y − X 2 F /2 + λ X * , where X * denotes the nuclear norm (sum of singular values). This is the analog, for matrices, of ℓ1 penalization in the vector case. It has been empirically observed that if X0 has low rank, it may be recovered quite accurately from the noisy measurement Y .In a proportional growth framework where the rank rn, number of rows mn and number of columns n all tend to ∞ proportionally to each other (rn/mn → ρ, mn/n → β), we evaluate the asymptotic minimax MSE M(ρ, β) = limm n ,n→∞ inf λ sup rank(X)≤rn MSE(X0,X λ ). Our formulas involve incomplete moments of the quarter-and semicircle laws (β = 1, square case) and the Marčenko-Pastur law (β < 1, nonsquare case). For finite m and n, we show that MSE increases as the nonzero singular values of X0 grow larger. As a result, the finite-n worst-case MSE, a quantity which can be evaluated numerically, is achieved when the signal X0 is "infinitely strong."The nuclear norm penalization problem is solved by applying soft thresholding to the singular values of Y . We also derive the minimax threshold, namely the value λ * (ρ), which is the optimal place to threshold the singular values.All these results are obtained for general (nonsquare, nonsymmetric) real matrices. This is an electronic reprint of the original article published by the Institute of Mathematical Statistics in The Annals of Statistics, 2014, Vol. 42, No. 6, 2413-2440. This reprint differs from the original in pagination and typographic detail. 1 2 D. DONOHO AND M. GAVISH1. Introduction. Suppose we observe a single noisy matrix Y , generated by adding noise Z to an unknown matrix X 0 , so that Y = X 0 + Z, where Z is a noise matrix. We wish to recover the matrix X 0 with some bound on the mean squared error (MSE). This is hopeless when X 0 is a completely general matrix, and the noise Z is arbitrary; but when X 0 happens to be of relatively low rank, and the noise matrix is i.i.d. standard Gaussian, one can indeed guarantee quantitatively accurate recovery. This paper provides explicit formulas for the best possible guarantees obtainable by a popular, computationally practical procedure.Specifically, let Y , X 0 and Z be m-by-n real matrices (a set we denote by M m×n ), and suppose that Z has i.i.d. entries, Z i,j ∼ N (0, 1). Consider the following nuclear-norm penalization (NNP) problem:

We consider recovery of low-rank matrices from noisy data by hard thresholding of singular values, in which empirical singular values below a threshold λ are set to 0. We study the asymptotic mean squared error (AMSE) in a framework, where the matrix size is large compared with the rank of the matrix to be recovered, and the signal-to-noise ratio of the low-rank piece stays constant. The AMSE-optimal choice of hard threshold, in the case of n-by-n matrix in white noise of level σ , is simply (4/ √ 3) √ nσ ≈ 2.309 √ nσ when σ is known, or simply 2.858 · y med when σ is unknown, where y med is the median empirical singular value. For nonsquare, m by n matrices with m = n the thresholding coefficients 4/ √ 3 and 2.858 are replaced with different provided constants that depend on m/n. Asymptotically, this thresholding rule adapts to unknown rank and unknown noise level in an optimal manner: it is always better than hard thresholding at any other value, and is always better than ideal truncated singular value decomposition (TSVD), which truncates at the true rank of the low-rank matrix we are trying to recover. Hard thresholding at the recommended value to recover an n-by-n matrix of rank r guarantees an AMSE at most 3 nrσ 2 . In comparison, the guarantees provided by TSVD, optimally tuned singular value soft thresholding and the best guarantee achievable by any shrinkage of the data singular values are 5 nrσ 2 , 6 nrσ 2 , and 2 nrσ 2 , respectively. The recommended value for hard threshold also offers, among hard thresholds, the best possible AMSE guarantees for recovering matrices with bounded nuclear norm. Empirical evidence suggests that performance improvement over TSVD and other popular shrinkage rules can be substantial, for different noise distributions, even in relatively small n.Index Terms-Singular values shrinkage, optimal threshold, low-rank matrix denoising, unique admissible, scree plot elbow truncation, quarter circle law, bulk edge. 0018-9448