On the Minimax Risk of Dictionary Learning

Abstract

We consider the problem of learning a dictionary matrix from a number of observed signals, which are assumed to be generated via a linear model with a common underlying dictionary. In particular, we derive lower bounds on the minimum achievable worst case mean squared error (MSE), regardless of computational complexity of the dictionary learning (DL) schemes. By casting DL as a classical (or frequentist) estimation problem, the lower bounds on the worst case MSE are derived by following an established information-theoretic approach to minimax estimation. The main conceptual contribution of this paper is the adaption of the information-theoretic approach to minimax estimation for the DL problem in order to derive lower bounds on the worst case MSE of any DL scheme. We derive three different lower bounds applying to different generative models for the observed signals. The first bound applies to a wide range of models, it only requires the existence of a covariance matrix of the (unknown) underlying coefficient vector. By specializing this bound to the case of sparse coefficient distributions, and assuming the true dictionary satisfies the restricted isometry property, we obtain a lower bound on the worst case MSE of DL schemes in terms of a signal to noise ratio (SNR). The third bound applies to a more restrictive subclass of coefficient distributions by requiring the non-zero coefficients to be Gaussian. While, compared with the previous two bounds, the applicability of this final bound is the most limited it is the tightest of the three bounds in the low SNR regime. A particular use of our lower bounds is the derivation of necessary conditions on the required number of observations (sample size) such that DL is feasible, i.e., accurate DL schemes might exist. By comparing these necessary conditions with sufficient conditions on the sample size such that a particular DL scheme is successful, we are able to characterize the regimes where those algorithms are optimal (or possibly not) in terms of required sample size.

Learning Fast Dictionaries for Sparse Representations Using Low-Rank Tensor Decompositions

“…Similarly, recent studies [15] on the minimax risk for the dictionary identifiability problem showed that the necessary number of samples for reliable reconstruction, up to a given mean squared error, of a KS dictionary within its local neighborhood scales with (m K k=1 n k m k ) compared to (m K k=1 n k m k ) for unstructured dictionaries of the same size [9].…”

Identifiability of Kronecker-Structured Dictionaries for Tensor Data

“…In the case of unstructured dictionaries, several works do provide analytical results for the dictionary identifiability problem [14]- [21]. These results, which differ from each other in terms of the distance metric used, cannot be trivially extended for the KS-DL problem.…”

“…In this work, we focus on the Frobenius norm as the distance metric. Gribonval et al [20] and Jung et al [21] also consider this metric, with the latter work providing minimax lower bounds for dictionary reconstruction error. In particular, Jung et al [21] show that the number of samples needed for reliable reconstruction (up to a prescribed mean squared error ε) of an m × p dictionary within its local neighborhood must be at least on the order of N = Ω(mp 2 ε −2 ).…”

Rank-One NMF-Based Initialization for NMF and Relative Error Bounds Under a Geometric Assumption

“…2) Second, here we focused on upper bounds on the relative error. To assess the tightness of these bounds, we hope to prove minimax lower bounds on the relative error similarly to Jung et al [41]. 3) Third, as mentioned in Section I-A1, our geometric assumption in (2) can be considered as a special case of the near-separability assumption for NMF [13].…”