Fundamental Limits of Exact Support Recovery in High Dimensions

Abstract

We study the support recovery problem for a high-dimensional signal observed with additive noise. With suitable parametrization of the signal sparsity and magnitude of its non-zero components, we characterize a phase-transition phenomenon akin to the signal detection problem studied by Ingster in 1998. Specifically, if the signal magnitude is above the so-called strong classification boundary, we show that several classes of well-known procedures achieve asymptotically perfect support recovery as the dimension goes to infinity. This is so, for a very broad class of error distributions with light, rapidly varying tails which may have arbitrary dependence. Conversely, if the signal is below the boundary, then for a very broad class of error dependence structures, no thresholding estimators (including ones with data-dependent thresholds) can achieve perfect support recovery. The proofs of these results exploit a certain concentration of maxima phenomenon known as relative stability. We provide a complete characterization of the relative stability phenomenon for Gaussian triangular arrays in terms of their correlation structure. The proof uses classic Sudakov-Fernique and Slepian lemma arguments along with a curious application of Ramsey's coloring theorem.We note that our study of the strong classification boundary is in a finer, point-wise, rather than minimax, sense. We also establish results on the finite-sample Bayes optimality and sub-optimality of thresholding procedures. Consequently, we obtain a minimax-type characterization of the strong classification boundary for errors with log-concave densities.

Five Shades of Grey: Phase Transitions in High-dimensional Multiple Testing

“…In terms of the more stringent exact support recovery probability (2.5), several well-known FWER-controlling procedures -including Bonferroni's procedure -have been shown to be optimal in the additive error model under one-sided alternatives [14]. Optimality results were also obtained for a specific procedure under the expected Hamming loss (E[ S△S]) in [7], where the asymptotic analyses focused exclusively on the two-sided alternatives.…”

“…Therefore, a theory for the chi-square model (1.1) naturally lends itself to the study of two-sided alternatives in the Gaussian additive error model (1.3). In comparing such results with existing theory on one-sided alternatives [2,14], we will be able to quantify if, and how much of a price has to be paid for the additional uncertainty when we have no prior knowledge on the direction of the signals.…”

“…To the best of our knowledge, so far, the literature on multiple testing have been concerned with criteria roughly symmetric in terms of false discovery and false non-discovery control. For example, Arias-Castro and Chen [2] studied conditions under which fractions of false discovery and fractions of non-discovery can vanish; Gao and Stoev [14] investigated family-wise error control for both types of errors; meanwhile, Butucea et al [7] analyzed the problem under the Hamming loss, which penalizes false discoveries and missed signals equally.…”

“…Unfortunately, it was falsely claimed that the boundary characterized the phase transition of the exact support recovery problem, and the alleged proof was left as an "exercise to the reader". This exercise was completed in [14], where the correct boundary (3.2) was identified.…”

“…Remark 5.1. The exact support recovery problem under one-sided alternatives in the dependent Gaussian additive error model (1.3) was studied in [14]. The parametrization of sparsity was identical to (2.9), while the range of the non-zero (and perhaps unequal) mean shifts µ(i) was parametrized as 2r log p ≤ µ(i) ≤ 2r log p, for all i ∈ S p ,…”