Needles and straw in a haystack: Robust confidence for possibly sparse sequences

Abstract

In the general signal+noise (allowing non-normal, non-independent observations) model we construct an empirical Bayes posterior which we then use for uncertainty quantification for the unknown, possibly sparse, signal. We introduce a novel excessive bias restriction (EBR) condition, which gives rise to a new slicing of the entire space that is suitable for uncertainty quantification. Under EBR and some mild exchangeable exponential moment condition on the noise, we establish the local (oracle) optimality of the proposed confidence ball. Without EBR, we derive the full coverage for confidence balls of at least σn 1/4 -radius, implying the local optimality only for cases when the oracle rate is at least of the order σn 1/4 . In passing, we also get the local optimal results for estimation and posterior contraction problems. Adaptive minimax results (also for the estimation and posterior contraction problems) over various sparsity classes follow from our local results. MSC2010 subject classification: primary 62G15, secondary 62C12.The best studied problem in the sparsity context is that of estimating θ in the many normal means model, a variety of estimation methods and results are available in the literature: [27]. However, even an optimal estimator does not reveal how far it is from θ. It is of importance to quantify this uncertainty, which can be seen as the problem of constructing confidence sets for θ.Bayesian approach and accompanying posterior contraction problem. Many inference methods have Bayesian connections. For example, even some seemingly non Bayesian estimators can be obtained as certain quantities (like posterior mode for penalized minimum contrast estimators) of the (empirical Bayes) posterior distributions resulting from imposing some specific priors on the parameter; cf.[17] and [1]. Although the Bayesian methodology is used or can be related to in constructing many (frequentist) inference procedures, only recently the posterior distributions themselves have been studied in the sparsity context: [13], [27], [19], [12], [7], [25], [23].In this paper, for inference on θ we use an empirical Bayes approach. Since any Bayesian approach always delivers a posterior π(ϑ|X) (in the posteriors for θ, we will use the variable ϑ to distinguish it from the "true" θ), an accompanying problem of interest is the contraction of the resulting (empirical Bayes) posterior to the "true" θ from the frequentist perspective of the "true" measure P θ , the distribution of X from (1). The quality of posterior is characterized by the posterior contraction rate. We pursue a novel local approach by allowing the posterior contraction rate to be a local quantity, i.e., depending on the true θ, whereas global minimax rates are typically studied in the literature on Bayesian nonparametrics.A common Bayesian way to model sparsity structure is by the so called two-groups priors. Such a prior puts positive mass on vectors θ with some exact zero coordinates (zero group) and the remaining coordinates (signal group) are drawn from a cho...

Empirical Priors and Coverage of Posterior Credible Sets in a Sparse Normal Mean Model

“…As for the fractional power, the results in Belitser and Nurushev (2017) and Belitser and Ghosal (2019) suggest that taking α = 1 might be possible, perhaps with some adjustments elsewhere, but we believe there are reasons to retain this flexibility, especially in the context of uncertainty quantification. In particular, existing results on coverage probability of credible sets require a blow-up factor-e.g., the factor L in Equation (3) of van der Pas et al (2017b)-to inflate the credible set beyond the size determined by the posterior distribution itself.…”

“…That is, for any posterior with the optimal concentration rate, there must be true parameter values that the 100(1 − γ)% posterior credible sets cover with probability (much) less than 1 − γ. Consequently, there is interest in identifying these troublemaker parameter values. Recent efforts along these lines in the sparse normal mean model include Belitser (2017), Belitser and Nurushev (2017), van der Pas et al (2017b), Castillo and Szabó (2019), and Nurushev and Belitser (2019); for details beyond the normal mean model, see, e.g., Szabó et al (2015), Belitser and Ghosal (2019), and Martin and Tang (2019).…”

Spike and slab empirical Bayes sparse credible sets

“…To achieve good frequentist coverage one has to introduce certain extra assumptions on the parameter set ℓ 0 [s]. We consider the excessive-bias restriction investigated in the context of the sparse normal means model in [4,32], i.e. we say that θ 0 ∈ ℓ 0 [s] satisfies the excessive-bias restriction for constants A > 1 and C 2 , D 2 > 0, if there exists an integer s ≥ ℓ ≥ log 2 n, with i:|θ0,i|<A…”

“…Proof of Lemma 17. Recall the definition of the score function S(α) = n i=1 B(X i , α) with B(X i , α) as in (4). The map α → S(α) is strictly decreasing with probability one, as the function α → B(X i , α) is, as soon as B(X i ) = 0, which happens with probability one.…”

Learning from a lot: Empirical Bayes for high‐dimensional model‐based prediction

“…For example, Johnstone and Silverman (2004) provide asymptotic optimality results for a spike-and-slab-type prior. In addition, Belitser and Nurushev (2015) present theoretical evidence that, in a sparse spike-and-slab setting, EB allows the use of a Gaussian slab to obtain good contraction rates of the posteriors, which is a prerequisite for obtaining correct coverage of credibility intervals. Such a Gaussian slab prior is not recommended for the ordinary sparse Bayes setting because it shows suboptimal contraction rates as compared with more heavy-tailed slab distributions (Castillo & van der Vaart, 2012).…”