Innovated higher criticism for detecting sparse signals in correlated noise

Abstract

The context of Donoho and Jin's [18] work was that where the noise is white, although a small number of investigations have been made of the case of correlated noise (Hall, Pittelkow and Ghosh [29], Hall and Jin [30], Delaigle and Hall [17]). However, that research has focused on the ability of standard HC, applied in the form that is appropriate for independent data, to accommodate the nonindependent case. In this paper we address the problem of how to modify HC by developing innovated higher criticism (iHC) and showing how to optimize performance for correlated noise.Curiously, it turns out that when using the iHC method tuned to give optimal performance, the case of independence is the most difficult of all, statistically speaking. To appreciate why this result is reasonable, note that if the noise is correlated then it does not vary so much from one location to a nearby location, and so is a little easier to identify. In an extreme case, if the noise is perfectly correlated at different locations then it is constant, and in this instance it can be easily removed.On the other hand, standard HC does not perform well in the case of correlated noise, because it utilizes only the marginal information in the data without much attention to the correlation structure. Innovated HC is designed to exploit the advantages offered by correlation and gives good performance across a wide range of settings.The concept of the "detection boundary" was introduced by Donoho and Jin [18] in the context of white noise. In this paper, we extend it to the correlated case. In brief, the detection boundary describes the relationship between signal sparsity and signal strength that characterizes the boundary between cases where the signal can be detected and cases where it cannot. In the setting of dependent data, this watershed depends on the correlation structure of the noise as well as on the sparsity and strength of the signal. When correlation decays at a polynomial rate we are able to characterize the detection boundary quite precisely. In particular, we show how to construct concise lower/upper bounds to the detection boundary, based on the diagonal components of the inverse of the correlation matrix, Σ n . A special case is where Σ n is Toeplitz; there the upper and the lower bounds to the detection boundary are asymptotically the same. In the Toeplitz case, the iHC is optimal for signal detection but standard HC is not.There is a particularly extensive literature on multiple hypothesis testing under conditions of dependence. It includes contributions to the control of family-wise error rate and false discovery rate, and work of Abramovich et al. [1], Benjamini and Hochberg [2], Benjamini and Yekutieli [3], Brown and Russel [7], Cai and Sun [9], Clarke and Hall [12], Cohen, Sackrowitz and Xu [13], Donoho and Jin [19], Dunnett and Tamhane [22], Efron [23], Finner and Roters [24], Genovese and Wasserman [25], Jin and Cai [40], Olejnik et al. [46], Rom [47], Sarkar and Chang [48] and Wu [54]. Work of Kuelbs and Vidyashankar [41] i...

Study design in high-dimensional classification analysis

“…Varying the structure and magnitude of the correlations based on available scientific knowledge is needed with our proposed approach. Further improvements in this direction may be possible by using the innovated HCT suggested by Hall and Jin (2010), or by developing sample size determination methods based on regularized regression-based approaches that do not require pre-filtering and hence do not rely on the marginal effects of the features. However, developing sample size calculations using regression-based procedures (e.g.…”

Optimal change point detection in Gaussian processes

“…, n − 1}, let ζ (d, t) := {(r, s) : r ∈ S n , j / ∈ S n , |r − s| = d} . According to Lemma A1 of [23], there is a C (f ) for which the entries of L −1 n admits the following inequality,…”

Detection of Correlations With Adaptive Sensing

“…This problem has received ample attention in the literature, see, for instance, [1], [7], [8], [16], [17], [22], [23] and references therein. The detection-of-means problem can be formulated as the multiple hypothesis testing problem…”