Higher criticism for detecting sparse heterogeneous mixtures

Donoho, David L.; Jin, Jiashun

doi:10.1214/009053604000000265

Cited by 615 publications

(1,084 citation statements)

References 21 publications

Supporting

Mentioning

1,057

Contrasting

Order By: Relevance

“…when there is no sample Y and only Z is observed), the problem that we consider here reduces to the problem of signal detection with growing dimension d (Ingster 1997;Ingster & Suslina 2001a,b, 2002aJin 2003Jin , 2004Donoho & Jin 2004;Jager & Wellner 2007), and our classification boundary coincides with the detection boundary established by Ingster (1997). Sharp asymptotics in the detection problem were studied by Ingster (1997) (see also Ingster & Suslina 2002a, ch.…”

Section: (B) Summary Of the Main Resultsmentioning

confidence: 75%

“…Distinguishing between P η and P Y presents a difficulty only when the vector u is close to 0. A particular type of closeness for large d can be characterized by the sparsity assumption (Ingster & Suslina 2002a;Donoho & Jin 2004) that we shall adopt in this paper. Similar to Ingster & Suslina (2002a) and Donoho & Jin (2004), we introduce the following set of sparse vectors in R d characterized by a positive number a d and a sparsity index β ∈ (0, 1]:…”

Section: (A) Model and Problemmentioning

confidence: 99%

See 1 more Smart Citation

Classification of sparse high-dimensional vectors

Ingster

Pouet

Tsybakov

2009

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

We study the problem of classification of d-dimensional vectors into two classes (one of which is 'pure noise') based on a training sample of size m. The main specific feature is that the dimension d can be very large. We suppose that the difference between the distribution of the population and that of the noise is only in a shift, which is a sparse vector. For Gaussian noise, fixed sample size m, and dimension d that tends to infinity, we obtain the sharp classification boundary, i.e. the necessary and sufficient conditions for the possibility of successful classification. We propose classifiers attaining this boundary. We also give extensions of the result to the case where the sample size m depends on d and satisfies the condition (log m)/ log d → γ , 0 ≤ γ < 1, and to the case of non-Gaussian noise satisfying the Cramér condition.

show abstract

Section: (B) Summary Of the Main Resultsmentioning

confidence: 75%

Section: (A) Model and Problemmentioning

confidence: 99%

Classification of sparse high-dimensional vectors

Ingster

Pouet

Tsybakov

2009

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

show abstract

“…In the statistical literature there has been a great deal of recent work on signal detection, such as the normal mixture detection problem (Ingster 1997, 1998, Donoho & Jin 2004): given Z -scores Z i , i = 1, …, n , test…”

Section: Methodsmentioning

confidence: 99%

“…In most large-scale genomics studies the proportion of signal ε is small and the signal strength μ is not very large. This has been termed the “rare and weak” regime by Donoho & Jin (2004), and is of considerable interest. It has been shown that in this regime there exist tests for (3) that are asymptotically adaptively optimal: they do not require knowledge of the unknown parameters, yet still asymptotically perform as well as the likelihood ratio test (Ingster 2002 a , b , Donoho & Jin 2004).…”

Section: Methodsmentioning

confidence: 99%

“…This has been termed the “rare and weak” regime by Donoho & Jin (2004), and is of considerable interest. It has been shown that in this regime there exist tests for (3) that are asymptotically adaptively optimal: they do not require knowledge of the unknown parameters, yet still asymptotically perform as well as the likelihood ratio test (Ingster 2002 a , b , Donoho & Jin 2004). In particular, Donoho & Jin (2004) showed that this is true of the higher criticism statistic of Tukey:

H C = sup_{z} n^{1 / 2} \frac{∣ \hat{F} (z) - Φ (z) ∣}{{false[normalΦ false(z false) {1 - normalΦ false(z false)} false]}^{1 / 2}},

where F̂ ( z ) is the empirical distribution function of the Z i and Φ( z ) is the null distribution of the Z i .…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Sparse Simultaneous Signal Detection for Identifying Genetically Controlled Disease Genes

Zhao

Cai

Cappola

et al. 2017

Journal of the American Statistical Association

View full text Add to dashboard Cite

Genome-wide association studies (GWAS) and differential expression analyses have had limited success in finding genes that cause complex diseases such as heart failure (HF), a leading cause of death in the United States. This paper proposes a new statistical approach that integrates GWAS and expression quantitative trait loci (eQTL) data to identify important HF genes. For such genes, genetic variations that perturb its expression are also likely to influence disease risk. The proposed method thus tests for the presence of simultaneous signals: SNPs that are associated with the gene’s expression as well as with disease. An analytic expression for the p-value is obtained, and the method is shown to be asymptotically adaptively optimal under certain conditions. It also allows the GWAS and eQTL data to be collected from different groups of subjects, enabling investigators to integrate public resources with their own data. Simulation experiments show that it can be more powerful than standard approaches and also robust to linkage disequilibrium between variants. The method is applied to an extensive analysis of HF genomics and identifies several genes with biological evidence for being functionally relevant in the etiology of HF. It is implemented in the R package ssa.

show abstract

Simultaneous and selective inference: Current successes and future challenges

2010

View full text Add to dashboard Cite

The previous decade can be viewed as a second golden for era Multiple Comparisons research. I argue that much of the success stems from our being able to address real current needs. At the same time, this success generated a plethora of concepts for error rate and power, as well as multiplicity of methods for addressing them. These confuse the users of our methodology and pose a threat. To avoid the threat, it is our responsibility to match our theoretical goals to the goals of the users of statistics. Only then should we match the methods to the theoretical goals. Considerations related to such needs are discussed: simultaneous inference or selective inference, testing or estimation, decision making or scientific reporting. I then further argue that the vitality of our field in the future - as a research area - depends upon our ability to continue and address the real needs of statistical analyses in current problems. Two application areas offering new challenges have received less attention in our community to date are discussed. Safety analysis in clinical trials, where I offer an aggregated safety assessment methodology and functional Magnetic Resonance Imaging.

show abstract

Higher criticism for detecting sparse heterogeneous mixtures

Cited by 615 publications

References 21 publications

Classification of sparse high-dimensional vectors

Classification of sparse high-dimensional vectors

Sparse Simultaneous Signal Detection for Identifying Genetically Controlled Disease Genes

Simultaneous and selective inference: Current successes and future challenges

Contact Info

Product

Resources

About