Detection of Correlations With Adaptive Sensing

Abstract

Abstract-The problem of detecting correlations from samples of a high-dimensional Gaussian vector has recently received a lot of attention. In most existing work, detection procedures are provided with a full sample. However, following common wisdom in experimental design, the experimenter may have the capacity to make targeted measurements in an on-line and adaptive manner. In this paper, we investigate such adaptive sensing procedures for detecting positive correlations. It is shown that, using the same number of measurements, adaptive procedures are able to detect significantly weaker correlations than their nonadaptive counterparts. We also establish minimax lower bounds that show the limitations of any procedure.

A Unified Framework for Correlation Mining in Ultra-High Dimension

“…Specifically the increment distribution ζ n,δ,ρ of the non-asymptotic compound Poisson distribution in Theorem III.11 depends on conditional probabilities in the random pseudo geometric graph as in (23). On the other hand, the increment distribution ζ n,δ of the limiting compound Poisson distribution in Theorem II.4 depends on probabilities in the random geometric graph as in (6), which is relatively simpler. For instance, when δ = 2, an analytical formula for ζ n,2 can be obtained (see Example V.10).…”

“…However, to get a reliable estimate of a general covariance matrix, the number of samples n must be at least Ω(p) as shown in [2,Section 5.4.3]. Even if the covariance matrix has a special structure like sparsity, covariance estimation requires the number of samples be of order Ω(ln p) The reader is referred to [3][4][5][6][7][8][9][10][11][12][13][14][15][16] and the references therein for related work in modern high dimensional covariance selection and estimation.…”