Foundational Principles for Large-Scale Inference: Illustrations Through Correlation Mining

Abstract

When can reliable inference be drawn in fue “Big Data” context? This paper presents a framework for answering this fundamental question in the context of correlation mining, wifu implications for general large scale inference. In large scale data applications like genomics, connectomics, and eco-informatics fue dataset is often variable-rich but sample-starved: a regime where the number n of acquired samples (statistical replicates) is far fewer than fue number p of observed variables (genes, neurons, voxels, or chemical constituents). Much of recent work has focused on understanding the computational complexity of proposed methods for “Big Data”. Sample complexity however has received relatively less attention, especially in the setting when the sample size n is fixed, and the dimension p grows without bound. To address fuis gap, we develop a unified statistical framework that explicitly quantifies the sample complexity of various inferential tasks. Sampling regimes can be divided into several categories: 1) the classical asymptotic regime where fue variable dimension is fixed and fue sample size goes to infinity; 2) the mixed asymptotic regime where both variable dimension and sample size go to infinity at comparable rates; 3) the purely high dimensional asymptotic regime where the variable dimension goes to infinity and the sample size is fixed. Each regime has its niche but only the latter regime applies to exa cale data dimension. We illustrate this high dimensional framework for the problem of correlation mining, where it is the matrix of pairwise and partial correlations among the variables fua t are of interest. Correlation mining arises in numerous applications and subsumes the regression context as a special case. we demonstrate various regimes of correlation mining based on the unifying perspective of high dimensional learning rates and sample complexity for different structured covariance models and different inference tasks.

Spatial Dynamic Functional Connectivity Analysis Identifies Distinctive Biomarkers in Schizophrenia

“…Followed by the estimation of time courses, a sliding window is applied on the time courses that divides it into consecutive windows and an analysis on the time points within each window is performed (Allen et al, 2014). The analysis of dFNC patterns depends on the length of the window, where the use of a longer window length increases the risk of averaging the temporal fluctuations of interest resulting in false negatives (Preti et al, 2017), and the use of a shorter window length has too few samples for a reliable computation of correlation (Hero and Rajaratnam, 2016), resulting in the temporal variations to capture spurious fluctuations and increasing the risk of false positives (Sakoğlu et al, 2010; Hutchison et al, 2013; Leonardi and Van De Ville, 2015). Previous studies have shown that a window length between 30 and 60 s successfully estimates temporal fluctuations in resting-state functional magnetic resonance imaging (fMRI) data (Preti et al, 2017), and for most cases higher window lengths do not alter the results significantly (Keilholz et al, 2013; Li et al, 2014; Liégeois et al, 2016).…”

“…Hence, spatio-temporal dFNC analysis relaxes the assumption of stationarity in both the spatial and temporal domain, and provides a more general framework for capturing time-varying FNC patterns (Ma et al, 2014; Kottaram et al, 2018; Kunert-Graf et al, 2018). The availability of higher number of samples in the spatial domain also guarantees reliable estimation of functional correlations (Hero and Rajaratnam, 2016), thus providing a promising direction for the use of spatial domain for dFNC analysis. However, the methods used to extract time-varying spatio-temporal patterns face few challenges.…”

Quickest Detection for Changes in Maximal kNN Coherence of Random Matrices

“…This is equivalent to the maximal kNN distance between columns, as measured by correlation distance. The theory from [17] helps us establish that the proposed summary statistic has a well defined exponential limiting distribution as p → ∞ for fixed n, the so-called "purely high dimensional regime" [1]. This summary statistic is related to the empirical distribution of the vertex degree of the correlation graph associated with the thresholded sample correlation matrix.…”

Non-parametric quickest change detection for large scale random matrices

“…The solution is optimal in the following sense. The theory from [6] establishes that a certain summary statistic, denoted by V (X), derived from an n × p random matrix X has a limiting distribution as p → ∞ for fixed n, the so-called "purely high dimensional regime" [16]. This summary statistic is related to the empirical distribution of the vertex degree of the correlation graph associated with the thresholded sample correlation matrix.…”