Decentralized nonparametric multiple testing

Abstract

Consider a big data multiple testing task, where, due to storage and computational bottlenecks, one is given a very large collection of p-values by splitting into manageable chunks and distributing over thousands of computer nodes. This paper is concerned with the following question: How can we find the full data multiple testing solution by operating completely independently on individual machines in parallel, without any data exchange between nodes? This version of the problem tends naturally to arise in a wide range of data-intensive science and industry applications whose methodological solution has not appeared in the literature to date; therefore, we feel it is necessary to undertake such analysis. Based on the nonparametric functional statistical viewpoint of large-scale inference, started in Mukhopadhyay (2016), this paper furnishes a new computing model that brings unexpected simplicity to the design of the algorithm which might otherwise seem daunting using classical approach and notations.°°°°M achine 1 Machine 2 Machine K Figure 1: The data structure and setting of decentralized large-scale inference problem. Massive collection of p-values distributed across large number of computer nodes. be unrealistic due to huge volume (too expensive to store), computational bottleneck † , and possible privacy restrictions. Driven by practical need, the interest for designing Decentralized Large-Scale Inference Engine has enormously increased in the last few years, due to their ability to scale cost effectively as the data volume continued to increase by leveraging modern distributed storage and computing environments. There is, however, apparently no explicit algorithm currently available in the literature to tackle this innocent-looking problem of breaking the multiple testing computation into many pieces, each of which can be processed completely independently on individual machines in parallel.Remark 1. To get a glimpse of the challenge, consider a specific multiple testing method, say the Benjamini Hochberg's (BH) FDR controlling procedure, which starts by calculating the global -rank of each p-value:The computation of global-ranks, from the partitioned p-values, without any communications between the machines, is a highly non-trivial problem. Difficulty with similar caliber also arises in implementing local false discovery type algorithms. † BH (Benjamini and Hochberg, 1995) and HC (Donoho and Jin, 2004) procedures start by ordering the p-values from smallest to largest incurring at least O(N log N ) computational cost and other method like local fdr (Efron et al., 2001) is of even greater complexity O(N 2 ), thereby making legacy multiple testing algorithms infeasible for such massive scale inference problems.

Breiman's "Two Cultures'' Revisited and Reconciled

“…One can also compute pvalues using the χ 2 m null distribution of qDIV. For more details see Mukhopadhyay (2018) and Mukhopadhyay and Wang (2020a).…”

“…fidence from the ability of our theory to tackle statistical problems as diverse as time-series analysis (Mukhopadhyay and Parzen, 2018), copula modeling , empirical Bayes , graph-theory (Mukhopadhyay and Wang, 2020b), multiple testing (Mukhopadhyay, 2018(Mukhopadhyay, , 2016, high-dimensional data analysis (Mukhopadhyay and Wang, 2020a), large-scale distributed learning (Bruce et al, 2019), etc. This is an ongoing and growing movement with the mission to structure the field in an understandable and effective way.…”

Detecting new signals under background mismodeling

“…where f h (x) is the pdf of the unexpected signal and assume its distribution to be normal with center at 37 and width 1.8. Let f b (x) and f s (x) be defined as in (11) and (13), respectively, and let η 1 = 0.15 and η 2 = 0.1. We can start with a nonparametric signal detection stage by setting g(x) = g bs (x) in (30), with f s defined as in (13) dicate a significant departure from the expected backgroundonly model and a prominent peak is observed in correspondence of the signal of interest centered around 25.…”

“…At this stage, if f s was unknown, we could proceed with a semiparametric signal characterization as in Case IIb. Whereas assuming that the distribution of the signal of interest is known and given by (13), we fit (33), aiming to capture a significant deviation in correspondence of the second bump. This is precisely what we observe in the bottom right panel of Fig.…”

“…This approach allows the unification of many of the standard results of classical statistics by expressing them in terms of quantiles and comparison distributions and provides a simple and powerful framework for statistical learning and data analysis. The interested reader is directed to [9][10][11][12][13] and references therein, for recent advancements in mode detection, nonparametric time series, goodness-of-fit on prior distributions, and large-scale inference using an LP approach.…”