How do bootstrap and permutation tests work?

Abstract

Resampling methods are frequently used in practice to adjust critical values of nonparametric tests. In the present paper a comprehensive and unified approach for the conditional and unconditional analysis of linear resampling statistics is presented. Under fairly mild assumptions we prove tightness and an asymptotic series representation for their weak accumulation points. From this series it becomes clear which part of the resampling statistic is responsible for asymptotic normality. The results leads to a discussion of the asymptotic correctness of resampling methods as well as their applications in testing hypotheses. They are conditionally correct iff a central limit theorem holds for the original test statistic. We prove unconditional correctness iff the central limit theorem holds or when symmetric random variables are resampled by a scheme of asymptotically random signs. Special cases are the m(n) out of k(n) bootstrap, the weighted bootstrap, the wild bootstrap and all kinds of permutation statistics. The program is carried out for convergent partial sums of rowwise independent infinitesimal triangular arrays in detail. These results are used to compare power functions of conditional resampling tests and their unconditional counterparts. The proof uses the method of random scores for permutation type statistics. Introduction.Over the last 20 years nonparametric resampling procedures have become a powerful tool for setting confidence intervals and critical values of tests for composite hypotheses. In practice nonparametric two-step testing procedures benefit from the strong computational efforts of the new computer generation. Special resampling methods are Efron's bootstrap or Fisher's permutation tests. A justification is largely given by asymptotic considerations in order to compare the quality of resampling procedures with other tests. Throughout, we offer a unified asymptotic approach for the treatment of conditional resampling tests mainly given by partial sums of arbitrary arrays. It is based on a new setup for linear resampling statistics which is of independent interest. We will have a careful look at all kinds of permutation statistics, the m(n) out of k(n)-bootstrap and the weighted bootstrap (including the wild bootstrap); see Section 5.Conditional limit theorems can be motivated by the following nonparametric testing problem. Let H 0 denote a composite null hypothesis of distributions (or the

A comparative study of topology-based pathway enrichment analysis methods

“…The permutation version can benefit certain methods, such as DEGraph, but is not ideal for methods such as CAMERA that exploit difference in correlations. Theoretical justifications for how permutation works are available in [29] and [30].…”

Artificial shorelines lack natural structural complexity across scales

“…Despite the lower sample size at larger spatial scales (fine scale: 240 quadrats; medium scale: 24 laser scans; large scale: 21 UAV flights), each mean rugosity calculation was obtained from over 1 million observations, providing confidence that the calculated values accurately represented site-level surface rugosity. To compliment the non-parametric bootstrap, we used permutation tests to formally test for differences in the distributions of mean surface rugosity between artificial structures and natural rocky shores [ 49 ] across the 12 spatial scales using the ‘coin’ package in R [ 50 ].…”

Development and validation of the COVID-19 severity index (CSI): a prognostic tool for early respiratory decompensation

“…For significance testing of AU-ROCs, we applied Welch’s t-test to average differences between permutation tests of models’ performance metrics. 26, 27 To assess for generalizability, AU-ROCs on the external validation cohort are also presented.…”