Asymptotic Permutation Tests in General Factorial Designs

Abstract

In general factorial designs where no homoscedasticity or a particular error distribution is assumed, the well-known Wald-type statistic is a simple asymptotically valid procedure. However, it is well known that it suffers from a poor finite sample approximation since the convergence to its χ 2 limit distribution is quite slow. This becomes even worse with an increasing number of factor levels. The aim of the paper is to improve the small sample behaviour of the Wald-type statistic, maintaining its applicability to general settings as crossed or hierarchically nested designs by applying a modified permutation approach. In particular, it is shown that this approach approximates the null distribution of the Wald-type statistic not only under the null hypothesis but also under the alternative yielding an asymptotically valid permutation test which is even finitely exact under exchangeability. Finally, its small sample behaviour is compared with competing procedures in an extensive simulation study.

GFD: An R Package for the Analysis of General Factorial Designs

“…Another possibility is to improve the small sample behavior of the WTS by applying a permutation procedure (Pauly et al 2015). To describe this procedure in detail, let…”

“…In order to test the null hypotheses formulated above in this general framework, we consider two generalizations of the two-sample Welch t statistic: The Wald-type statistic (WTS) as discussed, e.g., in Pauly et al (2015), and the ANOVA-type statistic (ATS) from Brunner et al (1997). The WTS is given by…”

“…It is well known that under rather weak assumptions the WTS has asymptotically a central χ 2 f distribution with f = rank(H) degrees of freedom under H µ 0 : Hµ = 0. However, the WTS requires large sample sizes to get a satisfactory approximation by using the quantiles of the limiting χ 2 distribution (Akritas, Arnold, and Brunner 1997;Vallejo et al 2010;Pauly et al 2015).…”

“…, A J . Pauly et al (2015) have shown that this algorithm yields a valid permutation approach and consistent level α test, i.e., the conditional distribution of the WTPS always approximates the null distribution of Q N . The test controls the preassigned level α under the null hypothesis and is even finitely exact if the pooled data is exchangeable under the hypothesis.…”

“…It is well known, however, that the Wald-type statistic requires large sample sizes to control the pre-assigned type-I error, see e.g., Vallejo, Fernández, and Livacic-Rojas (2010). Its small sample behavior may be improved by applying an adequate permutation procedure, see Pauly, Brunner, and Konietschke (2015) for the theoretical background. The only comparable test included in the R function oneway.test is the Welch (1951) test for heteroscedastic one-way layouts.…”