The approximate randomization test as an alternative to the F test in analysis of variance

Abstract

In experimental psychology i t is usually difficult to show that populations sampled meet the requirements for the use o f t or F tests, or even that they are similar to populations sampled in Monte Carlo experiments designed to demonstrate the robustness of these parametric tests. Consequently, a test which makes weaker requirements without sacrificing power or versatility should be preferred. It is shown that this is true of modified approximate randomization tests, which, like the randomization tests on which they are based, use observed scores to set up a sampling distribution. The tests are versatile, since they can be used on factorial designs of any complexity and the results of Monte Carlo experiments indicate that their power approximates that of the F test when the assumptions underlying the latter are met and is significantly greater when the populations sampled are made up of two normal distributions with different means. It is concluded that, where adequate computing facilities are available, the approximate randomization test is preferable to an F test for analysis of variance designs.Over the last 40 years or so analysis of variance and related parametric statistical techniques have been widely used in experimental psychology. The use of such techniques in psychology has often been criticized (Siegel, 1956;Bradley, 1968), and partly as a result of such criticisms non-parametric techniques are now also widely employed. Some writers have considered replacing null hypothesis testing with estimation and Bayesian methods (Edwards et al., 1963), and Sidman (1960) recommended dispensing with statistics altogether. Nevertheless statistical analysis remains an important part of most papers in experimental psychology. The analysis nearly always involves the test of a null hypothesis, and the test used is usually a parametric test (Student's t or F ) in conjunction with analysis of variance. The reasons are obvious. The experimental designs required by analysis of variance are well suited to experimental psychology since the effects of any number of independent variables and their interactions can be examined; and they are satisfyingly elegant, yet relatively simple to understand and apply. I n these respects, especially in point of versatility, and the possibility of examining the effects of several independent variables in a single experiment, analysis of variance has no rivals. But, if critics such as Bradley (1968) are correct, analysis of variance, notwithstanding its admirable positive features, should be used on psychological data with much greater caution than is the case. psychology? The most obvious and common criticism is that the assumptions underlying the use of the F test with analysis of variance are not met, or not known to be met, in psychological experiments. One pair of assumptions is that the populations from which the samples are drawn are normally distributed and have equal variances. In the early days of analysis of variance in psychology it was customary to carry out tests of normality...

Basic-level kinds and object persistence

“…Many participants' ratings reflected strong agreement or disagreement with the questions posed, and few ratings fell in the middle range of the scale, resulting in a violation of normality in the data. Although ANOVA is robust to departures from normality, especially with a balanced design (Glass, Peckham, & Sanders, 1972), we also analyzed the data in all experiments using approximate randomization tests, which provide a nonparametric test of the main effects (Still & White, 1981). We obtained precisely the same pattern of significant results in all three experiments using these tests as we did using ANOVA.…”

An exact permutation method for testing any effect in balanced and unbalanced fixed effect ANOVA

“…Another approximate method is the permutation of residuals under the reduced model (Still and White, 1981;Freedman and Lane, 1983). In this method, the residuals are computed from a model that includes all parameters except the parameter(s) of interest.…”

Analysis of Variance for Gene Expression Microarray Data

“…We did not want to rely on the F-distribution to test the signi cance of these effects as we have not established normally distributed error. Instead, we employed a nonparametric version of the F-test to determine the signi cance of these interactions, following an example in Manly (1997, p. 128) motivated by Still and White (1981). We rst adjusted the data to remove the overall effects of the other factors.…”