A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes.

Abstract

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods.

Confidence intervals and sample size calculations for the standardized mean difference effect size between two normal populations under heteroscedasticity

“…Unlike the two previous procedures, the standardizer depends not only on the population variances, but also on the group size allocation ratios. It is noted in Keselman et al (2008) that the particular formulation of Kulinskaya and Staudte (2007) raises a practical problem about its general use as an effect size measure. Specifically, the concern of Keselman et al is about its dependence upon sample sizes.…”

Robust tests for multivariate factorial designs under heteroscedasticity

“…Second, if the distribution from which the data are sampled is heavy-tailed, the adverse effects of nonnormality can probably be overcome by substituting robust measures of location (e.g., trimmed mean) and scale (e.g., Winsorized covariance matrices) for the usual mean and covariance matrices. According to Keselman, Algina, Lix, Wilcox, and Deering (2008), one argument for the trimmed mean is that it can have a substantial advantage in terms of accuracy of estimation when sampling from heavy-tailed symmetric distributions without altering the hypothesis tested, because it represents the center of the data. In particular, Wilcox (2005) has shown that with modest sample sizes, the 20% trimmed mean performs well in many situations because it is able to handle a high proportion of outliers.…”

Testing for normality in the multi-group problem: is this a good practice?

“…A number of choices are available: © C I C E d i z i o n i I n t e r n a z i o n a l i (45,46). Option two has been shown to be very effective in withstanding the effects of non-normality (and variance heterogeneity) when testing hypotheses regarding treatment group equality (6,7,8,11,12,13,46). Researchers however, must be comfortable in testing the equality of population trimmed means; many consider this a reasonable approach because the trimmed mean is most representative of the typical score when data are non-normal.…”

“…If the results of the preliminary test indicate that the empirical data in each treatment group conforms to a theoretical normal distribution, researchers can go on to test for mean equality with the t-or F-test (assuming that the other assumptions are examined and believed to be true as well). However, if the result of the test for normality indicates the empirical data are not normally distributed within each treatment group, researchers must take remedial action [e.g., See Keselman, Algina, Lix, et al (11,12); Wilcox & Keselman (13); McCullagh & Nelder (14)]. Researchers can attempt to assess whether the data from their experiments conform to the validity requirements associated with classical test statistics (e.g., normality); see for example, Muller and Fetterman (15).…”