Accuracy of p-values of approximate tests in testing for equality of means under unequal variances

Abstract

ABSTRACT. Seemingly, testing for fixed effects in linear models with variancecovariance components has been solved for decades. However, even in simple situations such as in fixed one-way model with heteroscedastic variances (a multiple means case of the Behrens-Fisher problem) the questions of statistical properties of various approximations of test statistics are still alive. Here we present a brief overview of several approaches suggested in the literature as well as those available in statistical software, accompanied by a simulation study in which the accuracy of p-values is studied. Our interest is limited here to the Welch's test, the Satterthwaite-Fai-Cornelius test, the Kenward-Roger test, the simple ANOVA F -test, and the parametric bootstrap test. We conclude that for small sample sizes, regardless the number of compared means and the heterogeneity of variance, the ANOVA F -test p-value performs the best. For higher sample sizes (at least 5 per group), the parametric bootstrap performs well, and the Kenward-Roger test also performs well.

Improvement of the accuracy in testing the effect in the cox proportional hazards model using higher order approximations

“…If the p-value is accurate, the curve for the empirical cumulative distribution function of the p-value nearly coincides with the identity line. The curve for a conservative p-value is above the identity line; the curve below the identity line indicates a liberal p-value [38].…”