Type I error and test power of different tests for testing interaction effects in factorial experiments

Abstract

A simulation study was conducted to investigate the effect of non normality and unequal variances on Type I error rates and test power of the classical factorial anova F‐test and different alternatives, namely rank transformation procedure (FR), winsorized mean (FW), modified mean (FM) and permutation test (FP) for testing interaction effects. Simulation results showed that as long as no significant deviation from normality and homogeneity of the variances exists, generally all of the tests displayed similar results. However, if there is significant deviation from the assumptions, the other tests are observed to be affected at considerably high levels except FR and FP tests. As a result, when the assumptions of factorial anova F‐test are not met or, in the case those assumptions are not tested whether met, it can be concluded that using FR and FP tests is more suitable than the classical factorial anova F‐test.

Comments on the paper ‘Type I error and test power of different tests for testing interaction effects in factorial experiments’ by M. Mendes and S. Yigit (Statistica Neerlandica, 2013, pp. 1–26)

“…Brunner and Neumann () have given an analytic counterexample demonstrating that, under the parametric hypothesis of no interaction, the statistic F R ( AB ) can become degenerate, that is, F R → ∞ for increasing sample size. It will be demonstrated in our Remark 5 that this is also true for the examples considered in the simulation study reported in Tables 8–13 of the paper by Mendes and Yigit ().…”

“…The main problem with the rank transform statistic FR did not become apparent in the simulations provided by Mendes and Yigit () because of the quite small sample sizes. As mentioned in our Remark 2, the hypotheses tested by the parametric F ‐test and by a nonparametric counterpart using ranks are different in general.…”

“…In Section 1.5.3 of Brunner and Puri (), it has been discussed in detail when the heuristic rank transform method works and how it works. From this discussion, it is obvious that the application of the rank transform technique as presented in Section 2.2 in the paper of Mendes and Yigit () is incorrect because of two reasons: Rank statistics in factorial designs are heteroscedastic in general, even if homoscedastic errors are assumed for the original observations. Therefore, the statistic cannot be simply scaled by a common variance estimator.…”

“…The properties of the Welch–James procedure and the ATS have been examined in extensive simulation studies in the aforementioned papers, and they provide satisfactory approximations except in the case of unequal sample sizes and extremely skewed distributions. These two important procedures are missing in the paper of Mendes and Yigit (). Our fourth remark relates to the permutation test in Section 2.5. On p. 7, l. 1–3 prior to formula (7), it is stated that ‘This P‐value is called “exact P‐value,” because P‐value was calculated from all possible permutations…’.…”

“…To demonstrate the impact of unequal sample sizes in combination with unequal variances, we provide simulations of the 5% level for normal distributions , where and . The results for the small and equal sample sizes 2, 3, 4, 5, and 10 are displayed in Table 8 on p. 21 of Mendes and Yigit (). In the left part of the following table, the simulated levels in case of positive pairing n 11 = n 12 = n 21 = 10 and n 22 = 20, 30, 50, 100 are displayed, whereas the right part contains the simulations for the case of negative pairing n 11 = n 12 = n 21 = 20, 30, 50, 100 and n 22 = 10.…”