Flexibility and power in comparisons among means.

A comparison of two procedures for testing multiple contrasts.

“…The experimenter pays a price in power for the privilege of testing all possible contrasts at a designated EW level. Dunn (1961) and Davis (1969) have suggested an alternative to the strategy of selecting contrasts post hoc and testing by the Scheffe procedure. They note that an investigator who specifies even a large number of contrasts in advance of the data collection and uses the Bonferroni t test for controlling EW for that designated finite set, may well have more power than if he had used the Scheffe test.…”

Type IV errors and interactions.

“…In point of fact, abandoning the F test might be the optimal strategy for a researcher to follow if the plan is to examine only a small number of contrasts, since the F test could lead to a nonrejection of Ho while one or more comparisons could be identified as significant with the more powerful Bonferroni or Dunn method. In such cases, the relative efficiency of the Scheffe to the Dunn procedure (defined, perhaps, in terms of the ratio of the respective critical values) may be determined prior to data collection, in order to reach a rational decision concerning the approach to adopt (see Davis, 1969, or Kirk, 1968). 6 4 "Appropriate" is used advisedly here, in the sense that the SchefK procedure is the only procedure that corresponds exactly to the initial test of hypothesis.…”

What do you do when the control group doesn't fit into the factorial design?

“…single control group is likely to considerably increase the number of comparisons between means that an investigator will want to make and, thereby, the probability that at least one of the comparisons will involve a Type I error. A number of excellent discussions of methods for controlling error rate experimentwise and the protection levels afforded by them have appeared recently in the present journal (Davis, 1969;Keselman, 1974;Perlmutter & Myers, 1973) and elsewhere (Games, 1971;Kirk, 1968;Miller, 1966;Myers, 1972). Consequently, it will suffice to direct attention to the problem; to the solutions proposed by Dunn (1961), Dunnett (1955), and Scheffe (1959) or some combination of these tests, depending upon the nature of the comparisons; and to the aforementioned discussions.…”