Performance of some multiple comparison tests under heteroscedasticity and dependency

An Updated Recommendation for Multiple Comparisons

“…The computing limitations at the time many of these PCPs were developed restricted the sample sizes tested to small numbers (e.g., as low as 6 in Games & Howell, 1976). Some recent evaluations of PCPs also maintained small sample sizes (as low as 4 in Demirhan et al, 2010). In part because the existing literature already discusses these small sample sizes, and in part because sampling error of group sizes so small can be detrimental to the precision of results, we chose to keep the minimum group size at 20.…”

“…The categories of low, medium, and high power are general assessments of power in relation to the other tests and are not meant to be precise descriptors. These summaries of performance were derived from the following sources: Boneau (1960); Carmer and Swanson (1973); Curran-Everett (2000); Demirhan, Dolgun, Demirhan, and Dolgun (2010); Duncan (1955); Dunn (1961); Dunnett (1980a, 1980b); Einot and Gabriel (1975); Gabriel (1978); Games (1971); Glass, Peckham, and Sanders (1972); Hsiung and Olejnik (1991); Keselman, Games, and Rogan (1979); Keselman and Rogan (1978); Kromrey and La Rocca (1995); Ozkaya and Ercan (2012); Petrinovich and Hardyck (1969); Ramsey (1978, 1981); Seaman, Levin, and Serlin (1991); Seco, de la Fuente, and Escudero (2001); Tamhane (1979); Waller and Duncan (1969). MRT = multiple range test; LSD = least significant difference; REGWF = Ryan-Einot-Gabriel-Welsch F test; REGWQ = Ryan-Einot-Gabriel-Welsch range test; HSD = honestly significant difference. a This column describes how power when assumptions are not met compares with power when assumptions are met. …”

“…However, power can be quantified in several different ways for PCPs. Specifically, one can examine average power, any-pair power, or all-pairs power (Demirhan, Dolgun, Demirhan, & Dolgun, 2010; Jaccard, Becker, & Wood, 1984). Average power is the proportion of true differences detected.…”

Optimization of Bayesian Multiple Comparison Tests dbayes and pbayes in R Code

“…HINKELMANN; KEMPTHORNE, 1987;TAMHANE, 1987;HSU, 1996). However, it is difficult to choose which MCP to use, since their performance varies in relation to the type I and power error rates (DEMIRHAN et al, 2010).…”

“…It is known that heteroscedasticity can affect the inference and may have a direct impact on the conclusions. The presence of variance heterogeneity may also have a serious effect on the validity of the F-test, especially when the sizes of these samples are unbalanced (CHEN;LEE, 2011;DEMIRHAN et al, 2010;KEYES;LEVY, 1997;LI, 2012;O'BRIEN, 1978;RAMSEY;RAMSEY;BARRERA, 2010;RAM-SEY;BARRERA;HACHIMINE-SEMPREBOM, 2011;SARMAH;GOGOI, 2015;SHINGALA;RAJYAGURU, 2015;TAMHANE, 1979). Furthermore, an alternative is the use of Bayesian procedures to make multiple comparisons FERREIRA, 2010;BERRY, 1988;BERRY;HOCHBERG, 1999;BRATCHER;HAMIL-TON, 2005;DUNCAN, 1965;GELMAN;HILL;YA-JIMA, 2012;GOPALAN;BERRY, 1998;SHAFFER, 1999;WALLER;DUNCAN, 1969).…”