Detecting Unit of Analysis Problems in Nested Designs: Statistical Power and Type I Error Rates of the F Test for Groups-within-Treatments Effects

Abstract

The nested analysis of variance (ANOVA) model is often recommended for analysis of educational data in which students receive treatments within classrooms. The Type I error rate and statistical power of the F test for groups-within-treatments effects associated with such nested ANOVA designs were evaluated in a Monte Carlo study. Data were generated for ANOVA models comprising two and three levels of the treatment variable; two, three, and five groups nested within each treatment; and 3, 10, and 30 observations within each group. The intraclass correlation among scores within groups was controlled at levels ranging from zero (independent observations) to .50, and the F test for groups-within-treatments effects was conducted at nominal alpha levels ranging from .05 to .30. The results indicated that the test for groups within treatments did not indicate sufficient power, even at the most liberal alpha level examined, to detect intraclass correlation in the sample data. The procedure was not effective in supporting decisions to use individual observations rather than group means as the unit of analysis.

Valid statistical approaches for clustered data: A Monte Carlo simulation study

“…Selection of two or more level-1 units per cell has prominent adverse effects on inferential statistics when analytic techniques fail to account for the nested data structure. Consistent with previous methodological work [e.g., 17–18, 26–28, 36–38], type 1 error rates were greater than the established α criterion of 0.05 in the fixed effects ANOVA; results which demonstrate that findings based on larger samples, different design characterizations, and simpler models (i.e., t- tests) extend to the types of parameters more commonly seen in preclinical studies. Notably, the profound negative bias in the standard error, which occurs even when the number of level-1 units per cell is small, likely promotes elevated type 1 error rates in the fixed effects ANOVA by decreasing within-group variance.…”

Exploring the Relationship Between Initial Mathematics Skill and a Kindergarten Mathematics Intervention

“…For the test of variances (only), we followed the logic of noninferiority trials and reversed the null and alternative hypotheses and the associated Type I and Type II error rates (Dasgupta, Lawson, & Wilson, 2010). For this reason, and because tests of variance structures have limited power (Kromrey & Dickinson, 1996), we set α = .20 as our Type I error rate and reported the more complex heteroscedastic model unless we were relatively certain the two variances were equivalent.…”

Testing the Efficacy of a Tier 2 Mathematics Intervention

“…Because we test for equivalence, or the noninferiority, of the simpler model when compared to the more complex model, we must reverse the null and alternative hypotheses and, hence, the α and β values that represent Type I and Type II error rates as we might in equivalence or noninferiority trials (e.g., Dasgupta, Lawson, & Wilson, 2010; Piaggio, Elbourne, Altman, Pocock, & Evans, 2006). For this reason, as well as the low statistical power to detect differences between variance structures (Kromrey & Dickenson, 1996), we compare models with likelihood ratio test using α = .20 as our criterion Type I error rate and, as a consequence, report the more complex model unless we are relatively certain the two are equivalent.…”