We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin's causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.

We conducted a theory-based analysis of the underlying structure of the Tripod student perception survey instrument using the Measures of Effective Teaching (MET) database (N = 1,049 middle school math class sections; N = 25,423 students). Multilevel item factor analyses suggested that an alternative bifactor structure best fit the Tripod items, and preliminary evidence suggests that both the general responsivity and the classroom management-specific dimensions are positively associated with teacher value-added scores. In our discussion, we consider the distinct characterizing features of adolescents as raters of teaching, the implications for teacher professional learning opportunities, and key areas for future research.

No abstract

No abstract

Recently, there has been an increase in the number of cluster randomized trials (CRTs) to evaluate the impact of educational programs and interventions. These studies are often powered for the main effect of treatment to address the ''what works'' question. However, program effects may vary by individual characteristics or by context, making it important to also consider power to detect moderator effects. This article presents a framework for calculating statistical power for moderator effects at all levels for two-and three-level CRTs. Annotated R code is included to make the calculations accessible to researchers and increase the regularity in which a priori power analyses for moderator effects in CRTs are conducted.

No abstract

Maximum likelihood estimation of multilevel structural equation model (MLSEM) parameters is a preferred approach to probe theories involving latent variables in multilevel settings. Although maximum likelihood has many desirable properties, a major limitation is that it often fails to converge and can incur significant bias when implemented in studies with a small to moderate multilevel sample (e.g., fewer than 100 organizations with 10 or less individuals/organization). To address similar limitations in single-level SEM, literature has developed Croon’s bias-corrected factor score path analysis estimator that converges more regularly than maximum likelihood and delivers less biased parameter estimates with small to moderate sample sizes. We derive extensions to this framework for MLSEMs and probe the degree to which the estimator retains these advantages with small to moderate multilevel samples. The estimator emerges as a useful alternative or complement to maximum likelihood because it often outperforms maximum likelihood in small to moderate multilevel samples in terms of convergence, bias, error variance, and power. The proposed estimator is implemented as a function in R using lavaan and is illustrated using a multilevel mediation example.

Designs that facilitate inferences concerning both the total and indirect effects of a treatment potentially offer a more holistic description of interventions because they can complement “what works” questions with the comprehensive study of the causal connections implied by substantive theories. Mapping the sensitivity of designs to detect these effects is of critical importance because it directly governs the types of evidence researchers can bring to bear on theories of action under realistic sample sizes. In this study, we develop closed-form expressions to estimate the variance of and the power to detect causally defined indirect effects in two-level group-randomized studies examining individual-level mediators (i.e., 2-1-1 mediation). We formulate our approach within the purview of typical multilevel mediation models and anchor their interpretation in the potential outcomes framework. The results provide power analysis formulas that reduce calculations to simple functions of the primary path coefficients (e.g., treatment–mediator and mediator–outcome relationships) and common summary statistics (e.g., intraclass correlation coefficients). Probing these formulas suggests that group-randomized designs can be well powered to detect indirect effects when carefully planned. The power formulas are implemented in the PowerUp software (causalevaluation.org).