Descriptive analyses of educational phenomena are a vital component of educational research. Such analyses yield reliable results when using representative individual participant data (IPD) from educational large-scale assessments (ELSAs). The meta-analytic integration of these results offers unique and novel research opportunities to provide strong empirical evidence of the consistency and generalizability of educational phenomena. This paper offers methodological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical complexities of ELSAs (e.g., sampling weights, clustered and missing IPD, dependent effect sizes), first, for conducting descriptive analyses (Stage 1), and second, for integrating the results with meta-analytic and meta-regression models (Stage 2). The two-stage approach is illustrated with IPD on reading achievement from PISA in order to integrate standardized mean differences (e.g., gender differences), correlations (e.g., with students’ SES), and interactions between educational characteristics at the participant level (e.g., the interaction between gender and SES).
To assess the meaningfulness of an intervention or policy effect on students’ achievement, researchers may apply empirical benchmarks as standards for comparisons, involving normative expectations for students’ academic growth as well as performance gaps between weak and average schools or policy-relevant groups (e.g., male and female students, students from socioeconomically advantaged or disadvantaged families, students with or without a migration background). Previous research offered these empirical benchmarks by drawing on student samples from the United States. How well these results generalize to student populations in other countries is an open question. We therefore provide novel meta-analytic evidence on these empirical benchmarks for students attending elementary and secondary schools in Germany for a broad variety of achievement outcomes (e.g., mathematics, science, information and communication technology, first and second language skills). Drawing on the results obtained for large, representative probability samples, we observed variations in each kind of benchmark across countries as well as across domains and student subpopulations within Germany. This pattern of results underscores that the assessment of the very same intervention effect may depend on the target population and outcome of the intervention. We conclude by illustrating and discussing the strengths and limitations of empirical benchmarks for assessing the magnitude of intervention effects.
To plan cluster-randomized trials with sufficient statistical power to detect intervention effects on student achievement, researchers need multilevel design parameters, including between-classroom and between-school differences and the amounts of variance explained by covariates at the student, classroom, and school level. Previous research has mostly been conducted in the U.S., focused on two-level designs, and limited to main achievement domains (i.e., mathematics, science, reading). Using representative data of students attending grades 1 to 12 from three German longitudinal large-scale assessments (3,963 ≤ N ≤ 14,640), we used three- and two-level latent (covariate) models to provide design parameters and corresponding standard errors for a broad array of domain-specific (e.g., mathematics, science, verbal skills) and domain-general achievement outcomes (e.g., basic cognitive functions). Three covariate sets were applied comprising (a) pretest scores, (b) sociodemographic characteristics, and (c) their combination. Design parameters varied considerably as a function of the hierarchical level, achievement outcome, and grade level. Our findings demonstrate the need to strive for an optimal fit between design parameters and target research context. The application of design parameters in power analyses is illustrated.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.