Students Are Rarely Independent: When, Why, and How to Use Random Effects in Discipline-Based Education Research

Abstract

Discipline-based education researchers have a natural laboratory—classrooms, programs, colleges, and universities. Studies that administer treatments to multiple sections, in multiple years, or at multiple institutions are particularly compelling for two reasons: first, the sample sizes increase, and second, the implementation of the treatments can be intentionally designed and carefully monitored, potentially negating the need for additional control variables. However, when studies are implemented in this way, the observations on students are not completely independent; rather, students are clustered in sections, terms, years, or other factors. Here, I demonstrate why this clustering can be problematic in regression analysis. Fortunately, nonindependence of sampling can often be accounted for with random effects in multilevel regression models. Using several examples, including an extended example with R code, this paper illustrates why and how to implement random effects in multilevel modeling. It also provides resources to promote implementation of analyses that control for the nonindependence inherent in many quasi-random sampling designs.

Using cueing from question pairs to engage students in reflective thinking: An exploratory study

“…The relevant variables are summarized in Table I. We have used the model selection methods described by Theobald [19] to inform our process, which recommend selecting the most parsimonious model among the best fitting models, using Akaike information criterion (AIC) to compare competing models. AIC is an estimator of the relative quality of the fit of model to data, and rewards simpler models by accounting for the number of variables in the model.…”

Beyond linear regression: A reference for analyzing common data types in discipline based education research

“…Insufficiently correcting nonindependence (i.e., violations of the assumption of independence of errors) can lead to spurious conclusions. This nonindependence can be accounted for with a statistical method called multilevel modeling, as detailed in Gelman and Hill [23] and applied to DBER in Theobald [30]. Adding a random effect term of section, year, or student (respectively, from the examples above) accounts for the nonindependent nature of the observations.…”

“…Nonindependence is a relatively common problem in education studies because of the frequent use of quasirandom study designs. Multilevel modeling can accommodate many types of the glms we discuss in this paper [23,30].…”

“…When employing backward selection, the researcher fits a complex model and singularly removes parameters in subsequent models [30,64]. Models are compared using AIC and the model with the lowest AIC is selected as best fit, with the important caveat that models within 2 AIC are considered to have equivalent fit [64].…”

“…In this example we have five possible predictors: gender, college GPA at the start of the course, class standing, gender×GPA, and class standing×GPA. As is standard practice with backward selection [30,64], we start by testing the inclusion of each interaction by comparing models, using likelihood ratio tests, with and without the interaction. Next, we test the main effects of gender, college GPA, and class standing.…”