Modernizing use of regression models in physics education research: A review of hierarchical linear modeling

Abstract

This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (also known as multilevel models) account for this hierarchical nested structure in the data. In this publication, we outline the theoretical differences between how single-level and multilevel models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field's understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multilevel datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. The Supplemental Material includes a sample dataset, R code to model the building and analysis presented in the paper, and an HTML output from the R code.

Equity in college physics student learning: A critical quantitative intersectionality investigation

“…Our models were all two‐level HLMs with student data in the first level and course data in the second level. Using HLMs allowed us to account for the nested nature of our dataset (Van Dusen & Nissen, in press). We explored whether including a third level for institution improved our models, but we found that it increased the total variance and did not substantively change the model coefficients.…”

Associations between learning assistants, passing introductory physics, and equity: A quantitative critical race theory investigation

“…AICc scores take into account the explanatory power of each variable without overly weighting model parsimony. While it is a common practices to use variance explained to select a final model (e.g., our own prior work [86,87]), using it to select models when investigating marginalized populations risks the models falling prey to a tyranny of the masses. Using predicted variance to select variables risks excluding marginalized students from groups with small representations in the data because the variance explained by a variable is proportional to its the sample size.…”

“…To examine whether the nested structure of our data necessitated the use of hierarchical models, we generated an unconditional model without predictor variables and calculated the intra-class correlation [88]. We found 8.6% of the variance at the course level; therefore, the best practice was to account for the hierarchical structure of the data in our model [87].…”

Educational debts incurred by racism and sexism in students’ beliefs about physics

“…Our models were 2-level hierarchical linear models with student data in the first level and course data in the second level. Using hierarchical linear models allowed us to account for the nested nature of our dataset [32]. We developed the models and pooled the results for the imputed datasets using the mitml [33] and lme4 [34] packages in R. We developed the models through a step-wise addition of variables.…”