Addressing diversity and inclusion through group comparisons: a primer on measurement invariance testing

Abstract

<p>As the field of chemistry education moves toward greater inclusion and increased participation by underrepresented minorities, standards for investigating the differential impacts and outcomes of learning environments have to be considered. While quantitative methods may not be capable of generating the in-depth nuances of qualitative methods, they can provide meaningful insights when applied at the group level. Thus, when we conduct quantitative studies in which we aim to learn about the similarities or differences of groups within the same learning environment, we must raise our standards of measurement and safeguard against threats to the validity of inferences that might favor one group over another. One way to provide evidence that group comparisons are supported in a quantitative study is by conducting measurement invariance testing. In this manuscript, we explain the basic concepts of measurement invariance testing within a confirmatory factor analysis framework with examples and a step-by-step tutorial. Each of these steps is an opportunity to safeguard against interpretation of group differences that may be artifacts of the assessment instrument functioning rather than true differences between groups. Reflecting on and safeguarding against threats to the validity of the inferences we can draw from group comparisons will aid in providing more accurate information that can be used to transform our chemistry classrooms into more socially inclusive environments. To catalyze this effort, we provide code in the ESI for two different software packages (R and Mplus) so that interested readers can learn to use these methods with the simulated data provided and then apply the methods to their own data. Finally, we present implications and a summary table for researchers, practitioners, journal editors, and reviewers as a reference when conducting, reading, or reviewing quantitative studies in which group comparisons are performed.</p>

Is This Science? Students’ Experiences of Failure Make a Research-Based Course Feel Authentic

“…The largest effect size between inquiry and CURE students was seen for the Iteration scale, though there was also a medium effect size for the Relevant Discovery scale. In comparing these observed means for the LCAS factors between CURE and inquiry students, we ideally would have first conducted strict measurement invariance tests between the two groups to establish that error variances were similar across groups; however, our CURE student group was too small ( N = 45) to conduct invariance tests ( Rocabado et al. , 2020 ).…”

Multi-institutional Study of Self-Efficacy within Flipped Chemistry Courses

“…Establishing scalar invariance provides support for the use of latent factor means when comparing groups . To do so, structured means modeling (SMM) was used.…”

“…However, if the structural validity of the data is not supported, investigations of the response process and/or content validity may need to be conducted. , Furthermore, if the measured data will be used to compare groups on the latent construct, evidence of consequential validity needs to be established. For self-reported quantitative data, this level of validity can be supported through measurement invariance to determine if group-bias is present in the data structure . Finally, when measures are only administered once per time point, an estimate of the single-administration reliability is warranted …”

Modifying the ASPECT Survey to Support the Validity of Student Perception Data from Different Active Learning Environments

“…Additionally, as Wiggins et al ( 10 ) noted, an important potential use of the data collected by these scales is to better understand if there are equitable outcomes and experiences across student and/or demographic groups in the same classroom. However, evidence of measurement invariance between different groups would first have to be evaluated ( 20 ).…”