This methodological paper examines current conceptions of reliability in chemistry education research (CER) and provides recommendations for moving beyond the current reliance on reporting coefficient alpha (α) as reliability evidence without regard to its appropriateness for the research context. To help foster a better understanding of reliability and the assumptions that underlie reliability coefficients, reliability is first described from a conceptual framework, drawing on examples from measurement in the physical sciences; then classical test theory is used to frame a discussion of how reliability evidence for psychometric measurements is commonly examined in CER, primarily in the form of single-administration reliability coefficients. Following this more conceptual introduction to reliability, the paper transitions to a more mathematical treatment of reliability using a factor analysis framework with emphasis on the assumptions underlying coefficient alpha and other single-administration reliability coefficients, such as omega (ω) and coefficient H, which are recommended as successors to alpha in CER due to their more broad applicability to a variety of factor models. The factor analysis-based reliability discussion is accompanied by R code that demonstrates the mathematical relations underlying single-administration reliability coefficients and provides interested readers the opportunity to compute coefficients beyond alpha for their own data.

This study examined similarities and differences in study approaches reported by general chemistry students performing at different achievement levels. The study population consisted of freshmen enrolled in a required year-long general chemistry course at the U.S. Naval Academy. Students in the first and second semesters of the course were surveyed using a modified version of the published Approaches and Study Skills Inventory for Students (ASSIST) referred to as the M-ASSIST (Modified Approaches and Study Skills Inventory). Responses to items associated with using deep or surface approaches to studying were examined for students of three achievement levels (A/B, C, and D/F course grades) using both ANOVA and Structured Means Modeling to look for differences in study approaches between achievement levels. Results show that, with only 12 items, the M-ASSIST can be used to measure differences in reported use of deep and surface approaches by students in different achievement groups; that Structured Means Modeling can uncover significant differences that are not apparent with an ANOVA analysis of the same data; and that A/B and D/F students can be classified as reporting using either using primarily deep (A/B students) or primarily surface (D/F) study approaches. C students reported study approaches characteristic of both the A/B and D/F groups, leading to the interpretation that C students may be in an intermediate and possibly transitional state between the higher- and lower-grade groups. These results suggest a new understanding of C students as those who may not fully implement deep approaches to studying but, in general, demonstrate less reliance on surface approaches than lower-achieving students.

As the field of chemistry education moves toward greater inclusion and increased participation by underrepresented minorities, standards for investigating the differential impacts and outcomes of learning environments have to be considered. While quantitative methods may not be capable of generating the in-depth nuances of qualitative methods, they can provide meaningful insights when applied at the group level. Thus, when we conduct quantitative studies in which we aim to learn about the similarities or differences of groups within the same learning environment, we must raise our standards of measurement and safeguard against threats to the validity of inferences that might favor one group over another. One way to provide evidence that group comparisons are supported in a quantitative study is by conducting measurement invariance testing. In this manuscript, we explain the basic concepts of measurement invariance testing within a confirmatory factor analysis framework with examples and a step-by-step tutorial. Each of these steps is an opportunity to safeguard against interpretation of group differences that may be artifacts of the assessment instrument functioning rather than true differences between groups. Reflecting on and safeguarding against threats to the validity of the inferences we can draw from group comparisons will aid in providing more accurate information that can be used to transform our chemistry classrooms into more socially inclusive environments. To catalyze this effort, we provide code in the ESI for two different software packages (R and Mplus) so that interested readers can learn to use these methods with the simulated data provided and then apply the methods to their own data. Finally, we present implications and a summary table for researchers, practitioners, journal editors, and reviewers as a reference when conducting, reading, or reviewing quantitative studies in which group comparisons are performed.

The Cronbach's alpha (α) statistic is regularly reported in science education studies. However, recent reviews have noted that it is not well-understood. Therefore, this commentary provides additional clarity regarding the language used when describing and interpreting alpha and other estimates of reliability.