Moving beyond Alpha: A Primer on Alternative Sources of Single-Administration Reliability Evidence for Quantitative Chemistry Education Research

Abstract

This methodological paper examines current conceptions of reliability in chemistry education research (CER) and provides recommendations for moving beyond the current reliance on reporting coefficient alpha (α) as reliability evidence without regard to its appropriateness for the research context. To help foster a better understanding of reliability and the assumptions that underlie reliability coefficients, reliability is first described from a conceptual framework, drawing on examples from measurement in the physical sciences; then classical test theory is used to frame a discussion of how reliability evidence for psychometric measurements is commonly examined in CER, primarily in the form of single-administration reliability coefficients. Following this more conceptual introduction to reliability, the paper transitions to a more mathematical treatment of reliability using a factor analysis framework with emphasis on the assumptions underlying coefficient alpha and other single-administration reliability coefficients, such as omega (ω) and coefficient H, which are recommended as successors to alpha in CER due to their more broad applicability to a variety of factor models. The factor analysis-based reliability discussion is accompanied by R code that demonstrates the mathematical relations underlying single-administration reliability coefficients and provides interested readers the opportunity to compute coefficients beyond alpha for their own data.

Is This Science? Students’ Experiences of Failure Make a Research-Based Course Feel Authentic

“…To determine the appropriate statistic to use as an estimate of the internal consistency of our instrument scales, we ran single-factor CFAs for each of the three factors using both a congeneric model (i.e., unrestricted factor loadings) and a tau-equivalent model (i.e., all factor loadings are forced to be equivalent; Komperda et al. , 2018b ).…”

Development of the Departmental Climate around Teaching (DCaT) survey: neither psychological collective climate nor departmental collective climate predicts STEM faculty’s instructional practices

“…In this study, the sample size met the minimum criteria of five participants per item. Since we did not expect certain items within a construct to be more important than others in describing the construct, we followed the guidelines provided by Komperda et al (2018) and used the Tauequivalent indicator. In this process, all factor loadings within the same factor are set to be identical to ensure that each item within the same construct is equally weighted.…”

“…Testing for internal consistency provides evidence that respondents to an instrument answer similar items in similar ways. The internal consistency of the survey was measured with Cronbach's alpha coefficient since it has been suggested to be an acceptable reliability measure for Tau Equivalent models (Komperda et al, 2018). A cut-off value of > 0.7 is recommended (Komperda et al, 2018;Streiner, 2003).…”

Design and Validation of an Instrument to Measure Students’ Interactions and Satisfaction in Undergraduate Chemistry Laboratory Classes

“…In the congeneric model, the degrees of association between each item and the common construct are not necessarily the same, while in the tau-equivalent model, the factor loadings are restricted to be equivalent (Cho and Kim 2015;Graham 2006;Harshman and Stains 2017). Cronbach's Alpha is appropriate under a condition when the assumptions for the tau-equivalent model are met, and Omega is suitable when the assumptions for the tau-equivalent model are not met but fit for the congeneric model (Komperda, Pentecost, et al 2018). In this study, as shown below, two items fit the tau-equivalent model and three items fit the congeneric model.…”

“…Consequently, an overall internal consistency value or a single-administration reliability coefficient for each interaction scale was calculated (Komperda, Hosbein, and Barbera 2018). Secondly, to determine which coefficient was appropriate in this study, a single-factor CFA model for each interaction scale was analysed using the congeneric and tau-equivalent models (Komperda, Pentecost, and Barbera 2018). In the congeneric model, the degrees of association between each item and the common construct are not necessarily the same, while in the tau-equivalent model, the factor loadings are restricted to be equivalent (Cho and Kim 2015;Graham 2006;Harshman and Stains 2017).…”