The accuracy of reliability coefficients: A reanalysis of existing simulations.

Abstract

Controversy over which reliability estimators should be used persists due to a lack of knowledge about their accuracy. Simulation is an effective tool to obtain an answer, but existing simulation studies yield contradictory results regarding which reliability estimators are the best. The causes of these inconsistent conclusions have yet to be discussed. This study reanalyzes existing studies to understand these contradictions. The most important reason is that previous studies consider only a few reliability estimators. This study examines approximately 30 reliability estimators and finds that there is no single, most accurate reliability estimator across all data types. Instead, several reliability estimators are accurate to comparable levels for unidimensional data (congeneric reliability, Guttman's lambda2, and ten Berge-Zegers's mu). Likewise, multiple reliability estimators perform similarly for multidimensional data (multidimensional parallel reliability, correlated factors reliability, and second-order factor reliability). Whereas many recent studies support factor analysis (FA) reliability estimators, this study shows that not all FA reliability estimators are accurate and that some cause severe overestimation.

Reliability and omega hierarchical in multidimensional data: A comparison of various estimators.

“…This study uses the performance of a as a criterion for inaccuracy. Existing empirical studies reach the same conclusion that a is inaccurate as a multidimensional reliability estimator (Cho, 2022;Kamata et al, 2003;S ims ek & Noyan, 2013). Using this result, A combination of congeneric reliability (Jöreskog, 1971) q BPCA A combination of Kaiser and Caffrey (1965) Congeneric reliability, a unidimensional CFA approach q PCA Kaiser and Caffrey (1965) PCA approach -Dependent q BFS Subtest reliability obtained from the bifactor model q SOFS Subtest reliability obtained from the second-order factor model q CFS Subtest reliability obtained from the correlated factors model -…”

“…Sample sizes of 50 and 1,000 are the most typical choices in empirical studies on reliability (Cho, 2022). A sample size of 300 approximates the median of a convenience sample of a dozen Journal of Applied Psychology articles.…”

“…Second, we do not know which of the several FA reliability estimators is the best. Cho (2022) claimed that q MP , q SOF , and q CF are more accurate than q BF and q EFA . However, his research does not examine nonproportional data.…”

“…Kamata et al (2003) report that a non-FA reliability estimator (i.e., stratified alpha) is more accurate than an FA reliability estimator (i.e., reliability based on the bifactor model). After comparing several FA and non-FA estimators, Cho (2022) argues that neither type of reliability estimator is superior to the other. However, these empirical studies examine only limited data structures, which may have been advantageous to non-FA estimators and disadvantageous to FA estimators.…”

Who Denigrates Today’s Youth?: The Role of Age, Implicit Theories, and Sharing the Same Negative Trait

“…For our investigation we used both subscales (see Supplementary Appendix for full question wording and scoring). Reliability in our sample was good for the two-factor correlated factor model [P CF = 0.85; Cho (2016 , 2022) ].…”