A Simulation Study on the Performance of Different Reliability Estimation Methods

Abstract

The accuracy of certain internal consistency estimators have been questioned in recent years. The present study tests the accuracy of six reliability estimators (Cronbach’s alpha, omega, omega hierarchical, Revelle’s omega, and greatest lower bound) in 140 simulated conditions of unidimensional continuous data with uncorrelated errors with varying sample sizes, number of items, population reliabilities, and factor loadings. Estimators that have been proposed to replace alpha were compared with the performance of alpha as well as to each other. Estimates of reliability were shown to be affected by sample size, degree of violation of tau equivalence, population reliability, and number of items in a scale. Under the conditions simulated here, estimates quantified by alpha and omega yielded the most accurate reflection of population reliability values. A follow-up regression comparing alpha and omega revealed alpha to be more sensitive to degree of violation of tau equivalence, whereas omega was affected greater by sample size and number of items, especially when population reliability was low.

The accuracy of reliability coefficients: A reanalysis of existing simulations.

“…x L4 1 ð Þ, x p 1 ð Þ, and GF show similar performance to the above three coefficients in most data sets, excluding Edwards et al (2021). However, because Edwards et al (2021) dataset is somewhat atypical, these three coefficients may be sufficiently accurate in a realistic environment.…”

“…However, my reanalysis shows that a produces a nonnegligible proportion of negative estimates in their data (i.e., 9.7%). In summary, Edwards et al (2021) low-reliability and small-sample data point to the problem of a and q C , not that of q C . Sijtsma (2009a) recommends using the GLB.…”

“…GLB = greatest lower bounds; GF = Gilmer-Feldt coefficient; KC = Kaiser-Caffrey's alpha. * = included inEdwards et al (2021) original analysis; + = proportion of estimates that are less than zero or greater than one; † = convergent solutions exist in all 2,000 repetitions; ‡ = no convergent solution was found in 498 (24.9%) of 2,000 repetitions of this condition, implying that this figure is underestimated; the underlined numbers are the minima in the columns.…”

“…The reason simulation studies are less cited for their importance is that existing studies draw inconsistent conclusions. Seven Monte Carlo simulation studies (Edwards et al, 2021; Kamata et al, 2003; Oosterwijk et al, 2017; Şimşek & Noyan, 2013; Tang & Cui, 2012; van der Ark et al, 2011; Yang & Green, 2010) and three non-Monte Carlo simulation studies (Osburn, 2000; Revelle & Zinbarg, 2009; Sijtsma, 2009a) recommend 10 reliability estimators as the best. There is little explanation about the causes of these divergent conclusions.…”

A Scoping Review of Stress Measurements and Psychometry in Police Research

“…As measurements cannot be valid unless they are reliable (Keszei et al 2010), reliability is a way of assessing the psychometric properties of measurements. Cronbach's alpha assesses the degree to which a scale or composite is consistent, whether internally or temporally, in measuring variation in a sample (Edwards et al 2021) and represents one way, among others, to assess reliability. Known discussions of the limitations of the use of Cronbach's alpha compared to other methods are its assumptions of uncorrelated errors, tau equivalence, unidimensionality, and normality (Trizano-Hermosilla and Alvarado 2016).…”