Improved Reliability Estimates for Small Samples Using Empirical Bayes Techniques

Abstract

Issues of equity and fairness across subgroups of the population (e.g., gender or ethnicity) must be seriously considered in any standardized testing program. For this reason, many testing programs require some means for assessing test characteristics, such as reliability, for subgroups of the population. However, often only small sample sizes are available for the subgroups of interest. Traditionally used reliability estimates (e.g., Cronbach's alpha) can have low precision for small samples. This study investigated whether an empirical Bayes (EB) technique could produce more precise reliability estimates than traditional methods in the presence of small samples. Several Bayesian estimates were compared to estimates obtained by other methods (e.g., the traditionally and currently used Cronbach's alpha coefficient), in terms of both bias and variance. A secondary purpose of this study was to compare the various EB approaches across different sample sizes. This paper also discusses EB estimates of standard error of measurement (SEM), their accuracy and precision, and how they compare with SEM estimates derived from the alpha.

The comparison of the scores obtained by Bayesian nonparametric model and classical test theory methods

“…20,47 In addition, unlike the classical model, using the correct covariates in the BNP model allows the analysis of the characteristics of included variables according to their subcategories. 19,20,22,25,34,35 The density of high scores in both booklets was shown to increase among students who had a positive attitude towards mathematics, and the density of low scores increased among students who showed a negative attitude. The high score density of male students was higher than female students, which can be expressed as male students had better scores than female.…”

“…15 The Bayesian model, which is based on probability, gives better results than the estimates made according to these theories and provides more detailed information about the research. [16][17][18][19][20][21][22] The Bayes' statistical method has a flexible structure and will contribute to the development of educational research. Recently, an equated scores approach applying Bayesian modelling has been used.…”

“…According to the Bayesian approach, each of the parameters is accepted as a probability variable and is based on obtaining inferential information using previous information. 22 , 26 , 27 Unlike classical methods, the Bayesian approach gives more relevant posterior inferences by including prior knowledge in the study to quantify certain uncertainties. 17 , 28 The Bayesian approach provides many advantages, and a few are mentioned below:…”

The Comparison of the Equated Tests Scores by Using Various Covariates using Bayesian Nonparametric Model

“…Ayrıca birçok büyük uygulamaları gerektiren sınavlarda ortak madde veya ortak test bulunmamaktadır. Bu durumda test puanları ile ilişkili ve gruplar arasındaki farkı açıklayabilen değişkenlerin kestirim sürecine ek bilgi olarak veya ortak testlerin yerine eklenmesi ile yanlılık ve ortalama standart hata azaltılabilir (Branberg & Wiberg, 2011;Liou ve diğerleri, 2001;Oh, Guo, & Walker, 2009;Wiberg, 2015;Wiberg & Branberg, 2015). Böylece kestirimin doğruluğunu arttırabileceği için eşitleme çalışmaları birçok yönden incelenebilecektir (Branberg & Wiberg, 2011;Kim, Livingston, & Lewis, 2009Oh ve diğerleri, 2009;Wiberg & Branberg, 2015).…”

“…Moreover, the tests to be equated may not have any common items or tests. In this case, the bias and mean standard error can be reduced by adding variables associated with the test scores to the test equating process, which allows to explain the difference between the groups (Branberg & Wiberg, 2011;Liou et al, 2001;Oh, Guo, & Walker, 2009;Wiberg, 2015;Wiberg & Branberg, 2015), and to increase the accuracy of the estimation (Branberg & Wiberg, 2011;Kim, Livingston, & Lewis, 2009Oh et al, 2009;Wiberg & Branberg, 2015). Wiberg and Branberg (2015) stated that using a single common variable that has a high correlation with the test scores could give results similar to a common test.…”