A Bayesian approach to estimating variance components within a multivariate generalizability theory framework

Abstract

In many behavioral research areas, multivariate generalizability theory (mG theory) has been typically used to investigate the reliability of certain multidimensional assessments. However, traditional mG-theory estimation-namely, using frequentist approaches-has limits, leading researchers to fail to take full advantage of the information that mG theory can offer regarding the reliability of measurements. Alternatively, Bayesian methods provide more information than frequentist approaches can offer. This article presents instructional guidelines on how to implement mG-theory analyses in a Bayesian framework; in particular, BUGS code is presented to fit commonly seen designs from mG theory, including single-facet designs, two-facet crossed designs, and two-facet nested designs. In addition to concrete examples that are closely related to the selected designs and the corresponding BUGS code, a simulated dataset is provided to demonstrate the utility and advantages of the Bayesian approach. This article is intended to serve as a tutorial reference for applied researchers and methodologists conducting mG-theory studies.

The Use of Multivariate Generalizability Theory to Evaluate the Quality of Subscores

“…Another would be to evaluate scriptG for examinees at different levels of ability, as implied by the work of Haladyna and Kramer (2004) who reported that low scoring examinees exhibited more variable score profiles. Another follow-up is to estimate variance and covariance components of the p• x i ° using (a) the confirmatory factor analysis (CFA) framework suggested by Marcoulides (1996) and/or (b) the Bayesian framework suggested by Jiang and Skorupski (2017). Another line of research is to evaluate the reliability of score profiles aggregated at the level of the classroom or institution.…”

Improving generalizability coefficient estimate accuracy: A way to incorporate auxiliary information

“…Alternatively, one can use a Bayesian framework to analyze a G theory study (Little and Rubin, 2014). During the Bayesian estimation process, missing data imputation can be accommodated simultaneously (Jiang and Skorupski, 2017; Qin, 2018).…”

Particle Swarm Optimization for Diagnosis Task: An Exploratory Study

“…However, the increment of the number of latent classes would slow down the computation dramatically. On the other hand, the MCMC algorithms have become popular over the past decades [6] [7], and the idea has been implemented in CDMs [8]. Jiang and Carter use MCMC to estimate LCDMs [9].…”